Replicating positive evaluations of our "true selves"

The content of my review is in the attached Word document, which is also signed. But I'll sign here too!

--
Andrew G. Christy, Ph.D.
Assistant Professor of Psychology
SUNY Plattsburgh
achri021@plattsburgh.edu

Thank you for the opportunity to review this work. I am enthusiastic about the proposed project. I think there is considerable value in replicating previous work. The authors here also include an extension element that could potentially further our understanding of the mechanisms at play in this paradigm. I applaud the authors on their efforts. I think this is a very strong proposal and I look forward to seeing what the results will be.

I have some comments for possible improvements (particularly in relation to theory and framing). I have provided my comments under the recommended evaluation criteria below.

1A. The scientific validity of the research question(s)

I have no concerns here. The authors propose a replication of previous work. They will test if people perceive the "true self" as being more reflective of morally good, rather than morally bad changes, and if perceptions of what is morally good are related to their own moral views. These questions are taken directly from previous work, and the authors present a research plan that is well suited to addressing them.

1B. The logic, rationale, and plausibility of the proposed hypotheses (where a submission proposes hypotheses)

I have no particular concerns with the hypotheses generally. They are taken from previous work, and the authors provide a good overview of this work. However, I think there is room for a more in depth discussion of the theoretical rationale for the hypotheses. This is especially true for H3 (the extension). The authors provide a descriptive rationale for why one might expect the proposed relationship, but this rationale does not appear to be grounded in existing literature or theoretical approaches.

I would also like to see an expanded discussion of the theory informing H1 and H2, why did the original authors arrive at these predictions? What is their relevance in contemporary theory? Please note however, that I'm not requesting an exhaustive review of the literature, but I would like to see a bit more depth of discussion than is currently provided.

In addition to unpacking the rationale that supported the development of the original hypotheses, there is some interesting work on moral character that I think is relevant to the current study that could enhance the framing of the introduction. In particular these offer an alternative perspective and potentially offering competing hypotheses. I have provided a very brief summary of some of the key points in my own work (see McHugh et al., 2022, pp 136-137) but I encourage the authors to study the original papers and either integrate these potential alternative predictions into their manuscript, or to clearly differentiate between these lines of research. The papers I am thinking of are Klein and O'Brien (2016), Siegel et al., (2018)

Klein and O’Brien (2016) demonstrate that when judging transformations of moral character, people appear to perceive negative transformations much quicker than positive transformations. Participants were presented with vignettes describing changes in patterns of behavior and asked to indicate how many consecutive instances of the new behavior would need to occur to convince them that the actor’s moral character had transformed. Participants relied on fewer instances to judge a negative transformation than a positive transformation.

Similarly, Siegel et al. (2018) report that beliefs about bad agents are more volatile than beliefs about good agents, noting that, “bad people often behave morally, but good people rarely behave immorally” (p. 750).

Both these examples potentially undermine the true self narrative. I suspect the difference in design (repeated vs single exposure) means these may not be directly comparable to the current work. However, I think there is broader theoretical overlap that I think could strengthen the current work, if integrated (e.g., offering a framework for interpreting results that run counter to H1).

Klein, N., & O’Brien, E. (2016). The Tipping Point of Moral Change: When Do Good and Bad Acts Make Good and Bad Actors? Social Cognition, 34(2), 149–166. https://doi.org/10.1521/soco.2016.34.2.149

McHugh, C., McGann, M., Igou, E. R., & Kinsella, E. L. (2022). Moral Judgment as Categorization (MJAC). Perspectives on Psychological Science, 17(1), 131–152. https://doi.org/10.1177/1745691621990636

Siegel, J. Z., Mathys, C., Rutledge, R. B., & Crockett, M. J. (2018). Beliefs about bad people are volatile. Nature Human Behaviour, 2(10), 750–756. https://doi.org/10.1038/s41562-018-0425-1

1C. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis or alternative sampling plans where applicable)

I commend the authors on their methodology and analysis pipeline. They have presented a comprehensive plan that, as far as I can tell, has left no stone unturned (so to speak). All deviations from the original study are well explained with accompanying justifications.

Minor: Re-check the results for Study 2. Table 12 is empty. I also couldn't see any mention of an interaction discussed when reporting the 2 (political view: liberal and conservative; between) x 2 (item types: liberal and conservative; within) mixed-model ANOVA (the original study reports such an interaction, p. 207)

1D. Whether the clarity and degree of methodological detail is sufficient to closely replicate the proposed study procedures and analysis pipeline and to prevent undisclosed flexibility in the procedures and analyses

There is more than enough detail to replicate this study and analyses. The inclusion of the qualtrics file and the jamovi file is a strength.

1E. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the obtained results are able to test the stated hypotheses or answer the stated research question(s).

I think the authors have done a good job in designing a research pipeline that should provide a good test of the stated hypotheses, e.g., the inclusion of continuous measures may mitigate against false null effects as they can be better suited to identifying small or subtle effects. There are appropriate manipulation checks included.

---

I am thankful for the opportunity to review this work. I appreciate the extensive work the authors have put into preparing a strong proposal. I believe the proposed protocol is suitable and the authors are almost ready to begin data collection. I have made some suggestions for improving the theoretical grounding of this work. I hope the authors don't find this too burdensome to implement.

Thanks again.

Cillian McHugh

This Stage 1 submission is clear, precise, and documents essential elements of an RR throughout. I commend the authors on their careful and thorough preparation, and I recommend the submission with no reservations. 

1A. The scientific validity of the research question(s). 

The registered report is a replication and extension of prior work, which adds value as part of a cumulative science. Moreover, the choice of these studies for replication is appropriate. The target article is well-known and cited in the literature, and directly addresses a core premise of the lay concept of the “true self”. However, the target article also reports large effects in relatively small samples and features some suboptimal design choices. The tension between these aspects of the target article makes high-powered replication of these studies a valuable endeavor.

1B. The logic, rationale, and plausibility of the proposed hypotheses (where a submission proposes hypotheses). 

The first two a priori hypotheses are derived from the target article, and are well articulated in the introduction and in the Study Design Table.

The hypothesis about social norms is clear and seems reasonable, but it not terribly well justified in the text beyond the assertion that, if true, it would suggest an interesting paradox (p. 13). Given that this is the only hypothesis not from the target article, another sentence or two of justification for this prediction would be helpful (indeed, wouldn’t the implication of a paradox suggest one should hypothesize the opposite?).

The specific use of the LeBel et al. criteria to evaluate replicability is a strength.

1C. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis or alternative sampling plans where applicable). 

In general, the design and analysis choices seem sound. The replication elements are extremely close, only differing to remove suboptimal or strange elements of the original design. I view the changes as improvements on the original designs. 

The extension elements seem appropriate and feasible, with only a minor quibble: The measure of intuitive true self belief (p. 26) seems odd. In true self ratings in both studies, true self is the particular target’s true self, that is, true self is localized. Yet for this general measure, true self becomes this disembodied universal that exists apart from individual people. I know the literature sometimes talks about true self in a way that projects this, but I’m skeptical that this is how lay people think about the concept of true self. Assuming this is an ad hoc measure, I would suggest the authors consider revising the items. Perhaps something calling out individuals’ true selves might help? e.g., “Most [all?] people’s true self is morally good”.

1D. Whether the clarity and degree of methodological detail is sufficient to closely replicate the proposed study procedures and analysis pipeline and to prevent undisclosed flexibility in the procedures and analyses.  

The excellent detail in the submission seems sufficient to closely replicate the proposed study, and the links between questions, hypotheses, and tests are clearly documented.

1E. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the obtained results are able to test the stated hypotheses or answer the stated research question(s). 

The authors have employed neutral control conditions and manipulation checks for key elements of their designs, supporting their ability to make inferences about their hypotheses.

Signed,

Caleb J. Reynolds, PhD

Presented pre-registered replication and extension is doubtlessly planned with great care and foresight. I applaud the use of pre-written code and analysis which is much more rigorous and allows for better review. I have few commments most of which I believe would not necesarily hamper author's chance at publication but might be a welcome addition to the manuscript. I welcome particularly the modifications to the original experiments adding manipulation checks and continous rather than dichotomous variables. However, i do feel that seriousness and language proficiency items deserve to be explicitly written in-text rather than left in supplementary material. As is it reads as quite confusing and the reader is left wondering how do authors assess these.

I believe proposed extensions, specially surrounding distinguishing social and moral norms are perfectly relevant and would certainly contribute to existent litterature. However, I am not convinced by hypothesis regarding social norms having a positive link to true self. I would expect quite the opposite effect as people might view conformity to a social (as opposed to moral) norm as reflecting the desire to please others or to avoid the social costs of counter-normative actions rather than the expression of personal conviction (i.e. arguably part of the true self). Of course, authors need not agree with me but I believe further justification of this hypothesis is certainly beneficial.

On a similar note, i think interpretation of this, possible, result as reflecting that participants believing that the true self is in sinergy with others is not the only plausible explanation to this pattern of results, admitting they are actually observed after data collection. One might also interpret the correlation between social norms and true self as reflecting the true self being influenced by what people believe others believe is moral without necessarily being "in synergy". I am interpreteing "in synergy" as some sort of coordination in real life. Perhaps this is simply an issue with this particular expression but I believe it leaves some ambiguity that authors might want to avoid.

On the subject of extensions. I believe these are properly justified and do enhance the manuscript quality. However, if authors find it interesting and doable with available resources I do suggest comparing scenarios in which the agent changes their behaviour and where they change their opinions/ thoughts. I am sure authors are familiar with literature on the effect of intentions on moral judgment or on harmless wrongs relevant to this topic. I believe a possible extension without more data being collected is actually observing whether true self link to moral or social norms is identical when considering actions or thoughts.

Finally, on to authors explicit request on reviewing data analysis strategy, specially effect size estimation, I believe they have done a great job at it and have nothing to suggest other than discouraging the use of G*Power as its interpretability and usefulness is hotly debated. I suggest a full simulation approach if at all possible. Even as a user of the pwr package I am not sure it allows for power and sensitivity analysis of interactions which, I believe, compels us to consider simulation approach.

I thank the authors for the opportunity to review this paper and for an interesting and rigorous project that will surely meaningfully contribute to literature. I hope I get to see this published manuscript soon and that my comments might make this process a bit less dificult.