Recommendation

Replicating positive evaluations of our "true selves"

based on reviews by Andrew Christy, Cillian McHugh, Caleb Reynolds and Sergio Barbosa
A recommendation of:
toto

Revisiting the link between true-self and morality: Replication and extensions of Newman, Bloom and Knobe (2014) Studies 1 and 2

Abstract
Submitted: 15 February 2022, Recommended: 13 June 2022

Recommendation

The concept of a “true self” – the deepest and most genuine part of a person’s personality – is fundamental to many aspects of psychology, with influences that extend deep into society and culture. For decades, research in psychology has consistently found that people see their true selves as positive and virtuous. But people also positively regard (and indeed overestimate) many other characteristics related to the self, such as their abilities and achievements, prompting the question of whether there is anything special about the “true self” as a psychological concept. In an influential study, Newman et al. (2014) found that people were more likely to attribute morally good than morally bad changes in the behaviour of other people to their true selves. Crucially, they also found that our tendency to view the true self positively is shaped by our own moral values – in essence, what we regard as morally or politically good, we see in the true selves of others. Newman et al’s findings suggest that the tendency for us to regard our true self in a positive light stems from the specific nature of true self as a concept. 
 
In the current study, Lee and Feldman (2022) propose to replicate two key studies from Newman et al. (2014) in a large online sample. In particular, they will ask whether true-self attributions are higher for changes in behaviour that are morally positive compared to morally negative or neutral, and, further, how true-self attributions are aligned with personal moral/political views. The authors also propose exploring the relationship between true-self attributions and perceived social norms.
 
The Stage 1 manuscript was evaluated over one round of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).
 
URL to the preregistered Stage 1 protocol: https://osf.io/v2tpf
 
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA. 
 
List of eligible PCI RR-friendly journals:
 
References
 
1. Newman, G. E., Bloom, P., & Knobe, J. (2014). Value judgments and the true self. Personality and Social Psychology Bulletin, 40, 203–216. https://doi.org/10.1177/0146167213508791
 
2. Lee, S. C. & Feldman, G. (2022). Revisiting the link between true-self and morality: Replication and extension of Newman, Bloom, and Knobe (2014) Studies 1 and 2, in principle acceptance of Version 2 by Peer Community in Registered Reports. https://osf.io/v2tpf
Cite this recommendation as:
Chris Chambers (2022) Replicating positive evaluations of our "true selves". Peer Community in Registered Reports, . https://rr.peercommunityin.org/articles/rec?id=174

Evaluation round #1

DOI or URL of the report: https://osf.io/p4rch/

Author's Reply, 17 Jun 2022

Download author's reply Download tracked changes file

Revised manuscript: https://osf.io/8rp96/

All revised materials uploaded to: https://osf.io/9fvtq/, updated manuscript under sub-directory "PCIRR Stage 1\PCI-RR submission following R&R

Decision by , 12 Apr 2022

Four reviewers have now kindly evaluated the Stage 1 manuscript, and I am happy to report that they are broadly very positive, which bodes well for the submission achieving in-principle acceptance. The assessments are detailed and constructive, following from careful reading of the manuscript and offering a range of suggestions for improvement. Without providing an exhaustive list, some of the headline issues that I noted include unpacking the theoretical rationale in key areas (including consideration of additional literature), justification of specific predictions (even though, broadly, the reviewers found the hypotheses well motivated), clarification of a range of methodological details, potential strengthening of data quality checks and expansion of manipulation checks, addressing concerns with the MTurk sample selection, resolving discrepancies between methodological details presented in the main text and supplementary information, and considering the validity of ‘true self’ measurements.
 
From an editorial perspective, all of the issues raised seem readily addressable, therefore I am pleased to invite a thorough revision and response.

Reviewed by , 08 Apr 2022

The content of my review is in the attached Word document, which is also signed. But I'll sign here too!

--
Andrew G. Christy, Ph.D.
Assistant Professor of Psychology
SUNY Plattsburgh
achri021@plattsburgh.edu

Download the review

Reviewed by , 06 Apr 2022

Thank you for the opportunity to review this work. I am enthusiastic about the proposed project. I think there is considerable value in replicating previous work. The authors here also include an extension element that could potentially further our understanding of the mechanisms at play in this paradigm. I applaud the authors on their efforts. I think this is a very strong proposal and I look forward to seeing what the results will be.

I have some comments for possible improvements (particularly in relation to theory and framing). I have provided my comments under the recommended evaluation criteria below.

1A. The scientific validity of the research question(s)

I have no concerns here. The authors propose a replication of previous work. They will test if people perceive the "true self" as being more reflective of morally good, rather than morally bad changes, and if perceptions of what is morally good are related to their own moral views. These questions are taken directly from previous work, and the authors present a research plan that is well suited to addressing them.

1B. The logic, rationale, and plausibility of the proposed hypotheses (where a submission proposes hypotheses)

I have no particular concerns with the hypotheses generally. They are taken from previous work, and the authors provide a good overview of this work. However, I think there is room for a more in depth discussion of the theoretical rationale for the hypotheses. This is especially true for H3 (the extension). The authors provide a descriptive rationale for why one might expect the proposed relationship, but this rationale does not appear to be grounded in existing literature or theoretical approaches.

I would also like to see an expanded discussion of the theory informing H1 and H2, why did the original authors arrive at these predictions? What is their relevance in contemporary theory? Please note however, that I'm not requesting an exhaustive review of the literature, but I would like to see a bit more depth of discussion than is currently provided.

In addition to unpacking the rationale that supported the development of the original hypotheses, there is some interesting work on moral character that I think is relevant to the current study that could enhance the framing of the introduction. In particular these offer an alternative perspective and potentially offering competing hypotheses. I have provided a very brief summary of some of the key points in my own work (see McHugh et al., 2022, pp 136-137) but I encourage the authors to study the original papers and either integrate these potential alternative predictions into their manuscript, or to clearly differentiate between these lines of research. The papers I am thinking of are Klein and O'Brien (2016), Siegel et al., (2018)

Klein and O’Brien (2016) demonstrate that when judging transformations of moral character, people appear to perceive negative transformations much quicker than positive transformations. Participants were presented with vignettes describing changes in patterns of behavior and asked to indicate how many consecutive instances of the new behavior would need to occur to convince them that the actor’s moral character had transformed. Participants relied on fewer instances to judge a negative transformation than a positive transformation.

Similarly, Siegel et al. (2018) report that beliefs about bad agents are more volatile than beliefs about good agents, noting that, “bad people often behave morally, but good people rarely behave immorally” (p. 750).

Both these examples potentially undermine the true self narrative. I suspect the difference in design (repeated vs single exposure) means these may not be directly comparable to the current work. However, I think there is broader theoretical overlap that I think could strengthen the current work, if integrated (e.g., offering a framework for interpreting results that run counter to H1).

Klein, N., & O’Brien, E. (2016). The Tipping Point of Moral Change: When Do Good and Bad Acts Make Good and Bad Actors? Social Cognition, 34(2), 149–166. https://doi.org/10.1521/soco.2016.34.2.149

McHugh, C., McGann, M., Igou, E. R., & Kinsella, E. L. (2022). Moral Judgment as Categorization (MJAC). Perspectives on Psychological Science, 17(1), 131–152. https://doi.org/10.1177/1745691621990636

Siegel, J. Z., Mathys, C., Rutledge, R. B., & Crockett, M. J. (2018). Beliefs about bad people are volatile. Nature Human Behaviour, 2(10), 750–756. https://doi.org/10.1038/s41562-018-0425-1


1C. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis or alternative sampling plans where applicable)

I commend the authors on their methodology and analysis pipeline. They have presented a comprehensive plan that, as far as I can tell, has left no stone unturned (so to speak). All deviations from the original study are well explained with accompanying justifications.

Minor: Re-check the results for Study 2. Table 12 is empty. I also couldn't see any mention of an interaction discussed when reporting the 2 (political view: liberal and conservative; between) x 2 (item types: liberal and conservative; within) mixed-model ANOVA (the original study reports such an interaction, p. 207)


1D. Whether the clarity and degree of methodological detail is sufficient to closely replicate the proposed study procedures and analysis pipeline and to prevent undisclosed flexibility in the procedures and analyses

There is more than enough detail to replicate this study and analyses. The inclusion of the qualtrics file and the jamovi file is a strength.

1E. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the obtained results are able to test the stated hypotheses or answer the stated research question(s).

I think the authors have done a good job in designing a research pipeline that should provide a good test of the stated hypotheses, e.g., the inclusion of continuous measures may mitigate against false null effects as they can be better suited to identifying small or subtle effects. There are appropriate manipulation checks included.

---

I am thankful for the opportunity to review this work. I appreciate the extensive work the authors have put into preparing a strong proposal. I believe the proposed protocol is suitable and the authors are almost ready to begin data collection. I have made some suggestions for improving the theoretical grounding of this work. I hope the authors don't find this too burdensome to implement.

Thanks again.

Cillian McHugh

Reviewed by , 12 Apr 2022

This Stage 1 submission is clear, precise, and documents essential elements of an RR throughout. I commend the authors on their careful and thorough preparation, and I recommend the submission with no reservations. 

 

1A. The scientific validity of the research question(s). 

The registered report is a replication and extension of prior work, which adds value as part of a cumulative science. Moreover, the choice of these studies for replication is appropriate. The target article is well-known and cited in the literature, and directly addresses a core premise of the lay concept of the “true self”. However, the target article also reports large effects in relatively small samples and features some suboptimal design choices. The tension between these aspects of the target article makes high-powered replication of these studies a valuable endeavor.

 

1B. The logic, rationale, and plausibility of the proposed hypotheses (where a submission proposes hypotheses). 

The first two a priori hypotheses are derived from the target article, and are well articulated in the introduction and in the Study Design Table.

The hypothesis about social norms is clear and seems reasonable, but it not terribly well justified in the text beyond the assertion that, if true, it would suggest an interesting paradox (p. 13). Given that this is the only hypothesis not from the target article, another sentence or two of justification for this prediction would be helpful (indeed, wouldn’t the implication of a paradox suggest one should hypothesize the opposite?).

The specific use of the LeBel et al. criteria to evaluate replicability is a strength.

 

1C. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis or alternative sampling plans where applicable). 

In general, the design and analysis choices seem sound. The replication elements are extremely close, only differing to remove suboptimal or strange elements of the original design. I view the changes as improvements on the original designs. 

The extension elements seem appropriate and feasible, with only a minor quibble: The measure of intuitive true self belief (p. 26) seems odd. In true self ratings in both studies, true self is the particular target’s true self, that is, true self is localized. Yet for this general measure, true self becomes this disembodied universal that exists apart from individual people. I know the literature sometimes talks about true self in a way that projects this, but I’m skeptical that this is how lay people think about the concept of true self. Assuming this is an ad hoc measure, I would suggest the authors consider revising the items. Perhaps something calling out individuals’ true selves might help? e.g., “Most [all?] people’s true self is morally good”.

 

1D. Whether the clarity and degree of methodological detail is sufficient to closely replicate the proposed study procedures and analysis pipeline and to prevent undisclosed flexibility in the procedures and analyses.  

The excellent detail in the submission seems sufficient to closely replicate the proposed study, and the links between questions, hypotheses, and tests are clearly documented.

 

1E. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the obtained results are able to test the stated hypotheses or answer the stated research question(s). 

The authors have employed neutral control conditions and manipulation checks for key elements of their designs, supporting their ability to make inferences about their hypotheses.

 

Signed,

Caleb J. Reynolds, PhD

Reviewed by , 08 Apr 2022

Presented pre-registered replication and extension is doubtlessly planned with great care and foresight. I applaud the use of pre-written code and analysis which is much more rigorous and allows for better review. I have few commments most of which I believe would not necesarily hamper author's chance at publication but might be a welcome addition to the manuscript. I welcome particularly the modifications to the original experiments adding manipulation checks and continous rather than dichotomous variables. However, i do feel that seriousness and language proficiency items deserve to be explicitly written in-text rather than left in supplementary material. As is it reads as quite confusing and the reader is left wondering how do authors assess these.

I believe proposed extensions, specially surrounding distinguishing social and moral norms are perfectly relevant and would certainly contribute to existent litterature. However, I am not convinced by hypothesis regarding social norms having a positive link to true self. I would expect quite the opposite effect as people might view conformity to a social (as opposed to moral) norm as reflecting the desire to please others or to avoid the social costs of counter-normative actions rather than the expression of personal conviction (i.e. arguably part of the true self). Of course, authors need not agree with me but I believe further justification of this hypothesis is certainly beneficial.

On a similar note, i think interpretation of this, possible, result as reflecting that participants believing that the true self is in sinergy with others is not the only plausible explanation to this pattern of results, admitting they are actually observed after data collection. One might also interpret the correlation between social norms and true self as reflecting the true self being influenced by what people believe others believe is moral without necessarily being "in synergy". I am interpreteing "in synergy" as some sort of coordination in real life. Perhaps this is simply an issue with this particular expression but I believe it leaves some ambiguity that authors might want to avoid.

On the subject of extensions. I believe these are properly justified and do enhance the manuscript quality. However, if authors find it interesting and doable with available resources I do suggest comparing scenarios in which the agent changes their behaviour and where they change their opinions/ thoughts. I am sure authors are familiar with literature on the effect of intentions on moral judgment or on harmless wrongs relevant to this topic. I believe a possible extension without more data being collected is actually observing whether true self link to moral or social norms is identical when considering actions or thoughts.

Finally, on to authors explicit request on reviewing data analysis strategy, specially effect size estimation, I believe they have done a great job at it and have nothing to suggest other than discouraging the use of G*Power as its interpretability and usefulness is hotly debated. I suggest a full simulation approach if at all possible. Even as a user of the pwr package I am not sure it allows for power and sensitivity analysis of interactions which, I believe, compels us to consider simulation approach.

I thank the authors for the opportunity to review this paper and for an interesting and rigorous project that will surely meaningfully contribute to literature. I hope I get to see this published manuscript soon and that my comments might make this process a bit less dificult.

 

User comments

No user comments yet