Recommendation
DOI or URL of the report: https://osf.io/w7n6z/
Version of the report: v2
2nd R&R revised manuscript: https://osf.io/zbqyr/
All revised materials uploaded to: https://osf.io/t5kz9/ , updated manuscript under sub-directory "PCI-RR submission following R&R 2"
, posted 06 Jun 2022Three of the four reviewers kindly returned to evaluate your revised manuscript. The reviews are broadly promising, with a few outstanding issues concerning the rationale for the replication, the determination of the SESOI, and whether a pilot study may be required (although I am personally cautious about the use of pilots in establishing effect size estimates). I would like see your response to these points before issuing a final decision.
I thank the authors for the responses to the reviews. I`m mostly satisfied with the changes the authors made. Some minor remaining points below:
1) I can follow your reasoning for the choice of replication target. Although I still don`t think that the section is strong enough (basically the same reasons as in my earlier review). You still do not describe why replicating this is e.g., important for theory. Likewise, the section on practical implications (p.8) still is written on a very general level and does not spell out the practical implications of their work. I would not recommend rejection based on this, but I do believe that the authors should very carefully consider their choices of replication targets and build a strong argument for their choice.
2) "We found it difficult to understand whether the bottom line we should take from this comment is
that you think we are too well-powered or not powered enough, or maybe a completely different
point entirely. Clearer guidelines and constructive examples would have been helpful. We looked
at the citations and we did not find a way to tie those into our replication effort, nor were we able
to find any instances of replications making use of this paradigm."
Let me try and rephrase: If you argue that a particular effect size is your smallest effect of interest, then in my view you need to answer the question why that is the case. Merely saying that this corresponds to a small effect in social psychology does not do the trick. If your argument is that it is the SESOI because it allows for a fair test to detect the original findings (paraphrased from your response letter), then add that to this section and make it explicit.
Best Regards,
Max Primbs (not a Dr. yet)
, 30 May 2022The authors addressed my comments very well and I do not have any further comments. I am looking forward to seeing the results.
DOI or URL of the report: https://osf.io/v46fw/
Revised manuscript: https://osf.io/w7n6z/
All revised materials uploaded to: https://osf.io/t5kz9/ , updated manuscript under sub-directory "PCI-RR submission following R&R"
, posted 29 Mar 2022Four reviewers have now completed their evaluations of the Stage 1 manuscript. Overall, I would judge their reviews to be critical but cautiously optimistic -- and highly detailed and constructive. All highlight the value of this replication attempt while also noting a range of areas that would benefit from improvement or clarification in order to meet the Stage 1 criteria.
Across the four sets of evaluations, some of the major issues to address include (in no particular order): strengthening the justification for a replication (see below for my view on this point), considering the pros and cons of a pre-test, including contingent analyses in light of potential failure of statistical assumptions, clarifying conditions under which hypotheses will be deemed unsupported, strengthening the justification of the SESOI, consideration of equivalance testing to provide positive evidence of absence, clarifying the rationale for studying framing effects, confirmation of ethics approval, tackling major methodological deviations from the original study (e.g. such as as the sampling method, and strategies to address it), clarification of study materials to match the original methods, and overall clarity of presentation in key areas.
Considering the the first point above -- strengthening the justification for conducting a replication -- under the PCI RR criteria, replications require no additional justification over and above any other kind of research, and unlike some journals, we do not evaluate Stage 1 RRs on the basis of the perceived importance or value of a research question (but rather, the scientific validity of that question). For this reason, there is no risk of rejection on the grounds that a replication may be unnecessary. That said, I think it would be very much in your interest to address this point constructively in revision because it will make the presentation more compelling for readers, and the reviewer who raised it offers some useful ideas on how to do so.
This list of issues is not exhaustive and you will find that the reviews provide a wide range of additional points. On the basis of their assessments, and my own reading of the manuscript, I am happy to offer the opportunity to submit a major revision.
, 24 Mar 2022The Stage 1 manuscript describes the proposal of a study replicating effects described by Lerner and Keltner (2001).
I found the study interesting and worth to be conducted. Moreover, the authors do their best to follow open science practices and perform an informative replication.
Nevertheless, in my opinion, in the current form, the proposal suffers several limitations that should be clarified.
1. General:
a. The submission was difficult to follow and, in my opinion, was a technical merge of different parts rather than a comprehensive and consistent text with a story the authors wanted to tell. If the manuscript contains so many tables and disclosures spread through the text, the reader could miss crucial details about the research problem, method, etc. I had first to read the original article by Lerner and Keltner (2001) thoroughly to understand the design of a proposed study. Of course, making science is very technical, but without these soft and narrative aspects, the manuscript would be less accessible to readers.
b. I like the idea of extending the original study on trait hope. Indeed it complements the design in the aspects of valence and controllability/certainty. Nevertheless, I recommend the authors elaborate more on this emotion and how it was linked to risk perception/preference in the previous research.
c. The manuscript often refers to the supplement, and the supplement itself is very extensive (20 pages). I think that it would be good to add in the manuscript the precise location in this supplement (e.g., “detailed description could be found in the supplementary Table SX, p. X”).
2. Ethics
a. Whether the authors received the agreement from their ethics committee? I can’t find this information in the manuscript (nevertheless, the HKU ethics committee is mentioned in the Qualtrcs survey). I also think that participants should be informed that their anonymized data will be publically shared.
3. Method:
a. I generally like the idea of merging three original studies into one, but I have some concerns about a new modified framing task. I think that, before the main replication study, authors should conduct a pilot one testing whether, in this particular form of a task (e.g., within-subject, different name, different numbers in the description), the classic framing effect would be found.
b. Based on a description, it is hard to find how the authors plan to deal with ambiguous and unambiguous events, e.g., how they would be scored based on certainty/controllability. Moreover, in some parts, the authors write about this variable as continuous and as dichotomous in others (e.g., in Table 4 or in hypotheses in PCIRR-Study Design Table).
c. Which 4 predictors were considered in power analysis? Dispositional emotions? Please be more precise. How about ambiguity?
d. I think that Table 3 is problematics in some aspects, e.g.,
- It is not clear what “year” means. In the case of Lerner and Keltner, it is the date of publication and not data collection (e.g., the manuscript was submitted on “October 4, 2000” and probably the data were collected even earlier)
- Some values from the simulated dataset do not make sense e.g., authors would not receive an age range from 0-100 as they only accept 18+ years old participants.
- There is also one critical difference between Study 3 and the proposed design that should be stressed: in Lerner and Keltner’s study 3, participants responded to items in a face-to-face interview. The original authors thought that “having participants respond orally rather than in an anonymous self-report might reduce the tendency for happy and angry individuals to see themselves as comparatively less vulnerable to negative life events.” (p. 152).
e. Please add sample items in the manuscript when presenting used methods.
4. Results:
a. The description of a result section is inconsistent. In some parts, there is a table with simulated data. In other, the authors describe that “[The regression tables of these analyses inserted here]” or just free space, e.g., “Third, we regressed the optimism estimates on the emotion dispositions separately for ambiguous and unambiguous events. For ambiguous events, we … For unambiguous events…”. Such description is difficult to follow and be adequately understood.
b. Please plan a table summarizing descriptive statistics for framing task, optimism, controllability/uncertainty, etc.
c. Please plan a comparison of whether the same situations as in the original one would be classified as ambiguous or unambiguous.
5. Supplementary materials:
a. It would be worth adding information about reverse coding when presenting items/scales in the supplementary materials.
b. It is also not clear how the responses in a framing task will be indexed.
6. Other:
a. The authors could consider adding the following keywords: “risk preference”, “hope”, “optimism”
b. Table 1. Why “trait anger” is written in italics?