Replicating the relationship between emotions and judgments of risk

based on reviews by Kelly Wolfe, Max Primbs, Agata Sobków and Karolina Scigala
A recommendation of:

Associations of fear, anger, happiness, and hope with risk judgments: Revisiting appraisal-tendency framework with a replication and extensions of Lerner and Keltner (2001)

Submission: posted 31 January 2022
Recommendation: posted 11 June 2022, validated 13 June 2022


How do emotions interact with cognition? The last 40 years has witnessed the rise of cognitive-appraisal theories, which propose that emotions can be differentiated along an axis of cognitive dimensions such as certainty, pleasantness, attentional activity, control, anticipated effort, and responsibility (Smith and Ellsworth, 1985). Early tests of such theories focused especially on the impact of the valence – pleasantness/unpleasantness – of emotions on judgment and decision-making, finding, for instance, that negative mood induction can heighten pessimistic estimates of risk (Johnson & Tversky, 1983).
The Appraisal-Tendency Framework proposed by Lerner and Keltner (2000) refined cognitive-appraisal theory by proposing that specific emotions trigger a predisposition to appraise future (or hypothetical) events in line with the central appraisal dimensions that triggered the emotion, even when the emotion and the judgment are unrelated. For example, an individual who is triggered to become fearful of a heightened risk, such as nuclear war, may then exhibit heightened pessimism about risks unrelated to war. The Appraisal-Tendency Framework also predicts relationships between traits, such as between fear, anger and risk-taking/risk-seeking tendencies. In an influential paper, Lerner and Keltner (2001) reported direct empirical support for the Appraisal-Tendency Framework, which aside from its influence in cognitive/affective psychology has had considerable impact in behavioural economics, moral psychology, and studies of consumer behaviour.
In the current study, Lu and Feldman (2022) propose to replicate three key studies from Lerner and Keltner (2001) in a large online sample. Through a combination of replication and extension, the authors will probe the relationship between various trait emotions (including fear, anger, happiness, and hope) and trait characteristics of risk seeking and optimistic risk assessment. The authors also propose examining how the ambiguity of triggering events moderates the relationship between specific emotions and risk judgments. 
The Stage 1 manuscript was evaluated over two rounds of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).
URL to the preregistered Stage 1 protocol:
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA. 
List of eligible PCI RR-friendly journals:
1. Smith, C. A., & Ellsworth, P. C. (1985). Patterns of cognitive appraisal in emotion. Journal of Personality and Social Psychology, 48, 813-838.
2. Johnson, E. J., & Tversky, A. (1983). Affect, generalization, and the perception of risk. Journal of Personality and Social Psychology, 45(1), 20–31.
3. Lerner, J. S., & Keltner, D. (2000). Beyond valence: Toward a model of emotion-specific influences on judgment and choice. Cognition & Emotion, 14, 473-493. 
4. Lerner, J. S., & Keltner, D. (2001). Fear, anger, and risk. Journal of Personality and Social Psychology, 81, 146–159.
5. Lu, S. & Feldman, G. (2022). Associations of fear, anger, happiness, and hope with risk judgments: Revisiting appraisal-tendency framework with a replication and extensions of Lerner and Keltner (2001), in principle acceptance of Version 3 by Peer Community in Registered Reports.
Cite this recommendation as:
Chris Chambers (2022) Replicating the relationship between emotions and judgments of risk. Peer Community in Registered Reports, .
Conflict of interest:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.

Evaluation round #2

DOI or URL of the report:

Version of the report: v2

Author's Reply, 10 Jun 2022

Download author's reply Download tracked changes file

2nd R&R revised manuscript:

All revised materials uploaded to: , updated manuscript under sub-directory "PCI-RR submission following R&R 2"

Decision by , posted 06 Jun 2022

Three of the four reviewers kindly returned to evaluate your revised manuscript. The reviews are broadly promising, with a few outstanding issues concerning the rationale for the replication, the determination of the SESOI, and whether a pilot study may be required (although I am personally cautious about the use of pilots in establishing effect size estimates). I would like see your response to these points before issuing a final decision.

Reviewed by , 16 May 2022

I thank the authors for the responses to the reviews. I`m mostly satisfied with the changes the authors made. Some minor remaining points below:

1) I can follow your reasoning for the choice of replication target. Although I still don`t think that the section is strong enough (basically the same reasons as in my earlier review). You still do not describe why replicating this is e.g., important for theory. Likewise, the section on practical implications (p.8) still is written on a very general level and does not spell out the practical implications of their work. I would not recommend rejection based on this, but I do believe that the authors should very carefully consider their choices of replication targets and build a strong argument for their choice.

2) "We found it difficult to understand whether the bottom line we should take from this comment is
that you think we are too well-powered or not powered enough, or maybe a completely different
point entirely. Clearer guidelines and constructive examples would have been helpful. We looked
at the citations and we did not find a way to tie those into our replication effort, nor were we able
to find any instances of replications making use of this paradigm."

Let me try and rephrase: If you argue that a particular effect size is your smallest effect of interest, then in my view you need to answer the question why that is the case. Merely saying that this corresponds to a small effect in social psychology does not do the trick. If your argument is that it is the SESOI because it allows for a fair test to detect the original findings (paraphrased from your response letter), then add that to this section and make it explicit.

Best Regards,

Max Primbs (not a Dr. yet)

Reviewed by , 30 May 2022

Reviewed by , 06 Jun 2022

The authors addressed my comments very well and I do not have any further comments. I am looking forward to seeing the results. 

Evaluation round #1

DOI or URL of the report:

Author's Reply, 13 May 2022

Download author's reply Download tracked changes file

Revised manuscript:

All revised materials uploaded to: , updated manuscript under sub-directory "PCI-RR submission following R&R"

Decision by , posted 29 Mar 2022

Four reviewers have now completed their evaluations of the Stage 1 manuscript. Overall, I would judge their reviews to be critical but cautiously optimistic -- and highly detailed and constructive. All highlight the value of this replication attempt while also noting a range of areas that would benefit from improvement or clarification in order to meet the Stage 1 criteria.

Across the four sets of evaluations, some of the major issues to address include (in no particular order): strengthening the justification for a replication (see below for my view on this point), considering the pros and cons of a pre-test, including contingent analyses in light of potential failure of statistical assumptions, clarifying conditions under which hypotheses will be deemed unsupported, strengthening the justification of the SESOI, consideration of equivalance testing to provide positive evidence of absence, clarifying the rationale for studying framing effects, confirmation of ethics approval, tackling major methodological deviations from the original study (e.g. such as as the sampling method, and strategies to address it), clarification of study materials to match the original methods, and overall clarity of presentation in key areas.

Considering the the first point above -- strengthening the justification for conducting a replication -- under the PCI RR criteria, replications require no additional justification over and above any other kind of research, and unlike some journals, we do not evaluate Stage 1 RRs on the basis of the perceived importance or value of a research question (but rather, the scientific validity of that question). For this reason, there is no risk of rejection on the grounds that a replication may be unnecessary. That said, I think it would be very much in your interest to address this point constructively in revision because it will make the presentation more compelling for readers, and the reviewer who raised it offers some useful ideas on how to do so.

This list of issues is not exhaustive and you will find that the reviews provide a wide range of additional points. On the basis of their assessments, and my own reading of the manuscript, I am happy to offer the opportunity to submit a major revision.

Reviewed by , 24 Mar 2022

Reviewed by , 29 Mar 2022

Reviewed by , 25 Mar 2022

The Stage 1 manuscript describes the proposal of a study replicating effects described by Lerner and Keltner (2001).  

I found the study interesting and worth to be conducted. Moreover, the authors do their best to follow open science practices and perform an informative replication.

Nevertheless, in my opinion, in the current form, the proposal suffers several limitations that should be clarified.

1.     General:

a.     The submission was difficult to follow and, in my opinion, was a technical merge of different parts rather than a comprehensive and consistent text with a story the authors wanted to tell. If the manuscript contains so many tables and disclosures spread through the text, the reader could miss crucial details about the research problem, method, etc. I had first to read the original article by Lerner and Keltner (2001) thoroughly to understand the design of a proposed study. Of course, making science is very technical, but without these soft and narrative aspects, the manuscript would be less accessible to readers.

b.     I like the idea of extending the original study on trait hope. Indeed it complements the design in the aspects of valence and controllability/certainty. Nevertheless, I recommend the authors elaborate more on this emotion and how it was linked to risk perception/preference in the previous research. 

c.      The manuscript often refers to the supplement, and the supplement itself is very extensive (20 pages). I think that it would be good to add in the manuscript the precise location in this supplement (e.g., “detailed description could be found in the supplementary Table SX, p. X”).  

2.     Ethics

a.     Whether the authors received the agreement from their ethics committee? I can’t find this information in the manuscript (nevertheless, the HKU ethics committee is mentioned in the Qualtrcs survey). I also think that participants should be informed that their anonymized data will be publically shared. 

3.     Method:

a.     I generally like the idea of merging three original studies into one, but I have some concerns about a new modified framing task. I think that, before the main replication study, authors should conduct a pilot one testing whether, in this particular form of a task (e.g., within-subject, different name, different numbers in the description), the classic framing effect would be found. 

b.     Based on a description, it is hard to find how the authors plan to deal with ambiguous and unambiguous events, e.g., how they would be scored based on certainty/controllability. Moreover, in some parts, the authors write about this variable as continuous and as dichotomous in others (e.g., in Table 4 or in hypotheses in PCIRR-Study Design Table). 

c.      Which 4 predictors were considered in power analysis? Dispositional emotions? Please be more precise. How about ambiguity?

d.     I think that Table 3 is problematics in some aspects, e.g., 

 - It is not clear what “year” means. In the case of Lerner and Keltner, it is the date of publication and not data collection  (e.g., the manuscript was submitted on “October 4, 2000” and probably the data were collected even earlier) 

 - Some values from the simulated dataset do not make sense e.g., authors would not receive an age range from 0-100 as they only accept 18+ years old participants. 

 - There is also one critical difference between Study 3 and the proposed design that should be stressed: in Lerner and Keltner’s study 3, participants responded to items in a face-to-face interview. The original authors thought that “having participants respond orally rather than in an anonymous self-report might reduce the tendency for happy and angry individuals to see themselves as comparatively less vulnerable to negative life events.” (p. 152). 

e.     Please add sample items in the manuscript when presenting used methods. 

4.     Results:

a.     The description of a result section is inconsistent. In some parts, there is a table with simulated data. In other, the authors describe that “[The regression tables of these analyses inserted here]” or just free space, e.g., “Third, we regressed the optimism estimates on the emotion dispositions separately for ambiguous and unambiguous events. For ambiguous events, we … For unambiguous events…”. Such description is difficult to follow and be adequately understood. 

b.     Please plan a table summarizing descriptive statistics for framing task, optimism, controllability/uncertainty, etc.

c.      Please plan a comparison of whether the same situations as in the original one would be classified as ambiguous or unambiguous. 

5.     Supplementary materials: 

a.     It would be worth adding information about reverse coding when presenting items/scales in the supplementary materials. 

b.     It is also not clear how the responses in a framing task will be indexed. 

6.     Other:

a.     The authors could consider adding the following keywords: “risk preference”, “hope”, “optimism”

b.     Table 1. Why “trait anger” is written in italics?

Reviewed by , 27 Mar 2022

User comments

No user comments yet