Understanding probability assessments with partitioned framing
Revisiting Partition Priming in judgment under uncertainty: Replication and extension Registered Report of Fox and Rottenstreich (2003)
Recommendation: posted 26 March 2025, validated 01 April 2025
Espinosa, R. (2025) Understanding probability assessments with partitioned framing. Peer Community in Registered Reports, 100877. 10.24072/pci.rr.100877
This is a stage 2 based on:
Recommendation
Decision-making based on limited information is a common occurrence. Whether it is the possibility of a cheaper product elsewhere or the unknown qualifications of election candidates, people are regularly forced to make a decision under ignorance or uncertainty. In such situations, information about certain events is unavailable or too costly to acquire and people rely on subjective probability allocation to guide decision-making processes. This allocation seems to result in what is known as ignorance priors, i.e., decision-makers assigning equal probabilities to each possible outcome within a given set. How events are grouped or partitioned is often subjective and may influence probability judgments and subsequent decisions. In such cases, the way the choices within a choice set are presented may shape the perceived likelihood of different outcomes. Understanding the impact of partitioning on probability estimation is crucial for both psychological and economic theories of judgment and decision-making.
In the current work, Ding and Feldman (2025) conducted a replication study of one of the foundational works on the topic: Fox and Rottenstreich (2003). In the original work, the authors provided exploratory evidence indicating that the framing of a situation affects the way individuals perceive probabilities of possible outcomes. They showed that people assigned uniform probabilities to sets of events described in a problem, such that the way the events are described partly determined people’s partitioning of those events and evaluations of the probabilities of the possible outcomes. Additionally, this partitioned framing affected judgments both under conditions of ignorance (where individuals have no information and rely solely on uniform probability assignments) and uncertainty (where individuals have some information but still rely on heuristics influenced by partitioning). This suggests that priors resulting from the inference of available evidence are sometimes partly contaminated by partitioning bias, affecting both uninformed and partially informed decision-making processes. As a consequence, the partitioning of events into different subsets might lead to varying evaluations of a single situation, resulting in inconsistencies and poorly calibrated probability assessments.
Ding and Feldman (2025) conducted a replication work on Studies 1a, 1b, 3, and 4 from Fox and Rottenstreich (2003). Their close replication relies on original data (US participants, Prolific, N=603) with a large statistical power (>95%). The replication aimed to assess whether the partitioned framing affects prior formation under ignorance (Studies 1a, 1b, and 4) and uncertainty (Study 3). The authors also proposed an extension examining estimations of complementary events contrasting estimations of the probabilities of the events happening versus the probabilities of the events not happening.
Overall, the authors successfully replicated the original study based on their pre-registered evaluation criteria, finding support for partition dependence for most scenarios under scrutiny, yet with weaker effect sizes than the original studies. Out of the eleven Cohen’s h estimated by the replication study, one is consistent with the original study’s estimate (i.e., the original point estimate lies within the confidence interval of the replication), seven go in the same direction but are smaller (i.e., same sign for the estimated effect but the original point estimate is outside the CI of the replication), and two are not statistically different from zero (i.e., the CI of the replication includes zero).
- Collabra: Psychology
- International Review of Social Psychology
- Journal of Cognition
- Meta-Psychology
- Peer Community Journal
- PeerJ
- Royal Society Open Science
- Social Psychological Bulletin
- Studia Psychologica
- Swiss Psychology Open
Replication and extension Registered Report of Fox and Rottenstreich (2003) [Stage 2]. Acceptance of Version 4 by Peer Community in Registered Reports. https://osf.io/xdpkt
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.
Evaluation round #1
DOI or URL of the report: https://osf.io/8cszr
Version of the report: 3
Author's Reply, 01 Mar 2025
Revised manuscript: https://osf.io/xdpkt
All revised materials uploaded to: https://osf.io/g9czs/ , updated manuscript under sub-directory "PCIRR Stage 2\PCIRR-S2 submission following RnR"
Decision by Romain Espinosa
, posted 18 Aug 2024, validated 18 Aug 2024
Dear authors,
Thank you very much for submitting your Stage 2 to PCI-RR. I was very pleased to see your completed study and look forward to recommending it. I have included below some of my comments on your manuscript.
The two referees who reviewed your Stage 1 gratefully accepted to also examine your completed study. As you will see, both referees have a very high opinion of your work. Olivier suggests some minor changes to improve the readability of the manuscript and to expand a bit your discussion horizon. I let you decide whether you think these changes are worth including. (The same applies for my own comments.)
Don suggests a much more drastic change in the manuscript as he believes that a restructuring would considerably help the readers appreciate your replication work. Please note that the PCI-RR rules do not require authors to change the structure of the paper with respect to what has been agreed upon at Stage 1. In other words, you are free not to make any changes in this regard. However, my personal opinion tends to align with Don’s view. A great strength of your study (and Feldman's work in general) is your meticulous work on all technical details and issues related to replication studies. This is highly appreciated and, in my opinion, greatly contributes to the production of a robust corpus of knowledge in the field. However, I think that an external reader, who is interested in whether the original study replicates or not, and who will have a quick read of your work just to know the results, might be discouraged because of all these methodological discussions. I am a bad judge myself as I tend to do the same as you do, but I received the very same comment for one of my works, and, having read your Stage 2 in detail for this new round, I now understand what the referee meant at that time. I think that all Feldman's replication efforts are also more impactful if 'standard' researchers (i.e., not replication people but readers who are just interested in the particular research question of the original study) can easily get to know the replication results.
After talking with the editorial board of PCI-RR, we let you the possibility of restructuring the paper if you want to, and move some methodological discussions in the Supplementary Materials. I would perfectly understand if you do not want to do so, and once again, this is optional and does not condition Stage 2 acceptance. I do not know your publication strategy, so you might want to leave it intact or have it restructured now, depending on what you have in mind.
I am looking forward to receiving your revised manuscript. I think Olivier's suggestions and mine should not take too long.
Thanks again for choosing PCI-RR for your work. We are lucky to have such high-quality replication studies at PCI-RR.
Best regards,
Romain
—
• Regarding the ‘reverse not happen’ element: I understand that there was an unexpected possibility to report a probability of the universe of events different from 100%. You decided to depart from your registered analysis plan by focusing on one probability and reconstructing adequate probability measures. I am satisfied with the deviation, and how it is implemented and justified.
• Regarding Table 13, I would suggest adding the table notes a brief explanation of the interpretation from Lebel et al. (2019). I think that your results fall into these three categories (copy-pasting from Lebel et al., 2019, which I went reading):
1) Signal – consistent: replication ES 95% confidence interval (CI) excludes 0 and includes original ES point estimate
2) Signal – inconsistent, smaller (same direction): replication ES 95% CI excludes 0 but also excludes original ES point estimate; the replication ES is smaller and in same direction as original ES.
3) No signal – inconsistent: replication ES 95% CI includes 0 but excludes original ES point estimate.
• In the Supplementary Materials, the header still indicates Stage 1.
• Please indicate what is plotted in Figure 1 (in the caption or Figure notes). Same for Figure 2.
• I greatly appreciated the discussion under "Implications, limitations, and directions for future research". It is a very valuable discussion and one of the most valuable lessons to draw from the paper.
Reviewed by Olivier L'Haridon, 19 Jul 2024
2A. Data can test the authors’ proposed hypotheses.
2B. The introduction, rationale and stated hypotheses are consistent with the approved Stage 1 submission.
2C. The authors adhered precisely to the registered study procedures.
2D. Unregistered exploratory analyses are justified, notably based on previous reviewer reports. Though inconclusive, additional analysis on order effects was worth investigating. The event analysis provided informative results. In Table 4, incentives for replication are expressed in £. Harmonization with other tables is needed, at least in a footnote, through, e.g., exchange rates. Figure 2 uses pie charts, which is not the best way to present proportions. A basic table with proportions sounds like a better choice.
2E. The authors’ conclusions are justified, given the evidence. The conclusion could foster the interpretation in terms of ambiguity attitudes for Study 1a Q3, in relation with violations of binary additivity. Similarly, the results for Study 4 suggest a difference between choice-based/revealed preferences procedures and other procedures, that should be pointed out more clearly.
Reviewed by Don Moore, 30 Jul 2024
Download the review
