Assessing the replicability of specific links between numeracy and decision-making

Chris Chambers

doi:10.24072/pci.rr.100376

Assessing the replicability of specific links between numeracy and decision-making

Chris Chambers based on reviews by Elena Rusconi

A recommendation of:

STAGE 2

Revisiting the links between numeracy and decision making: Replication Registered Report of Peters et al. (2006) with an extension examining confidence

Minrui Zhu, Gilad Feldman https://doi.org/10.17605/OSF.IO/4HJCK version 5

Read report on server

Abstract

ZH-CN

Submission: posted 16 January 2023
Recommendation: posted 21 March 2023, validated 23 March 2023

Cite this recommendation as:
Chambers, C. (2023) Assessing the replicability of specific links between numeracy and decision-making. Peer Community in Registered Reports, 100376. 10.24072/pci.rr.100376

This is a stage 2 based on:

Revisiting the links between numeracy and decision making: Replication of Peters et al. (2006) with an extension examining confidence
Minrui Zhu, Gilad Feldman
https://osf.io/8z6ga/

Recommendation

Numeracy – the ability to understand and work with numbers – is associated with a wide range of social and health-related outcomes, including socioeconomic status, employment, literacy, reasoning, and life satisfaction. A substantial body of evidence has also shown links between numeracy and decision-making, prompting the question of how it relates to finer-grained measures of reasoning, judgment and affect/emotion.

In the current study, Zhu and Feldman repeated four influential experiments from a study by Peters et al. (2006), which reported links between numeracy and performance on a variety of decision-making tasks, including attribute framing, frequency-percentage framing, susceptibility to affective influences, and various cognitive biases. The authors also explored several extended questions, including refinements of the original hypotheses and an examination of the relationship between numeracy and confidence in numeric judgments (subjective numeracy).

The results broadly constitute a successful replication, with higher numeracy associated with weaker attribute framing and susceptibility to bias. The relationship between numeracy and the frequency-percentage framing effect – that is, the change in decision-making when numbers are presented as frequencies (e.g. 5 out of 100) rather than percentages (e.g. 5%) – was inconclusive for the main analysis that treated numeracy as a categorical variable (low vs. high); however the link emerged reliably in exploratory analyses that considered numeracy as a continuous variable. The outcomes of the extended analyses were mixed, revealing evidence for a potentially weak relationship between numeracy and confidence.

The Stage 2 manuscript was evaluated over one round of in-depth review. Based on detailed responses to the reviewer's comments, the recommender judged that the manuscript met the Stage 2 criteria and awarded a positive recommendation.

URL to the preregistered Stage 1 protocol: https://osf.io/r73fb

Level of bias control achieved: Level 6. No part of the data or evidence that was used to answer the research question was generated until after IPA.

List of eligible PCI RR-friendly journals:

References

1. Zhu, M. & Feldman, G. (2022). Revisiting the links between numeracy and decision making: Replication Registered Report of Peters et al. (2006) with an extension examining confidence. Acceptance of Version 5 by Peer Community in Registered Reports. https://osf.io/62wqb

2. Peters, E., Västfjäll, D., Slovic, P., Mertz, C. K., Mazzocco, K., & Dickert, S. (2006). Numeracy and decision making. Psychological Science, 17, 407-413. https://doi.org/10.1111%2Fj.1467-9280.2006.01720.x

PDF recommendation

Conflict of interest:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.

Reviews

Evaluation round #1

DOI or URL of the report: https://osf.io/2bme7

Version of the report: 3

Author's Reply, 09 Mar 2023

Download author's reply Download tracked changes file

Revised manuscript: https://osf.io/3nxrq

All revised materials uploaded to: https://osf.io/4hjck/ , updated manuscript under sub-directory "PCIRR-S2 submission following R&R"

Decision by Chris Chambers, posted 02 Mar 2023, validated 02 Mar 2023

One of the original Stage 1 reviewers was available to evaluate your Stage 2 submission, and I have decided that we can proceed on the basis of this assessment and my own reading of the manuscript. As you will see, the reviewer is positive about your completed study, while offering some suggestions for revisions to clarify specific points and correct minor errors. I agree with the reviewer's evaluation and I anticipate being able to accept your manuscript without further review following a round of revision.

Reviewed by Elena Rusconi, 19 Feb 2023

The authors did a very thorough, clear and comprehensive job, the Stage 2 report is to the point, well organised and delivers a clear message. I am happy to recommend publication pending a few minor revisions.

Methods: a typo in the Power Analysis section (between-subjective).

Page 22: consider splitting this sentence into two and clarifying what you mean esp. with “when some studies replicate successful whereas others do not”.

Table 7: Extension dependent variable “How confident are you that you made an accurate assessment of the five students?” pls check (this appears to belong to Table 4)

Page 28: “the target article ran data collection for each of the studies separately using pencil and paper”. I agree this is most likely – it does seem to contradict your previous Table where sample characteristics are reported and “pencil and paper” is only indicated for Study 1.

Results: I find the main deviation from the original plan (i.e. analyses conducted without previously planned exclusions) to be justified and functional to the objective of this work, given that the analyses with exclusions have also been provided.

Page 33: replication, dichotomised numeracy - interaction effect for study 1; after reporting a significant interaction, the authors “concluded support for the hypothesis that the less numerate…” however at this point we do not really know in what direction the interaction is going; it may be useful to provide planned contrasts, or means and stdev for each numeracy group (although this was not done in the target paper), and/or refer to Figure 1 (as in the target paper).

Figure 1: in addition to providing exact p values, it might be useful to provide significance levels with reference to a standard threshold in the legend for better readability (the same observation applies to the following figures)

General evaluative statements about the replication and extension outcomes are repeated throughout the Tables, the Results and Discussion sections - consider limiting these statements to where they are most necessary. The discussion is largely descriptive, concise and coherent with the reported results. Several limitations have been identified, which can be useful for future research.