A Registered Report demonstration that the SNARC effect depends on absolute as well as relative number magnitude

I have now had a chance to examine the revisions made to this paper, and I think they are generally fine. It is very good also that you spotted the error in the Bayesian analysis, which myself and the reviewer had missed (and also reassuring that it has not affected the pattern of outcomes).

I do not think there is a need for further external review, and I am more-or-less ready to recommend this Stage 2 report formally. However, there are two points that I think you should attend to, one of which relates to material newly added at this revision.

1. Figure 5 is helpful, but I think it may be potentially confusing that you have reversed the statement of the hypothesis in (red coloured) cases where there was evidence against the hypothesis. This makes it look (to me) like there was evidence against the reversed statement. I think you would be better to keep the statement of the alternative hypothesis consistent, with the colour of the box (and the tick or cross) symbol indicating whether there was evidence for or against that statement. This in my opinion would be clearer, but you might want to check with some other colleagues what they find most intuitive.

Also, is there a cross missing from the H3 box in Figure 5a? Also for the statement of H3, I think it would be clearer to write "Different SNARC slope" rather than "Different SNARC strength", and even better if you could adjust the end of the arrows (maybe to short lines running parallel to a portion of the fit lines) to graphically indicate that it is slopes (not points) that are being compared.

2. In the abstract you state, "In two highly powered Registered Report online experiments (https://doi.org/10.17605/OSF.IO/AE2C8, IPA by PCI: 12/03/2023), we observed and here report such a dependency on absolute magnitude (in addition to the replication of effects of relative magnitude)."

There are a few issues with this statement.

i) It is misleading to imply "In two experiments we observed a dependency on absolute magnitude". You did not observe such a dependence in Experiment 1, only in Experiment 2. There are good reasons for this, so perhaps you could report your outcomes with more nuance?

ii) It doesn't seem quite right to state that your experiments were "highly powered", because the concept of statistical power does not apply directly in Bayesian statistics. It would be correct to say that the experiments were designed to provide strong evidence for or against relative and absolute magnitude dependency, or some other form of words to this effect.

iii) You don't need to insert the doi to the Stage 1 report in your Abstract.

Finally since I am returning this manuscript to you for final minor revisions, I am taking the opportunity to share my draft of the recommendation text for this Stage 2 report. This is the text that will appear on PCI RR site when the report is formally recommended. If you think I have misrepresented anything about your experiment in this draft, then please let me know.

Best wishes,

Rob

A Registered Report demonstration that the SNARC effect depends on absolute as well as relative number magnitude

The Spatial-Numerical Association of Response Codes (SNARC) effect refers to the fact that smaller numbers receive faster responses with the left hand, and larger numbers with the right hand (Dehaene et al., 1993). This robust finding implies that numbers are associated with space, being represented on a mental number line that progresses from left to right. The SNARC effect is held to depend on relative number magnitude, with the mental number line dynamically adjusting to the numerical range used in a given context. This characterisation is based on significant effects of relative number magnitude, with no significant influence of absolute number magnitude. However, a failure to reject the null hypothesisis not firm evidence for the absence of an effect. In this Registered Report, Roth and colleagues (2023) report two large sample online experiments, with a Bayesian statistical approach to confirm—or refute—an effect of absolute number magnitude in modulating the classic SNARC effect (smallest effect size of interest, d = 0.15).

Experiment 1 closely followed Dahaene’s (1993) original methods, and found strong evidence for an influence of relative magnitude, and moderate-to-strong evidence against an influence absolute magnitude. However, Experiment 2 was designed to exclude some potential confounds in the original method, and this second experiment found strong evidence for both relative and absolute magnitude effects, of comparable effect sizes (in the range of d = .24 to .42). This registered study demonstrates that the SNARC effect is not ‘fully flexible’, in the sense of depending only on relative magnitude; it is also shaped by absolute number magnitude.

This Stage 2 manuscript was evaluated over one round of in-depth review, by the recommender and one external reviewer. Following appropriate minor revisions, the recommender judged that the manuscript met the Stage 2 criteria for recommendation.

Congratulations on completing this fascinating experiment, and commendations on the very clear and thorough write-up of your findings. The Stage 2 review process for Registered Reports is typically quite light touch, provided that the authors have not modified the Stage 1 components (which you have not), and have followed their pre-registered analysis plan (which you have).

I have received helpful comments from one expert reviewer (Claudia Gianelli), but was not able to get both original Stage 1 reviewers to return. I have read over the Stage 2 report myself, and noted only a few very minor issues to attend to (in addition to considering Claudia Gianelli’s suggestions):

1) In your footnote 2 on page 28 (within Results), I think that you may under-emphasise the importance of the differences in outcomes that follow from the exclusion of numbers 1 and 4 (from the lower and higher ranges respectively). What you have written if perfectly correct, but the reader may miss that the “… did not change substantially…” applies specifically to H1a, and may overlook the later statement that the change made the evidence become inconclusive for H2b and H3. Just a subtle change of wording here would likely avoid such a misreading, and make the reporting of Results more congruent with the discussion of this issue on p44).

2) Your focus on reporting results is on statistical outcomes (BF’s), but it would be very informative to provide estimates of effect size for the RM and AM components of the SNARC, both in terms of raw effect sizes (dRT in ms) and a suitably standardised measure (e.g. Cohen’s d). Within the Discussion, you should pick up upon the relative strength of the RM and AM effects. This would make it clear that the RM effect is massively larger than the AM effect. I do not think this takes away at all from the importance of your finding that AM also matters, but providing a concrete sense of scale (beyond BF’s) would be helpful, and should inform your discussion (and Abstract).

3) When you report “strong evidence for a correlation” between SNARC slopes in Experiment 2, it might be helpful to specify, “strong evidence for a modest correlation”, because a reader who was not paying sufficient attention might assume that you mean evidence for a strong correlation. Again, what you have already written is perfectly correct; this is just to guard against a possible misreading.

4) When a BF has seven digits before the decimal, it feels like spurious precision to report digits beyond the decimal place. I would be tempted to round such large BF’s to integers. However, I also recognise that 2 dpl is the convention you have used elsewhere and so it is fine to keep it this way if you prefer.

Please note that I will be away now until August 5th, but would be pleased to deal with a final version on my return. If you submit it sooner than that, then it may be dealt with by another member of the PCI RR board (which is fine by me too – I do not think these final revisions are in any way problematic).

Best wishes,

Rob

=====

Note from PCI RR Managing Board: As you will be aware, we are now in July-August shutdown period. During this time, authors are generally unable to submit new or revised submissions. However, given the relatively minor revisions required in your case, we are going to give you the opportunity to resubmit despite the shutdown. You won't be able to do this the usual way. Instead, please email us (at contact@rr.peercommunityin.org) with the following:

1. A response to the reviewer(s) and recommender (attached to the email as a PDF)
2. The URL to a completely clean version of the revised manuscript on the OSF
3. The URL to a tracked-changes version of the revised manuscript on the OSF
    

In the subject line of the email please state the submission number (#) and title. We will then submit the revision on your behalf.