Close printable page
Recommendation

A Registered Report demonstration that the SNARC effect depends on absolute as well as relative number magnitude

ORCID_LOGO based on reviews by Claudia Gianelli
A recommendation of:

One and only SNARC? Spatial-Numerical Associations are not fully flexible and depend on both relative and absolute magnitude

Abstract

EN
AR
ES
FR
HI
JA
PT
RU
ZH-CN
Submission: posted 10 June 2024
Recommendation: posted 30 August 2024, validated 06 September 2024
Cite this recommendation as:
McIntosh, R. (2024) A Registered Report demonstration that the SNARC effect depends on absolute as well as relative number magnitude. Peer Community in Registered Reports, 100808. 10.24072/pci.rr.100808

This is a stage 2 based on:

One and only SNARC? A Registered Report on the SNARC Effect’s Range Dependency
Lilly Roth, John Caffier, Ulf-Dietrich Reips, Hans-Christoph Nuerk, Krzysztof Cipora
https://osf.io/gr94f

Recommendation

The Spatial-Numerical Association of Response Codes (SNARC) effect refers to the fact that smaller numbers receive faster responses with the left hand, and larger numbers with the right hand (Dehaene et al., 1993). This robust finding implies that numbers are associated with space, being represented on a mental number line that progresses from left to right. The SNARC effect is held to depend on relative number magnitude, with the mental number line dynamically adjusting to the numerical range used in a given context. This characterisation is based on significant effects of relative number magnitude, with no significant influence of absolute number magnitude. However, a failure to reject the null hypothesis is not firm evidence for the absence of an effect. In this Registered Report, Roth and colleagues (2024) report two large-sample online experiments, with a Bayesian statistical approach to confirm—or refute—a role for absolute number magnitude in modulating the classic SNARC effect (smallest effect size of interest, d = 0.15).
 
Experiment 1 closely followed Dehaene’s (1993) original methods, and found strong evidence for an influence of relative magnitude, and moderate-to-strong evidence against an influence of absolute magnitude. Experiment 2 was designed to exclude some potential confounds in the original method, and this second experiment found strong evidence for both relative and absolute magnitude effects, of comparable effect sizes (in the range of d = .24 to .42). This registered study demonstrates that the SNARC effect is not ‘fully flexible’, in the sense of depending only on relative number magnitude; it is also shaped by absolute magnitude.
 
This Stage 2 manuscript was evaluated by the recommender and one external reviewer. Following appropriate minor revisions, the recommender judged that the manuscript met the Stage 2 criteria for recommendation.
 
URL to the preregistered Stage 1 protocol: https://osf.io/ae2c8
 
Level of bias control achieved: Level 6. No part of the data or evidence that was used to answer the research question was generated until after IPA.
 
List of eligible PCI RR-friendly journals:
 
 
References
 
1. Dehaene, S., Bossini, S., & Giraux, P. (1993). The mental representation of parity and number magnitude. Journal of Experimental Psychology: General, 122, 371–396. https://doi.org/10.1037/0096-3445.122.3.371
 
2. Roth, L., Caffier, J., Reips, U.-D., Nuerk, H.-C., Overlander, A. T. & Cipora, K. (2023). One and only SNARC? Spatial-Numerical Associations are not fully flexible and depend on both relative and absolute magnitude [Stage 2]. Acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/epnd4
PDF recommendation pdf
Conflict of interest:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.

Reviews

Evaluation round #2

DOI or URL of the report: https://osf.io/nwjhf

Version of the report: 2

Author's Reply, 30 Aug 2024

Decision by ORCID_LOGO, posted 20 Aug 2024, validated 22 Aug 2024

I have now had a chance to examine the revisions made to this paper, and I think they are generally fine. It is very good also that you spotted the error in the Bayesian analysis, which myself and the reviewer had missed (and also reassuring that it has not affected the pattern of outcomes).

I do not think there is a need for further external review, and I am more-or-less ready to recommend this Stage 2 report formally. However, there are two points that I think you should attend to, one of which relates to material newly added at this revision.

1. Figure 5 is helpful, but I think it may be potentially confusing that you have reversed the statement of the hypothesis in (red coloured) cases where there was evidence against the hypothesis. This makes it look (to me) like there was evidence against the reversed statement. I think you would be better to keep the statement of the alternative hypothesis consistent, with the colour of the box (and the tick or cross) symbol indicating whether there was evidence for or against that statement. This in my opinion would be clearer, but you might want to check with some other colleagues what they find most intuitive.

Also, is there a cross missing from the H3 box in Figure 5a? Also for the statement of H3, I think it would be clearer to write "Different SNARC slope" rather than "Different SNARC strength", and even better if you could adjust the end of the arrows (maybe to short lines running parallel to a portion of the fit lines) to graphically indicate that it is slopes (not points) that are being compared.

2. In the abstract you state, "In two highly powered Registered Report online experiments (https://doi.org/10.17605/OSF.IO/AE2C8, IPA by PCI: 12/03/2023), we observed and here report such a dependency on absolute magnitude (in addition to the replication of effects of relative magnitude)."

There are a few issues with this statement.

i) It is misleading to imply "In two experiments we observed a dependency on absolute magnitude". You did not observe such a dependence in Experiment 1, only in Experiment 2. There are good reasons for this, so perhaps you could report your outcomes with more nuance?

ii) It doesn't seem quite right to state that your experiments were "highly powered", because the concept of statistical power does not apply directly in Bayesian statistics. It would be correct to say that the experiments were designed to provide strong evidence for or against relative and absolute magnitude dependency, or some other form of words to this effect.

iii) You don't need to insert the doi to the Stage 1 report in your Abstract.

 

Finally since I am returning this manuscript to you for final minor revisions, I am taking the opportunity to share my draft of the recommendation text for this Stage 2 report. This is the text that will appear on PCI RR site when the report is formally recommended. If you think I have misrepresented anything about your experiment in this draft, then please let me know.

Best wishes,

Rob

 

A Registered Report demonstration that the SNARC effect depends on absolute as well as relative number magnitude

The Spatial-Numerical Association of Response Codes (SNARC) effect refers to the fact that smaller numbers receive faster responses with the left hand, and larger numbers with the right hand (Dehaene et al., 1993). This robust finding implies that numbers are associated with space, being represented on a mental number line that progresses from left to right. The SNARC effect is held to depend on relative number magnitude, with the mental number line dynamically adjusting to the numerical range used in a given context. This characterisation is based on significant effects of relative number magnitude, with no significant influence of absolute number magnitude. However, a failure to reject the null hypothesisis not firm evidence for the absence of an effect. In this Registered Report, Roth and colleagues (2023) report two large sample online experiments, with a Bayesian statistical approach to confirm—or refute—an effect of absolute number magnitude in modulating the classic SNARC effect (smallest effect size of interest, d = 0.15).

Experiment 1 closely followed Dahaene’s (1993) original methods, and found strong evidence for an influence of relative magnitude, and moderate-to-strong evidence against an influence absolute magnitude. However, Experiment 2 was designed to exclude some potential confounds in the original method, and this second experiment found strong evidence for both relative and absolute magnitude effects, of comparable effect sizes (in the range of d = .24 to .42). This registered study demonstrates that the SNARC effect is not ‘fully flexible’, in the sense of depending only on relative magnitude; it is also shaped by absolute number magnitude.

This Stage 2 manuscript was evaluated over one round of in-depth review, by the recommender and one external reviewer. Following appropriate minor revisions, the recommender judged that the manuscript met the Stage 2 criteria for recommendation.

Evaluation round #1

DOI or URL of the report: https://osf.io/e9pdz

Version of the report: v1

Author's Reply, 13 Aug 2024

Decision by ORCID_LOGO, posted 11 Jul 2024, validated 11 Jul 2024

Congratulations on completing this fascinating experiment, and commendations on the very clear and thorough write-up of your findings. The Stage 2 review process for Registered Reports is typically quite light touch, provided that the authors have not modified the Stage 1 components (which you have not), and have followed their pre-registered analysis plan (which you have).

I have received helpful comments from one expert reviewer (Claudia Gianelli), but was not able to get both original Stage 1 reviewers to return. I have read over the Stage 2 report myself, and noted only a few very minor issues to attend to (in addition to considering Claudia Gianelli’s suggestions):

1) In your footnote 2 on page 28 (within Results), I think that you may under-emphasise the importance of the differences in outcomes that follow from the exclusion of numbers 1 and 4 (from the lower and higher ranges respectively). What you have written if perfectly correct, but the reader may miss that the “… did not change substantially…” applies specifically to H1a, and may overlook the later statement that the change made the evidence become inconclusive for H2b and H3. Just a subtle change of wording here would likely avoid such a misreading, and make the reporting of Results more congruent with the discussion of this issue on p44).

2) Your focus on reporting results is on statistical outcomes (BF’s), but it would be very informative to provide estimates of effect size for the RM and AM components of the SNARC, both in terms of raw effect sizes (dRT in ms) and a suitably standardised measure (e.g. Cohen’s d). Within the Discussion, you should pick up upon the relative strength of the RM and AM effects. This would make it clear that the RM effect is massively larger than the AM effect. I do not think this takes away at all from the importance of your finding that AM also matters, but providing a concrete sense of scale (beyond BF’s) would be helpful, and should inform your discussion (and Abstract).

 

3) When you report “strong evidence for a correlation” between SNARC slopes in Experiment 2, it might be helpful to specify, “strong evidence for a modest correlation”, because a reader who was not paying sufficient attention might assume that you mean evidence for a strong correlation. Again, what you have already written is perfectly correct; this is just to guard against a possible misreading.

 

4) When a BF has seven digits before the decimal, it feels like spurious precision to report digits beyond the decimal place. I would be tempted to round such large BF’s to integers. However, I also recognise that 2 dpl is the convention you have used elsewhere and so it is fine to keep it this way if you prefer.

 

Please note that I will be away now until August 5th, but would be pleased to deal with a final version on my return. If you submit it sooner than that, then it may be dealt with by another member of the PCI RR board (which is fine by me too – I do not think these final revisions are in any way problematic).

Best wishes,

 

Rob

 

=====

Note from PCI RR Managing Board: As you will be aware, we are now in July-August shutdown period. During this time, authors are generally unable to submit new or revised submissions. However, given the relatively minor revisions required in your case, we are going to give you the opportunity to resubmit despite the shutdown. You won't be able to do this the usual way. Instead, please email us (at contact@rr.peercommunityin.org) with the following:

 

  1. A response to the reviewer(s) and recommender (attached to the email as a PDF)
  2. The URL to a completely clean version of the revised manuscript on the OSF
  3. The URL to a tracked-changes version of the revised manuscript on the OSF
     

In the subject line of the email please state the submission number (#) and title. We will then submit the revision on your behalf.

Reviewed by , 10 Jul 2024

This is a Stage 2 submission of a previously reviewed Stage 1 registered reports. 

The authors appear to have followed their pre-registration closely without any major deviations, and the results were discussed very cautiously​. This wil make for a solid addition to the SNARC literature. 

I have some suggestions to improve readibility of the manuscript and the associated materials. 

1. OSF repository: the authors have shared materials and raw data as planned. However, at the moment the repository is a bit hard to navigate. It would be helpful to guide the user a bit more and ideally provide a read me file detailing what can be found and where. Also, I know demo versions of both experiments as link in the text, but doing so within OSF would probably be easier to navigate - and again, a txt file describing the technical details of the online experiment would help future implementations/replications. 

2. Reporting of data collection and pre-processing is suboptimal at the moment. The authors list all rejections and state that some participants were rejected for multiple reasons, but it is hard to tell how many and at which stage of pre-processing. 

3. The rationale behind exploratory analyses should be made more transparent to the reader, even the one less familiar with the topic. At the moment, exploratory analyses start right the confirmatory ones quite abruptly. 

4. I would reccomend a summary table guiding the reader to have a visual overview of the observed evidence (in terms of both tested HPs and quality of evidence).