Recommendation
DOI or URL of the report: https://psyarxiv.com/3472y
Version of the report: 2
Dear Dr. Sherman,
Thank you for bringing to our attention the fact that we overlooked this small but indeed crucial detail about the priors. Thank you also for your suggestions and for the link.
We have discussed this among us and settled on using the default JASP settings (JZS prior). Our main analyses (in the context of the #EEGManyLabs project) do focus on frequentist statistics. As such, we want the BFs to be comparable to these results, while still allowing us to obtain evidence for the null. This can be achieved, we believe, by using the JZS prior.
We were not 100% sure of what you meant with "reporting the robustness region". We checked in JASP and the Bayesian paired-sample t test module does include a robustness check option which runs the analysis again with a "wide prior" (Cauchy width = 1.0) and "ultrawide prior" (Cauchy width = 1.4). We assumed that this is what you meant and have therefore decided to also report this robustness check.
We will also upload the .jasp files on our repository when we submit the stage 2 report. These files keep all the analyses and dataset within them, which allow any user to easily load them in JASP and change any settings (such as priors) in any analysis. So readers will easily be able to check whether Bayesian results hold with their prior of choice.
All changes are marked in red in the tracked changes manuscript and are located on page 6.
Kind regards,
Martin Constant and Heinrich Liesefeld
(on behalf of all listed co-authors)
, posted 11 May 2023, validated 11 May 2023Dear Martin Constant, Heinrich Liesefeld, and co-authors
Thank you for your thorough revision, which addresses all the reviewer comments. There is just one very minor issue outstanding, and then I'm very happy to recommend in-principle acceptance:
In the section Methods - Original pipeline, you write:
"In addition to these frequentist t tests, we will additionally perform directed Bayes Factor (BF) t tests with JASP (most recent version available at the time of data analysis; JASP Team, 2023; Love et al., 2019). A BF in favor of the null ≥ 3 or a BF in favor of the alternative ≥ 6 will be considered as sufficient evidence"
The text needs to include the prior you will use (e.g. "BFs will be calculated using the default Cauchy prior of width 0.707").
I'm assuming these will be calculated with the default Cauchy prior with scale parameter 0.7. This corresponds to a Cohen's d of approximately 2 (https://xeniaschmalz.blogspot.com/2019/09/justifying-bayesian-prior-parameters-in.html), which is very large - the predicted effect size is so large that it's relatively "easy" to find sensitive evidence for the null. Ideally the prior would be determined by the effect size of Eimer 1960, but (as was typical at the time) it's not reported, and descriptives aren't given/error bars aren't plotted. So, could you please make one of the following changes -
1. Calculate the approximate standardised effect size from Eimer 1960, estimating the means (eyeballed from the plots) and estimating the SE from the ANOVA F stat and dfs, then use an informed prior (converting cohens d to a scale parameter using the information in that link above). Either a directional cauchy or half-normal prior is fine.
2. Use a standardised effect size from another, related study to inform your prior - doesn't matter whether you use a half-normal or cauchy
3. Use the default cauchy prior of 0.7 (again, one-tailed) and also report the robustness region (I'm 99% sure JASP can give you this). This way, readers can see whether the same effect would be found under their prior of choice
Whichever you choose, could you please describe your choice of prior in the manuscript.
There is a small chance that the bayesian and frequentist results will diverge. It may be worth stating that you will draw your main inferences from the frequentist statistics (assuming that is indeed your preference).
Best wishes
Maxine
DOI or URL of the report: https://psyarxiv.com/3472y
Version of the report: 1
, posted 11 Apr 2023, validated 11 Apr 2023Dear Dr Constant,
Thank you for submission, and please accept my apologies for the delay in getting back to you. Your Stage One submission has received two high-quality reviews from Dr Clayton Hickey and Dr Reny Baykova, and I have provided a third review, which I have appended below.
All reviewers agree that the submission is excellent, and request only very minor points of clarification. Once these have been completed (or rebutted) then I will go ahead recommend the report.
Dr Baykova suggested that Eimer's third hypothesis in the original 1996 paper is also included as a target for replication here - it is not neccessary for you to do this, but if you decide against could you briefly explain why in the Introduction.
Kind regards,
Maxine Sherman
Review: Maxine Sherman
The submission is excellent and I have only two very minor comments.
1) The authors write:
“The most representative result are the effects of contralaterality in Study 2 (which is the replicated study) for electrode pair OL (corresponding to PO7/8 in the 10-10 system) in the time range 220 – 300 ms for form discrimination F(1,9) = 57.10, p < .001 and color discrimination F(1,9) = 17.48, p = .002; thus the smallest of these two F values (17.48) is used to compute the effect size”
Could you clarify what is meant by representative result? Do you mean the result in Experiment 2 that is most representative of the interpretations of the original paper?
2) The authors write that they will report a meta analytic effect size for the original pipeline. Will this also be done for the alternative pipelines? If not, how will the across-labs results from these pipelines be reported, and how (if at all) will they be compared to results from the original pipeline?
Typos
Should be "The N2pc forms appeared"
‘…usually consists in’ should be ‘usually consists of’
result should be results
The protocol is careful and entirely suitable for reproduction of the target study. I see no need for revision.
I have a couple of minor comments on the introduction:
P 2 – ‘Eimer’s finding does not exclude alternative interpretations… might be a composite reflecting both enhancement of the relevant stimulus and suppression of the irrelevant stimulus…’ There is nuance that is missed here. In Hickey et al., 2009, we suggested a couple of alternatives. As described in the RR, we concluded with some confidence that the Pd reflects direct suppression of the neural response to the distractor. We were less sure what computational mechanism the Nt represented. We offered the possibility that it might represent some sort of target enhancement. But we kept open the idea that it might reflect a mechanism of distractor suppression (in addition to the mechanism reflected in Pd). That is, the Nt could reflect activation of cells responsible for the ‘silencing’ of distractor input into cortex responsible for representation of the target (eg. by inhibiting cells in cortical layers receiving lateral or feedback input). This kind of sheltering mechanism would emerge in tissue that is physically proximal to cortex representing the target, and would emerge contralateral to the target (not the distractor).
This idea was broadly motivated by literature emerging at that time on the role of alpha in vision, and on the possibility of GABA-ergic circuits involved in attentional control. One reason to like the sheltering account is that it easily explains why the N2pc is so sensitive to the proximity, presence, and nature of distractor stimuli. It’s also broadly in line with the strong role for distractor suppression that emerges in animal electrophysiology studies of attention. To my knowledge, it remains a valid interpretation.
P 2 – ‘…only few N2pc studies have presented the relevant stimulus without surrounding elements…’ We went even a little further than this in Hilimire, Hickey, & Corballis, 2011, Psychophysiology, and only presented one stim at a time.
Signed, Clayton Hickey