Recommendation
DOI or URL of the report: https://osf.io/preprints/psyarxiv/jwgv3/
Version of the report: 1
, posted 28 Nov 2023, validated 28 Nov 2023I would like to thank the authors for their patience in the review process, and the two expert reviewers who have provided constructive suggestions for this Stage 1 manuscript. As you will see, the reviewers have made some suggestions for improvement prior to data collection. In summary, the reviewers both propose a careful consideration of how best to measure: 1) how unconscious processing can be established, and 2) how to optimally measure whether a priming effect occurs in this design. I agree on these points, and the reviewers have some useful suggestions the authors may wish to consider. My only other comment was one that Dr Tsikandilakis touched on near the end of the review, in terms of considering the potential hurdle of acquiring permission to use the images provided in the appendix. I would suggest stating the strategy that will be taken to make stimuli available in this case.
It was a delight to read this pre-registration. I was happy to see that it came with pilot data, and that all analysis steps were well thought out, following the best guidelines at our disposal to date in the field of consciousness science. This is an example for the field.
I do have some suggestions / recommendations, in particular relating to establishing unconscious processing.
First, instead of testing whether the magnitude of their priming effect is larger than zero, the authors plan to test whether it is larger than the 'priming effect' that would be expected by regression to the mean alone (following Shanks' approach). Thus far, however, I don't recall having seen (m)any effect(s) in the literature convincingly passing this test; although I am happy to be convinced otherwise. I therefore fear that, in this regard, the pre-registered analyses might be too conservative, and wonder whether a null effect would be interpretable as evidence of absence.
Second, from another perspective, the described approach might be too liberal. To replicate the original effect, the researchers would need to (1) establish invisibility of the primes, and (2) observe a significant priming effect. Following the logic of Meyen and colleagues (2022), however, we would succomb to the interaction fallacy were we to interpret this as evidence for unconscious processing; that is, the fact that effect 1 is significant, and effect 2 is not, does not mean that the difference between these effects is significant (the same holds true for a Bayesian approach). Thus, a key analysis to include would be the paired comparison of the direct task with the indirect task (i.e., is the priming effect stronger - in standardized units - than the visibility of the primes?).
> Meyen, S., Zerweck, I. A., Amado, C., von Luxburg, U., & Franz, V. H. (2022). Advancing research on unconscious priming: When can scientists claim an indirect task advantage?. Journal of Experimental Psychology: General, 151(1), 65.
Some very minor points:
In the results of the pilot (page 13) it is clearly stated how many participants are excluded, but not how many are included in the analysis.
It is unclear whether the stastistical tests reported about the pilot data already compare the priming effect to the effect of regression to the mean or compare the priming effect to zero.
Page 21, bottom paragraph states "trails" instead of "trials".
The Study Design Template should probably be rotated by 90 degrees in landscape orientation, because the narrow columns imposed by portrait orientation make it very difficult to read.