DOI or URL of the report: https://doi.org/10.31234/osf.io/6xhw4
Version of the report: 2
Hi,
Both reviewers are generally happy with the stage 2, however, before acceptance there are just some minor points from the reviewer relating to presenting of all analyses which I wonder if you could clarify.
Andy
I commend the authors for their scientific rigor and the clarity with which they have conducted and presented their research. The Stage 1 manuscript provided a well-structured and thorough introduction to the study, with clearly articulated aims and a clear description of the study methods and analyses. The Stage 2 manuscript now presents the results and discussion with the same level of detail and transparency. The results are clearly described, align well with the preregistered analysis plan, and are complemented by exploratory analyses that, for the most part, are accompanied by a clear and thoughtful rationale for their inclusion. I was particularly interested to see how the findings addressed the study aims and added to the understanding of the research question. However, I have a few questions and suggestions regarding the exploratory analyses that I believe could further enhance the clarity and impact of the manuscript:
In the section ‘exploratory analyses’ p. 26-27, the authors describe additional analyses that were performed on participants’ performance in the training, which were not preregistered. However, the results section also includes other non-preregistered analyses such as those on the memory tasks and post-training ratings on p.35 under ‘Further exploratory analyses (not pre-registered)’. For clarity and transparency, it would be helpful if the authors would include a description of all conducted non-preregistered exploratory analyses within in the section ‘exploratory analyses’ p. 26-27.
In addition, memory tasks were included to test memory for the stimulus-response contingencies during training. The authors also note that these tasks were included as previous work has shown that memory of stimulus-response contingencies correlated with training effects. However, no such tests are mentioned in the paper in the section ‘Further exploratory analyses (not pre-registered)’ p. 35. It is only indicated that ‘the approach/avoidance group remembered the approach vs. avoidance conditions of items better than the go/no-go group, whereas the go/no-go group remembered the go vs. no-go conditions of items better than the approach/avoidance group.’ While informative, the actual rationale for including these memory tasks seemingly was to test whether contingency awareness correlated with training effects, which is currently not directly tested.
Finally, the discussion is well-balanced, offering a clear and concise summary of the study’s findings while effectively placing them within the context of prior research. The authors thoughtfully discuss their results in light of relevant theoretical frameworks, providing valuable insights into the broader implications of their findings. The authors also pinpoint the study’s limitations along with clear suggestion for further research on this topic.
The manuscript submitted was an interesting and enjoyable read, with a well-defined research question, procedure, and appropriate analysis plan matching what was submitted at Stage 1. Below are my assessments against the Stage 2 criteria, and I would recommend this manuscript is accepted as a Stage 2 Registered Report.
2A. Whether the data are able to test the authors’ proposed hypotheses (or answer the proposed research question) by passing the approved outcome-neutral criteria, such as absence of floor and ceiling effects or success of positive controls or other quality checks.
This criteria has been met, with pre-registered sample sizes achieved after exclusions.
2B. Whether the introduction, rationale and stated hypotheses (where applicable) are the same as the approved Stage 1 submission.
This criterion has been met.
2C. Whether the authors adhered precisely to the registered study procedures.
This criterion has been met, with the authors providing reasonable clarifications to pre-registered analyses. In one place they deviated from their registered analysis (removing random slopes from Bayesian ANOVAs), though this was justified.
2D. Where applicable, whether any unregistered exploratory analyses are justified, methodologically sound, and informative.
This criterion has been met, with unregistered analyses clearly labelled, and the authors have not relied on exploratory analyses when drawing their conclusions.
2E. Whether the authors’ conclusions are justified given the evidence.
The authors have made sound conclusions that are justified from the procedure and results achieved.
Alexander MacLellan