DOI or URL of the report: https://osf.io/gfjsy
Version of the report: 1
Dear authors,
Many thanks for submitting your Stage 2 report for review. Please find attached the 3 reviewer opinions of your Stage 2 report, which are generally favourable.
Reviewers 1 & 3 have made a few minor points that should be reflected in your re-submission.
Reviewer 2 makes a stronger point around the interpretation of your non-significant EEG results. After consideration, and re-review of your Stage 1 plan, I think the difference lies in the precise terminology that was accepted at Stage 1. You had stated in the pre-registration that "no definitive conclusions will be drawn from a non-significant result", and I think on balance that this does allow for some interpretation of the observed result compared to if the pre-registration had stated that no conclusions would be drawn at all. Whether this should have, in hindsight, been tightened up at Stage 1 is an open question, but I do believe that your interpretations fit within the previously accepted plan.
Best wishes,
Gemma Learmonth
The stage 2 manuscript accurately follows the outlines of the stage 1 manuscript. The results have shown primarily negative findings, indicating a lack of a relationship between phase-locked responses and ongoing oscillations on the one hand and pain perception on the other hand. This conclusion is hampered by the limited sample size and failure to induce pain in most stimulation conditions.
The manuscript presents the results clearly and discusses the findings appropriately. Some clarifications and added details might further improve the manuscript:
1. P.6, first line. Remove “be”.
2. P. 6, third paragraph. Why were the stimulation temperatures changed compared to the temperatures specified at stage 1?
3. P. 6, third paragraph. Replace “will be” with “were”
4. P. 8, line 6. Remove “be”
5. P. 10, second paragraph. “the resulting amplitude was multiplied by the number of averaged chunks.” Multiplied or divided by the number of chunks?
6. P. 24, first paragraph. “neither the expectation of a similar stimulus nor the mismatch in perception for condition HM seemed to have an influence on the recorded amplitude.” This sentence was not clear to me. Please rephrase.
7. P. 25, last paragraph. “the smallest possible effect size that we would still be interested in.” What would be the smallest effect size the authors would be interested in? And what were the criteria for defining this effect size?
8. P. 26, first paragraph. The study's main result is a negative finding that cannot be conclusively interpreted. This somehow disappointing outcome is mainly due to the limited sample size. The authors might discuss their sample size calculation and lessons learned for future studies more critically and openly.
The authors have conducted the analyses they said they would; though I now note that it slipped us all by, that the precise analyses were not absolutely nailed down beforehand; for example, exactly how post hoc tests would be performed appears not to be pre-registered. However, whatever extra flexibility snuck through, the key theoretical finding was non-significant. So nothing need be done about this point.
My main point is that the pre-registration declares no conclusion follows from nonsignficant results for the EEG. The way the authors deal with this is make conclusions but add a paragraph saying do not take them seriously. That is to write in contradictions. The correct thing to do is draw no conclusions in the first place - a point which applies to the abstract as well. The discussion and summary of results in the abstract need a major re-write therefore. I realise the authors may wonder what to write about. They could say "whatever the difference is between HM and LM it lies in this interval" and give a confidence interval, and declare no conclusion can be made yet as to whether or not there is a difference of scientific relevance.