Dear authors

Your submission to PCI:RR has now been reviewed by two experts in the field. Both were impressed by your plan and agree that this is a worthwhile and timely study to be conducted. However, there are various open points in need of clarification and providing details that could ensure replicability. We therefore invite you to submit a revision. Please include a response letter where you address each comment by reviewers point-by-point (including the more responses to more general RR questions that one reviewer made). To facilitate a quick turnaround, please also include a version with changes tracked/highlighted. Also please ensure that the link on your submission directly leads to the manuscript. This can be the version with highlighted changes - upon in-principle acceptance of the Stage 1 manuscript we the highlights can be removed.

One note about the reviewer comments. For clarity, it is fine to explicitly state if some analyses are not planned. However, please do not include any description of any exploratory analyses for which you will not have a detailed preregistered plan. Exploratory analyses can always be added at Stage 2, provided they are explicitly labelled as such, but they are not part of the Stage 1 protocol.

Best wishes
Sam Schwarzkopf

Dear authors

We regularly triage Stage 1 submissions before sending them out to expert reviewers to ensure various criteria for RRs are met. Your submission is already in a great shape but there are a several smaller issues that I thought merit fixing to avoid confusing reviewers.

OSF link

Please ensure that when you submit that the OSF link points to the manuscript directly, not the general OSF project. If you change or update the manuscript, it will update the link so the link may then be broken. This issue occurred in your previous submission - our team was able to salvage the correct link but this was only by luck. Please ensure that the link to the manuscript works and points to the latest version when you submit.

Statements precluding outcome
Your manuscript is somewhat unusual for a Stage 1 RR in that there are several statements that seem to preclude the outcome. In fact, you have a whole Discussion and Conclusions section. These are fine because they can be replaced at Stage 2 (only Intro and Methods and Design is set at Stage 1). However, the second-to-last sentence in the Introduction also could be seen as precluding the outcome: "Furthermore, we argue that the initial QA/QC on unprocessed data of neuroimaging studies must be critically carried out before defacing to avoid these biases".

I realise that this is based on your pilot data and that you have a strong expectation that you will confirm those earlier results. Nevertheless, the results should not yet be known at this stage. Based on your description currently I judge the bias control level of this project to have a relative high risk Level 3 or 4 (see section 2.6 in the Guide for Authors) but your plan to use blinded, randomised rating should help mitigate this. Nevertheless, I advise you to be more circumspect in your expectations. You can certainly describe your expectations but in a way that requires no further changes to the Intro at Stage 2 if your results show the opposite.

Why only 3T data?
You say you will only use the 3T for the manual rating. There are probably good reasons for that but I would suggest explaining them.

Hypotheses 1 and 2
To my reading, the first two hypotheses are really part of the same. In RRs it is particularly useful to condense the preregistered plan down to the simplest statistical comparison (1-df test) necessary to answer the research question. In your case this seems to be a one-tailed paired t-test or non-parametric alternative on ratings between defacing statuses, plus your Bland-Aldman plots. Is the ANOVA/LMM analysis in Hypothesis 1 adding anything to that? If so, please explain.

Inconsistent power analysis
For a project like this, determining the minimal effect size for a prespecified power and alpha level makes sense. However, this seems to be inconsistently applied. For example, Figures 3 and 6 mention an alpha=0.02 but in the text and the Design Table the same power analyses are described as alpha=0.05. Moreover, it would be worth mentioning the power level in the text, not only the figure captions. Note that some RR-friendly journals expect an alpha=0.02 - if you plan to submit your final Stage 2 manuscript to one of these journals this is indeed the threshold you should set.

Minor issues

• In first paragraph of Introduction: "...the ears themselves." The "themselves" doesn't seem to make sense to me (but I may be wrong, in which case ignore this comment)
• Figure 4: when describing the 95% confidence intervals I assume you mean "dotted" not "dashed" lines (the latter are the means)?
• Typo in Design Table, Hypothesis 1, Question: "bias" instead of "biases"
• Also in Design Table, all cells of Rationale column: reported "in" Figure

Sam Schwarzkopf