Thank you for the revised version of this Stage 1 RR, and your careful responses to reviewer comments. I am sufficiently satisfied with these that I do not think it necessary to involve the reviewers again at this point. However, there are still some minor issues to be sorted out fully before IPA is issued for this study, and I list these below.

1) There is still ambiguity about how the outcomes of the different hypothesis tests will be combined to inform conclusions with respect to the overarching theoretical question (LPP vs HPP). What follows may seem like nitpicking, but it is crucially important to be clear up front about how your conclusions will follow from your results.

At line 145 you state the overall rationale for the study as follows:

“If prior expectations are weaker in VR (LPP account), the magnitude of both the SWI and the MWI will be smaller in VR compared to the real world (see hypotheses H1A and H1B in table of questions). Additionally, the difference in peak grip force and load force rates between small and large objects (SWI), or more and less dense-looking objects (MWI), will be smaller in VR than in the real world (see hypotheses H2A and H2B in table of questions).”

What you seem to state here is that H1a AND H1b AND H2a AND H2b all test the same hypothesis (and the final column of your design table implies the same). If this is the case, then the implication would be that they must all have significant outcomes (in the same direction) for the hypothesis (LPP or HPP) to be supported. If so, then the hypothesis would be confirmed by the congruent conjunction of all four outcomes, and not by any other set of outcomes. This may or may not be what you mean to state, but it is what I would infer from what is written.

If this is not what you mean to state, then you will need to specify how you would interpret a set of results that provided partial support for one hypothesis (i.e. that had some outcomes significant but not others), or where different outcomes supported different hypotheses (LPP vs HPP). At present, it is not clear how you will draw conclusions across the pattern of results. This arises because all hypothesis tests are related back to the same global hypotheses.

2) This uncertain status is compounded by the design table, in which you state:

“NOTE: these hypotheses for the SWI and MWI tasks are being treated as individual hypotheses that are related to the same question¬¬, rather than employing a disjunctive or conjunctive logic (Rubin, 2021).

There is some linguistic ambiguity here. Either they are individual (i.e. separable) hypotheses, in which case they lead to conclusions on separate questions, or they relate to the same question (in which case they are two tests of the same hypothesis). The word ‘relate’ may be intended to mean only that there is a thematic relationship between the separable questions (e.g. LPP vs HPP for SWI; LPP vs HPP for MWI), but you need to be clear about whether the tests can lead to separate conclusions or not.

3) In the design table, in several places, you make statements about the conclusions that will follow from non-significant outcomes, e.g. “No statistically significant difference between conditions would indicate no difference in strength of prior expectations.”; “No statistically significant difference in pGFRdiff scores would indicate no difference in strength of prior expectations.” Etc. In NHST, a failure to find a significant difference does not allow one to accept the null hypothesis (only to fail to reject it), and so you should remove these statements.

4) You are very clear that the main hypothesis tests will not be meaningful if manipulation checks are not passed. I would advise you to state which checks apply to which tests (e.g. failing a manipulation check for the MWI would presumably not stop you testing hypotheses relating to the SWI). Also, if a manipulation check is failed, so that a hypothesis test is deemed uninformative, does this mean that you will not run that hypothesis test? If so, state this. If you will still run it, then state why (given that it will be uninformative).

5) In the Abstract, you have the statement: “hypothesis posits increased reliance on predictions relative to current sensory information due to sensory uncertainty.” I think you should probably delete the “due to sensory uncertainty”, which wrongly implies that the cause of a relative change in weighting could be known in this experiment.

6) “windsorised” >> “winsorised”

Thank you for submitting this revised Stage 1 plan, which has now received external review from two relevant experts. I think that their reviews are very helpful, both at a theoretical and practical level, and that consideration of these comments should help you clarify the Stage 1 plan.

To my reading (although I could be wrong), the reviewers make very similar points regarding the framing of your hypotheses, which mean that - at least with the present design - your experiment is only able to inform about the relative influence of priors vs sensory information. You should clarify what conclusions your hypothesis tests can and cannot support, and perhaps re-name the hypotheses if this will help to avoid confusion.

Also, Reviewer#1 was somewhat confused by the status of the exploratory questions, which seems to have resulted from your removing these from the manuscript (as suggested at the previous round), but keeping the detailed design table as supplementary material. I understand the desire to keep these design components online, but it will be much clearer to pre-register only the primary hypothesis tests, and to save exploratory investigations until Stage 2 (keeping the design table for exploratory parts as supplemental material occupies an uneasy half-way house).

I hope that the reviewers' thoughtful comments will be helpful for you in fine-tuning your plans. If you choose to address these comments in a revision, then you should include a responses document that specifies how you have addressed each comment.

Best wishes,

Rob

Thank you for submitting your Stage 1 manuscript to PCI-RR. As the recommender assigned to this manuscript, it is my role to perform an initial triage assessment to determine whether the submission is ready to be sent for external review. This assessment is primarily with respect to the RR aspects of the proposal, rather than its specific topical content. On the basis of this initial assessment, I would say that the manuscript is generally very well prepared, but that there are a number of specific points that you should consider before it sent for external review. These all relate to the analysis plan/design table, which is what I focused my assessment on.

1. It is great that you have included manipulation checks, but you must be explicit, in each case, about how your conclusions would be affected if the check were failed. Normally, a manipulation check would be to confirm an effect that would be expected if the task is working as intended, or is otherwise a necessary precondition for your experiment to be deemed capable of testing the experimental hypotheses of interest. If your manipulation check has this status, then it should be made explicit (your first manipulation check seems like it would probably have this status, but I am less sure for 2-4). If it does not have this status, then it is not an important manipulation check, and should probably be omitted.

2. Manipulation check 1 has two parts that support two different conclusions (about the SWI and MWI respectively). It should therefore probably be split into two (perhaps labelled 1a and 1b). In general, a separate hypothesis should be defined for each test that supports a separate conclusion.

3. As noted, you should consider whether manipulation checks 2-4 are necessary. Is it essential that there are no differences here (and would any differences invalidate or qualify the conclusions you can draw from your main experimental hypothesis tests)? If so, then why is it considered adequate to have 90% power only to detect a rather large effect size (dz = 0.66). If it is indeed necessary to confirm no differences then you should frame an equivalence test, rather than just failing to reject the null (or use a Bayesian approach). Also, since the verbal statement concerns ‘general differences’ between real and VR, it seems like you should have a simple independent t-test (which tests the overall difference between conditions), rather than an ANOVA (which would be sensitive to any pattern of differences between means).

4. Although manipulation checks are not your experimental hypotheses, it would still be conventional to label them as hypotheses (e.g. H1). You can still contextualise as a manipulation check, explaining how your experiment is affected if the hypothesis is not confirmed.

5. The targeted effect sizes for your experimental hypotheses reflect mean estimates from prior research. Unless these prior studies were registered reports, then it is likely that these are over-estimates of the true effect (indeed, the very fact that you are choosing to follow up these findings may also mean they are likely to be over-estimates). It might be preferable to target a lower-bound or otherwise conservative effect size. In any case, you need to provide a rationale for why the targeted effect size is appropriate (i.e. that failing to detect an effect of this size would be an important message for the field).

6. H2a, H2b: A conventionally ‘medium’ correlation is selected as the smallest effect size of interest, but no rationale is provided for why this is a theoretically or practically relevant SEOSI.

7. For some hypotheses, the associated theoretical conclusion will be drawn if the associated test is significant for either SWI or MWI. This ‘disjunction’ logic (X if either Y or Z) probably requires alpha adjustment (e.g. Rubin, 2021. https://doi.org/10.1007/s11229-021-03276-4).

8. Exploratory research questions should not be included in the Stage 1 plan, but can be added at Stage 2. This would also help to simplify the current Stage 1 manuscript.

9. In the Data treatment section of the main text, it is stated that “Data will be checked for extreme deviations from normality based on skewness and kurtosis scores. Assuming data adhere to these assumptions the tests outlined in the table of questions will be run. Non-parametric alternatives will be used if data deviate substantially from normality.” You need to provide precise statements of what will constitute sufficient deviations for you to switch analysis strategy, and you also need to state what the non-parametric alternative will be. Note that the above assumptions are not necessary for your regression analyses (which assumes normality of residuals).

10. In the Data treatment section of the main text, it is also stated that “Bayes factors using a symmetric Cauchy prior will also be used to quantifying the strength of evidence for the alternative and null hypotheses.” These do not feature in the design table, so I assume they play not role in your inferential logic. It is OK to mention that these BFs will be calculated, if it is your plan, but you should make it clear that your main conclusions will be driven only by the outcomes of the tests specified in the design table. If the purpose is to try to quantify evidence for null results, and you regard this as important for your conclusion, then perhaps a Bayesian approach would be more appropriate as your main strategy (or you could include some frequentist tests of equivalence).

====

December 2023 submission closure: Please note that to accommodate reviewer and recommender holiday schedules, PCI RR will be closed to ALL submissions from 1st December 2023 until 10th January 2024. During this time, reviewers will remain able to submit reviews, and recommenders can issue decision letters for ongoing submissions, but no new or revised submissions can be made by authors.