DOI or URL of the report: https://psyarxiv.com/bpdn2
Version of the report: Version 2, 02/25/2022 22:10:57
Your clarifications are useful, but a key issue remains regarding analytic flexibility. The reason we ask for the design table is to make sure analytic and inferential flexibility are tied down. Your predictions could be confirmed or disconfirmed by relevant interactions; or by relevant partial interactions or simple effects; or by relevant pairwise comparisons. So what the Design Table needs to include is the key test of each hypothesis. Will you test interactions? Your analysis method involves using credibility intervals; so that necessitates one degree of freedom tests (hence, for example, the use of partial rather than full interactions when it comes to outcomes with more than two categories). Which contrasts will you do precisely? Will you test interactions before moving to simple effects? Under what conditions will you do that? What patterns of results will legitimate what conclusions? How will confidence ratings feed into your conclusions? Do any tests of them consitute a crucial test of any of of your hypotheses?
In sum, for Stage 1 you want to have thought through the precise patterns that will confirm or disconfirm each hypothesis. Any analyses that do not crucially test your listed hypotheses should be left unstated at this time; they can be put in a non-pre-registered subsection of the Results.
DOI or URL of the report: https://psyarxiv.com/bpdn2
Version of the report: Version 2, 02/25/2022 22:10:57
The two reviewers are largely positive about your revision. The first reveiwer asks for clarifcation on a number of issues. A few further points to address:
1) In your Design Table, list each hypothesis as a row, so it is clear exactly which test addresses which hypothesis and what follows from possible outcomes in each case.
2) Be clear about when you will not conclude anything, i.e. when the ROPE lies entirely within the HDI (which may happen if your maximum N is reached before the HDI acquires a smaller width than the ROPE).
3) You refer to the bottom limit of the 95% CI for particular past studies in terms of minimal plausible effects; what if you looked at the bottom limit of the CI for the combined effect from the studies you list? Just a thought. You could go with a 90% CI. Otherwise, it is not clear, as you also indicate, you have really found a minimally interesting plausible effect size. Given that an interesting effect may be smaller than the limits of your ROPE, due caution should then be reflected in the conclusions that follow (e.g. explicitly indicating, including in the abstract, that effects smaller than a certain amount are possible).
Thank you for your clear and concise response to the queries raised. I am happy that my comments/questions have been addressed and look forward to reading your conclusions once the data has been collected. Good luck.
DOI or URL of the report: https://psyarxiv.com/bpdn2
Two reviewers have seen your submission and are generally positive. They suggest some clarifications and some procedural adjustments. One issue they bring up I would like to comment on: While your N has a practical upper limit, you can still show how severely your study tests your hypotheses by simulating with your N how often the HDI would be inside, straddling, or outside the ROPE, given either that a predicted effect sze is true, or that the ROPE is true (see https://psyarxiv.com/yc7s5/) so it is clear how adequate your N is (e.g. is it enough to squeeze an HDI into the ROPE?). As per the last reference you might also see if you can justify more precisely your minimal interesting effect size. While your suggested value seems reasonable given the effect size of the past studies you quote, one previously recommended heuristic (see ref above) is to chose the lower limit of the 95% CI of the effect in relevantly similar studies (i.e. probably the ones you cite).
This is a very thorough consideration of the planned research, and the associated materials and analyses. The research questions make sense considering the theories that are described, and the hypotheses are clearly stated and defined. Based on the study design and analyses, participant responses should provide data which will allow the research question and hypotheses to be answered. The procedural details, and the materials in the appendix, are especially detailed, and will easily allow replication by an expert. This assumes that an example question is stated. The analysis is stated step-by-step with no obvious gaps (or flexibility) in relation to the tests that should be run. The only caveat is that whilst I am familiar with most of the proposed analyses, there are elements that I am not familiar enough with to comment (Priors and ROPE). There is clear evidence of what evidence will, or will not, support the hypotheses. There are manipulation checks incorporated in the study design, and attention checks to identify participants who are not fully attended to the task. Finally the report offers a novel study, with an interesting approach that I am looking forward to reading once the research is published.
There are just a couple of details that I would like to raise:
(1) The report states that resource constraints dictated the sample size. Whilst I completely understand this, it might be helpful to indicate how far away from an ideal sample size the proposed 150 (x2) sample is. If the difference is minimal, then this can be mentioned as a rationale. If there is a big difference then this really needs to be justified
(2) I am slightly unsure about the claim that there should not be a response preference in the Cooperative/low-worthwhileness conditions.e.g. hypothesis 1b and 2b. Having read the low-worthwhile questions for the first time, and assuming that I am a cooperative interviewee, I would have tended to choose the complete response option each time. This could be my own personal disposition, but this might be worth thinking about when interpreting the data.
(3) There are a couple of language/spelling errors in the materials that will need to be addressed .e.g. under confidence measures via bets (Appendix, pg.6).
(4) Apologies if I missed this, but I presume that when asking participants about what % of their compensation they are willing to bet, that this is not an actual bet. How is this explained to participants, and how might this effect their responses. If this is an actual bet, then this has potential ethical implications.