Recommendation
I am happy that the authors have made a robust response to each of the comments I made. We are now at a stage where whatever disagreements or misinterpretations remain are a matter of opinion rather than material concern. Thus, it is appropriate I think to congratulate the authors for the work they have put in to get this pre-registered study into good shape, run the study and see what happens!
First, my full and sincere apologies for the delay in responding/reviewing. It's been a particularly hectic period but I appreciate that's my (persistent) problem.
I read through the response letter and associated revisions in the draft. I'm glad the authors found the comments useful and as far as I'm concerned, the authors have addressed my comments satisfactorily. I'm also happy to roll with the notion that some issues/outcomes are unknowable until the data come in.
I have no further substantive comments at this time. I wish the authors the very best of luck - and I'm very interested to see what they find in this novel study.
DOI or URL of the report: https://osf.io/2zn8r/?view_only=b2148cce80c5430baef8b31fc3ede111
Version of the report: https://osf.io/2zn8r/?view_only=b2148cce80c5430baef8b31fc3ede111
, posted 12 Mar 2022Dear David
I now have two expert reviews in, and I am very pleased with their thoughtful and constructive engagement, in the collaborative spirit that RRs help. Both reviewers are excited by an ambitious attempt to deal with a complex and important question. But both feel serious reworking is needed to enable the study to achieve its goals. The issues are partly methodological, whether the precise procedure is the best one to use; partly conceptual, thinking about what predictions are well motivated. Please give a detailed point by point response to each of their issues in your reply.
I very much look forward to seeing your revision,
best
Zoltan
DOI or URL of the report: https://osf.io/2zn8r/?view_only=b2148cce80c5430baef8b31fc3ede111
Version of the report: https://osf.io/2zn8r/?view_only=b2148cce80c5430baef8b31fc3ede111
Dear Prof. Dienes,
Many thanks for your suggestions we have now edited the manuscript accordingly. We have phrased the language to imply estimation of potential effects. Additionally, we have identified the specific effects that will examine our hypotheses. The Analysis Plan and Table 1 reflect the changes. Apologies for the tardiness in returning with this revision.
Sincerely,
Authors
, posted 21 Jan 2022Dear Dr Neequaye
Thank you for your revision. PCI RR has no power requirements; it is fine to calculate power for that minimally interesting effect which you have appropraitely identified, and indicate the power is too low for a non-signfiicant result to be taken as grounds for assserting Ho. That is just a matter of being explicit about what follows from a given inferential procedure. So do state the power which you do have for each effect of interest (if you are testing hypotheses).
Bear in mind that different journals, including the PCI RR friendly ones, that you may (or may not) wish to publish in afterwards sometimes do have power requirements, if power is used (though Royal Society Open Science does not, so that would be a possiblity). But an alternative, as you say, is not to phrase your study in terms of hypothesis testing at all, but estimation. In that case your goals would not be stated as establishing whether or not there is an effect, and all such language would have to be carefully avoided; the goal is to estimate relevant effects with say a confidence or credibility interval (or similar). The hypotheses could be phrased as implying a certain interaction etc is relevant (not present or absent, that language would be avoided). In fact, your phrasing is largely of the estimation type "to what extent", except for row 1 coloumn 1 of the Design table. You could change this, then the important results are, as I say, the confidence (etc) intervals for each effect of interest. In that case, power is not relevant, but you should indicate the expected precision, or width of the CI, that your N should provide.
Whichever way you go on that, thanks for adding the table; but each hypothesis is related in the table to a set of effects. This allows inferential flexibility. Can you identify a specific question with a specific effect?
In sum the changes needed for me to send out to review are minor, it is just a matter of being a bit clearer about your inferential goals and capabilities.
best
Zoltan
DOI or URL of the report: https://osf.io/2zn8r/?view_only=b2148cce80c5430baef8b31fc3ede111
, posted 06 Jan 2022Dear Dr Neequaye
Thank you for your submission "Exploring How Members of Illicit Networks Navigate Investigative Interviews." Before I send for review, there are a few issues to address:
1) Include a Study Design Template in the main manuscript. See here for details: https://rr.peercommunityin.org/help/guide_for_authors#h_27513965735331613309625021
This involves creating a table aligning specific theoretical questions with hypotheses with the specific statistical test that will test that hypothesis, and the conditions under which support will be inferred for the hypthesis, or rather the results will actively count against the hypothesis.
2) Make sure you remove all analytic flexibility from the analysis plan. Thus, be clear how convergence will be assessed, precisely when it will be decided to not have not occurred; precisely when parameters, like random slopes, will be dropped, etc.
3) Use an inferential procedure that can provide justification for the claim there was not an effect, i.e. the data counted against a hypothesis predicting an effect. For an overview, see https://psyarxiv.com/yc7s5/.
4) In the light of 3), indicate how your N was determined, and what properties this gives your statistical procedure in terms of the likelihood of results counting for or against your hypothesis (e.g. power if you are using frequentist tests; see previously cited article for overview).
Let me know if there are any questions about how to proceed on any of these matters.
best
Zoltan