How an interviewee knows what information is key to disclose or withhold
How Intelligence Interviewees Mentally Identify Relevant Information
Abstract
Recommendation: posted 21 October 2022
Dienes, Z. (2022) How an interviewee knows what information is key to disclose or withhold. Peer Community in Registered Reports, . https://rr.peercommunityin.org/PCIRegisteredReports/articles/rec?id=188
Related stage 2 preprints:
David A. Neequaye & Alexandra Lorson
https://doi.org/10.31234/osf.io/bpdn2
Recommendation
In this study, Neequaye and Lorson will ask subjects to take the role of an informant about a criminal gang, with the further instructions to be cooperative or resistant in helping the interviewer obtain the information they want. In one study, the participants will be asked merely to identify what information the interviewer wants. In the second study, the participants will answer the interviewer's questions, disclosing whatever information they feel best suits their interest. Crucially, the level of detail of the questions will be manipulated, such that the question specifies a clear objective or not.
The Stage 1 manuscript was evaluated over three rounds of in-depth review. Based on detailed responses to the reviewers’ comments and edits to the stage 1 report, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).
URL to the preregistered Stage 1 protocol: https://osf.io/82qtn
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.
List of eligible PCI RR-friendly journals:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.
Evaluation round #3
DOI or URL of the report: https://psyarxiv.com/bpdn2
Version of the report: Version 2, 02/25/2022 22:10:57
Author's Reply, 12 Oct 2022
Decision by Zoltan Dienes, posted 16 Sep 2022
Your clarifications are useful, but a key issue remains regarding analytic flexibility. The reason we ask for the design table is to make sure analytic and inferential flexibility are tied down. Your predictions could be confirmed or disconfirmed by relevant interactions; or by relevant partial interactions or simple effects; or by relevant pairwise comparisons. So what the Design Table needs to include is the key test of each hypothesis. Will you test interactions? Your analysis method involves using credibility intervals; so that necessitates one degree of freedom tests (hence, for example, the use of partial rather than full interactions when it comes to outcomes with more than two categories). Which contrasts will you do precisely? Will you test interactions before moving to simple effects? Under what conditions will you do that? What patterns of results will legitimate what conclusions? How will confidence ratings feed into your conclusions? Do any tests of them consitute a crucial test of any of of your hypotheses?
In sum, for Stage 1 you want to have thought through the precise patterns that will confirm or disconfirm each hypothesis. Any analyses that do not crucially test your listed hypotheses should be left unstated at this time; they can be put in a non-pre-registered subsection of the Results.
Evaluation round #2
DOI or URL of the report: https://psyarxiv.com/bpdn2
Version of the report: Version 2, 02/25/2022 22:10:57
Author's Reply, 06 Sep 2022
Decision by Zoltan Dienes, posted 29 Aug 2022, validated 21 Oct 2022
The two reviewers are largely positive about your revision. The first reveiwer asks for clarifcation on a number of issues. A few further points to address:
1) In your Design Table, list each hypothesis as a row, so it is clear exactly which test addresses which hypothesis and what follows from possible outcomes in each case.
2) Be clear about when you will not conclude anything, i.e. when the ROPE lies entirely within the HDI (which may happen if your maximum N is reached before the HDI acquires a smaller width than the ROPE).
3) You refer to the bottom limit of the 95% CI for particular past studies in terms of minimal plausible effects; what if you looked at the bottom limit of the CI for the combined effect from the studies you list? Just a thought. You could go with a 90% CI. Otherwise, it is not clear, as you also indicate, you have really found a minimally interesting plausible effect size. Given that an interesting effect may be smaller than the limits of your ROPE, due caution should then be reflected in the conclusions that follow (e.g. explicitly indicating, including in the abstract, that effects smaller than a certain amount are possible).
Reviewed by anonymous reviewer 1, 15 Aug 2022
Reviewed by anonymous reviewer 2, 26 Aug 2022
Thank you for your clear and concise response to the queries raised. I am happy that my comments/questions have been addressed and look forward to reading your conclusions once the data has been collected. Good luck.
Evaluation round #1
DOI or URL of the report: https://psyarxiv.com/bpdn2
Author's Reply, 21 Jul 2022
Decision by Zoltan Dienes, posted 24 May 2022
Two reviewers have seen your submission and are generally positive. They suggest some clarifications and some procedural adjustments. One issue they bring up I would like to comment on: While your N has a practical upper limit, you can still show how severely your study tests your hypotheses by simulating with your N how often the HDI would be inside, straddling, or outside the ROPE, given either that a predicted effect sze is true, or that the ROPE is true (see https://psyarxiv.com/yc7s5/) so it is clear how adequate your N is (e.g. is it enough to squeeze an HDI into the ROPE?). As per the last reference you might also see if you can justify more precisely your minimal interesting effect size. While your suggested value seems reasonable given the effect size of the past studies you quote, one previously recommended heuristic (see ref above) is to chose the lower limit of the 95% CI of the effect in relevantly similar studies (i.e. probably the ones you cite).
Reviewed by anonymous reviewer 1, 11 Apr 2022
Reviewed by anonymous reviewer 2, 23 May 2022
This is a very thorough consideration of the planned research, and the associated materials and analyses. The research questions make sense considering the theories that are described, and the hypotheses are clearly stated and defined. Based on the study design and analyses, participant responses should provide data which will allow the research question and hypotheses to be answered. The procedural details, and the materials in the appendix, are especially detailed, and will easily allow replication by an expert. This assumes that an example question is stated. The analysis is stated step-by-step with no obvious gaps (or flexibility) in relation to the tests that should be run. The only caveat is that whilst I am familiar with most of the proposed analyses, there are elements that I am not familiar enough with to comment (Priors and ROPE). There is clear evidence of what evidence will, or will not, support the hypotheses. There are manipulation checks incorporated in the study design, and attention checks to identify participants who are not fully attended to the task. Finally the report offers a novel study, with an interesting approach that I am looking forward to reading once the research is published.
There are just a couple of details that I would like to raise:
(1) The report states that resource constraints dictated the sample size. Whilst I completely understand this, it might be helpful to indicate how far away from an ideal sample size the proposed 150 (x2) sample is. If the difference is minimal, then this can be mentioned as a rationale. If there is a big difference then this really needs to be justified
(2) I am slightly unsure about the claim that there should not be a response preference in the Cooperative/low-worthwhileness conditions.e.g. hypothesis 1b and 2b. Having read the low-worthwhile questions for the first time, and assuming that I am a cooperative interviewee, I would have tended to choose the complete response option each time. This could be my own personal disposition, but this might be worth thinking about when interpreting the data.
(3) There are a couple of language/spelling errors in the materials that will need to be addressed .e.g. under confidence measures via bets (Appendix, pg.6).
(4) Apologies if I missed this, but I presume that when asking participants about what % of their compensation they are willing to bet, that this is not an actual bet. How is this explained to participants, and how might this effect their responses. If this is an actual bet, then this has potential ethical implications.