Managing costs and rewards when choosing to disclose information
Managing Disclosure Outcomes in Intelligence Interviews
Recommendation: posted 21 April 2023, validated 21 April 2023
Dienes, Z. (2023) Managing costs and rewards when choosing to disclose information. Peer Community in Registered Reports, . https://rr.peercommunityin.org/PCIRegisteredReports/articles/rec?id=315
Related stage 2 preprints:
David A. Neequaye, Timothy J. Luke, Kristina Kollback
https://doi.org/10.31234/osf.io/tfp2c
Recommendation
Neequaye et al. (2023) will invite participants to assume the role of an informant, with the goal of maximizing their points according to stated probabilities of costs and benefits of disclosing pieces of information relating to given scenarios. Then the degree to which each type of information is disclosed in a subsequent interview will be assessed: this way the crucial interaction can be tested.
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.
List of eligible PCI RR-friendly journals:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.
Evaluation round #1
DOI or URL of the report: https://psyarxiv.com/tfp2c
Author's Reply, 19 Apr 2023
Decision by Zoltan Dienes, posted 28 Mar 2023, validated 28 Mar 2023
Once again sorry for the far too long delay in getting back to you. In the end I asked 60 potential reviewers - and three responded who were experts in relevant areas, and I was very pleased those were the ones we now have. They are largely happy with your proposal, but have various comments concerning details of the protocol and requested clarifications.
Reviewed by Jason Chin, 22 Mar 2023
Reviewed by Yikang Zhang, 06 Mar 2023
Reviewed by Tyler Jacobs, 19 Mar 2023
In this article, the authors propose and aim to test a new framework for understanding when people will disclose information during an intelligence interview called the Disclosure Outcomes Management Model. This model frames the decision to provide information as a self-interest dilemma in which the interviewee must balance the potential benefits (e.g., community safety, upholding morality) of disclosing risky information with the potential harms to the self (e.g., retaliation from the group being reported on). They then describe the calculations of risk and benefit result in four categories, with low-stakes and guarded information (theoretically) being less likely to be disclosed, unguarded information being more likely, and high-stakes information being variable. The authors then report a preliminary study that supported these hypotheses, and then begin the registered report for the planned study.
Strengths:
-The theory is described well in the Introduction and Figure 1 displayed the model in a clear way.
-The authors use sophisticated multilevel models to account for random effects in their complex experimental design, which is important for their research design.
-In my opinion, the design has strong internal validity.
Concerns and Comments:
Overall, I think that this is a reasonable design with strong proposed analyses. I do not see any major issues. However, there are few aspects that I would like the authors to consider.
-First, on the theory-side, I found the definition of “self-interest” to be unusual (“broadly encompass[ing] any outcome an interviewee may want to achieve or avoid”). In social psychology, self-interest is typically defined as the motivation to achieve outcomes that benefit the individual and avoid those that do not (Miller, 1999; Gerbasi & Prentice, 2013). Additionally, many theories specifically state that if the outcome is intended to primarily benefit others, it is not self-interested (Cropanzano et al., 2005; Holley, 1999). Thus, I would argue that “act[ing] in the best interests of other associates” is not actually self-interest. Their definition is closer to a purely economic definition of self-interest (maximizing one’s gains, minimizing losses), but I am not sure that this fits either, and the interview situation is not a purely economic one. Could authors either provide further justification and citations for their definition of self-interest, or consider if another term would fit better?
-Could the authors more clearly state their exclusion criteria (e.g., how much missing data is too much or how many memory checks can be failed)? The authors could also consider reporting the results without exclusions in the Supplementary Materials. I would reccomend this especially given the large number of excluded participants in Study 1 (transperency would be best).
-In lieu of an a priori sample size analysis in Study 1, could authors perform a sensistivity power analysis ?
-For Study 2, given the large number of excluded participants in Study 1, could the authors describe the number of participants they will recruit (before exclusions) in order to meet the minimum sample size?
-In addition to the model fit stats (AIC), could the authors report effect sizes for their models (R^2 or f^2; or ICC for random effects)?
-I appreciated the authors' discussion of internal and external validity. However, I would note that concerns about external (and construct) validity go beyond just how interviews would have more psychological realism. In a real-life interrogation situation, the consequences go far beyond collecting points and competing to receive a monetary reward and could involve fear for one’s life, fear for loved ones, fear of implicating one’s self in a crime, etc. Thus, despite the efforts to vary the consequences and incorporating a choice structure with incentives, this design likely does not perfectly capture the construct of these situations in the real world. That being said, this artificiality is common in psychology research and is necessary for internal validity. This idea, though, could be added to the discussion on lower external validity.
I thank the authors for their efforts, and hope that this feedback is helpful.