Decoding Interviewer’s Intent: How Interviewees Infer Information Goals

The authors have addressed all my previous comments and questions. I believe that the latest amendments have improved the manuscript by providing clarification, making some claims and specific hypotheses more accessible and providing more context to some methodological decisions. 

Thank your for the opportunity to review this report again. I appreciate the additions the authors have made to the report - it is much clearer now. However, I believe that my main points have gone unaddressed. While I am not going to reiterate them here, let me point out the following:

It is indeed true that a study that sets two theoretical models from which two competing hypotheses have been derived and where the data will then support one of these hypotheses is optimal for scientific progress. I do not believe that this is the case here. The authors merely state that if the main hypotheses are not supported, then the theory needs to be revised. This is quite standard and for this reason the revision hypotheses do not seem to be necessary. 

I also continue to believe that the analytical approach is overly complex for the analytically simple questions that are being asked. 

Finally, the authors state: "Before an interviewer poses any question, it is reasonable for interviewees to assume that the elicitation of complete details is the de facto purpose of an interview. All things being equal, any investigator would want the complete details an interviewee holds, given that complete information would be more beneficial to any investigation than partial details." I do not believe these can be assumed without accompanying evidence. I would not agree with them in any case and they seem to go against the pragmatic principle of optimal relevance in communication. 

The authors have responded to most of my comments/questions. Below I outline a few remaining issues that are straightforward to address. In fact, for all except the first comment below, the authors responded in full in the response letter and I simply ask that they add these statements in the manuscript too.

 - I made the following comment before: 'in practice cooperative interviewees may understand that the interviewer wants highly specific/complete details but may be unable to report these from memory. Some consideration to the implications of the current assumption risking cooperative interviewees being perceived as uncooperative is warranted'. The authors replied stating that the focus here is about pragmtic considerations and not research on memory or disclosure. Fair, and I understood the authors' thinking. My comment is about considering the practical implications of their reserach findings - and any unintended consequences. As the authors state: ''By asking high- or low-specificity questions, interviewers can influence the extent to which interviewees might discern what they want to know" and depending on the findings it is plausible that interviewers can assume that their objective is clear and without awareness of memory issues wrongly infer the interviewee's response means they are being uncooperative when they are unable to remember - and saying so would not necessarily make a difference. If this seems too obvious, I would say that it is still worth addressing especially since it can very easily be addressed by adding a few sentences in the introduction where the authors can both emphasise that disclosure based on memory is not the current focus and clarify that factors based on memory can prevent cooperative individuals from providing specific information even if they recognise the objective. The point could also be considered in the discussion depending on the findings as a caution, but it is up to the authors at that point.

- The authors addressed another of my comments by stating that in the current reserach they 'simply claim that one must understand a question’s purpose before disposition or other contextual factors most proximate to disclosure can take effect. Identifying the question’s purpose has an indirect influence because it tells the cooperator or resistor how best to achieve their goals. Deciding what to do, in this case disclosure or nondisclosure, is when disposition kicks in'. Given how clear this statement is including it in the manuscript would be ideal to avoid confusion.

- Regarding comment 5, I appreciate that the authors have included detailed information in the Study design template. My mistake, I should have clarified that I was referring to the first mention of this statement on p. 9-10, and my comment referred to clarity in text so that it is easier for the reader to follow. The authors have now added a clarification on this point although it might be worth using very direct language i.e. participants will be more likely to identify that the interviewer aims to focus on highly specific details.

- ICC value and coding strategy: yes, it was clear that the authors only referred to the possibility of the ICC value being below 0.6. To clarify my point was that if the ICC is below this value then preferably the authors would report on this in the final manuscript to some (appropriate) extent as it would be a limitation. The reason I raised this is because they state that they would 'discuss the causes of disagreements and resolve them independently' before recoding the data, but ideally some more detail about the issue would be reported. Hopefully this is clear and ideally not even needed in the final manuscript.

- In the response letter, the authors state that they will explore all the data points that receive the designation 'cannot decide' for any obvious patterns and label them as exploratory. I would then suggest that they mention this already in the manuscript too, before it is accepted for submission.

Thank you for the opportunity to review this plan for a registered study. While I found the overall research question (how interviewees determine what they think interviewers want to know) interesting and worthwhile, my enthusiasm was dampened by several issues with the proposed registered report. These were:

-The authors would like to test whether their pilot study had a confound by replicating that within-subjects design study and then running a study with a between-subjects design and compare the results (If I understood the aim correctly). Why not just vary the presence or absence of different question types experimentally in one experiment? That would seem a much more straightforward and effective way of addressing the issue of whether the assumption that interviewers want to know all dependend on the within-subject nature of the design. Also, it seems to me that the whole setup with core hypotheses and revision hypotheses is not appropriate for a registered report that should, in my opinion, more clearly commit to a particular hypothesis/es that flow from a theoretical framework. 

-I feel that the planned analyses are overly complex for the relatively straightforward and simple experimental designs proposed. Why not use more classical statistical tests? At a minimum the proposed analytical approach needs to be motivated much more clearly. Admittedly, I am not competent to assess all the choices made but then many readers will probably feel the same leading to difficulties in communicating the findings clearly. 

-Most importantly, I feel that the theoretical model is not explained clearly enough. The authors suggest that "Then, researchers can better specify how various interviewing methods might exert their effects." referring to the potential of understanding what the interviewer wants to know. This claim should be further explained using examples. Also, the authors state "Before an interviewer poses any question, it is reasonable for interviewees to assume that the elicitation of complete details is the de facto purpose of an interview." I do not think this is at all self-evident. It would depend on the context. In fact, the Cognitive Interview is trying to counter-act the tendency of witnesses to assume that the police does not want to know all the witness may know. Finally, I did not understand core hypothesis 1: "When interviewees determine what an interviewer wants to know, high-versus low- specificity questions should elicit a greater preference for pragmatic correspondence (Core Hypothesis 1)." What do you mean by preference here? 

-In terms of measurement of the main variable (i.e. what the interviewees think the interviewer wants to know), why not ask the interviewees to indicate for each item in the scenario they have been given if they think the interviewer's question indicates them wanting to know this item or not?

-Finally, it would seem also interesting to include the perspective of the interviewer in this line of research. This would allow investigating whether there is a correspondence between what the interviewer wants to know and what the interviewee thinks the interviewer wants to know. However, I understand that both perspectives may not be possible to include in the same (series of) experiment(s). 

The current submission describes three replications of Neequaye and Lorson (2023) to examine which of two mechanisms under consideration better captures how interviewees (as informants) perceive what interviewers want to know based on question formulation. In Replication 1, high- and low-specificity questions will be tested in a within-subjects design, vs in a between-subjects design in Replication 2. Based on results, a third replication will use a between-subjects design and new stimulus materials to address any limitations. There is thorough consideration of the method across replications to clarify previous findings and address hypotheses; a carefully planned power and statistical analysis.The investigation of the two competing mechanisms is interesting and I can see the practical implications for the field (although contextual differences are important to consider). Some thoughts below about the link to the ‘disposition’ factor and questions about aspects of procedure/coding.

1.      The introduction could draw from more theoretical sources and previous research to strengthen the rationale for the research question and its operationalisation. For instance, I would expect to see reference to Grice’s maxims of conversation, links to related terminology and theory in applied cognition such as diagnosticity of cues when prompting relevant information from memory, fine vs coarse reporting etc. If relevant theory has been used in previous publications as cited in the current submission, a brief summary would still be helpful to the reader.

2.      The rationale to examine the link between the disposition (which is instructed to participants) and the participants’ reported understanding of the interviewer’s goals (high/low specificity questions) is not entirely clear based on the introduction.  The authors state that ‘this honing process can affect whether an interviewee cooperates or resists’. A few thoughts on this point:

-          based on the examples provided in the method and the particular scenario, the purpose remains that the investigator wants information that can help investigate a criminal gang (overall purpose of questioning), so isn’t it possible for interviewees to assume that complete details are the most informative and the objective of the investigator’s questions? My point is that the scenario and related instructions can also affect participants’ understanding of the goal of the questioning so instructions could be amended and/or this might need to be the focus of replication 3 (as the issue of the stimulus material is mentioned).

-          In real practice, cooperative interviewees may understand that the interviewer wants highly specific/complete details but may be unable to report these from memory. Some consideration to the implications of the current assumption risking cooperative interviewees being perceived as uncooperative is warranted.

-          Participants are actually instructed from the start (or prior to questioning) to be cooperative, semi-cooperative, or resistant (which would indeed be the case). Related to this, I understand that Hypothesis 3 is essentially a null hypothesis to show that there should not be an effect of disposition on the interviewee’s understanding of the question’s purpose, but this seems to contradict the point above and the following one: ‘Interviewees must identify their interviewer’s objectives before they can determine the extent to which they might cooperate with or resist their interviewer’s requests’ – a decision which to be fair can be made based on multiple other factors, also depending on context.

3.      Thank you to the authors for making the protocol available on Qualtrics. About ‘S1 Manipulation check’: although a manipulation check to ensure that participants understand their role based on the instructions is necessary, this check asks about their preferred strategy, which is not exactly the same. It would be interesting to collect data on participants’ strategies at the end of the study, but I am not sure it can serve as a manipulation check on its own.

4.      On this note, could there be inclusion of ratings (or ranking of most relevant disposition) in the current research to see to what extent participants accurately perceive their instructed disposition? My concern is that the semi-cooperative and resistant dispositions could be perceived similarly by interviewees as in the latter, they still need to avoid suspicion, which can induce a semi-cooperative mindset.

5.      The outcome for confirmation of hypothesis 1 could be stated more explicitly to how participants can respond in the experiment – e.g., does ‘elicit a greater preference for pragmatic correspondence’ mean that participants will be more likely to identify that the interviewer aims to focus on highly specific details – and how?

6.      Questions 5 and 6 (p. 34) – high vs low specific questions are formulated quite differently relative to others (with respect to the target of the question). For example, question 6b ‘Have you made any observations about KET22’s customers?’ could refer to a personal description of the customers, vs how they contact KET22, which is a lot more specific in 6a. I understand that is part of the point, but my concern is that questions 5 and 6 are much broader than in all other question sets, where e.g., instead of ‘exact location’ for high-specificity, ‘anything about where’ is used for low-specificity, thus still obviously capturing the ‘exact location’.

7.      Coding strategy:

-          Shouldn’t there be a discussion of a limitation of the coding strategy if the ICC value is below 0.6?

-          I am curious about the choice of ‘cannot decide’ in the following case: (3) “when the choice fits neither the high- or low-specificity question”. Wouldn’t this response be rather informative for the research question and worth coding in itself – maybe as exploratory?

8.      Could the authors provide some more context/justification for the following decision: ‘Our desired level of precision is that the width of parameter coefficients’ 95% HDIs should be equal to or smaller than 16’?

Note: I tried to attach as a separate document, but it wasn't clear if it uploaded or not, so I just pasted here. You can ignore the attached if there. 

1A. The scientific validity of the research question(s). 
 
The research questions are scientifically valid, in that they are answerable via research. I had no concerns regarding the ethics. However, I found the theoretical basis to be weak. There was little-to-no reference to the research literature beyond the authors’ own previous study (see below for more on this).
 
 
1B. The logic, rationale, and plausibility of the proposed hypotheses, as applicable. 
 
I found the hypotheses to be appropriately precise, and sufficiently conceivable as to be worthy of investigation. That said, I found the wording of “preference for pragmatic correspondence” to be confusing, as the outcome isn’t the participants’ preferences. A slight rewording would be helpful.
 
However, as noted in relation to criteria 1A, I did not find them to be based in any clear theory, or follow from past findings (again, with the obvious exception that the authors were seeking to replicate their prior work and explore the boundaries of those findings). What was missing for me was more explanation of the mechanism. Are the predictions based in theories of memory activation? Or based more on how people interact socially / conversational norms? When considering how to provide answers to questions (a different but related topic) we know people tend to consider both accuracy and informativeness. Is that part of the thinking here as well?
 
On page 6 I was a bit thrown by the paragraph starting “Before an interviewer proposes any question, it is reasonable for the interviewees to assume that the elicitation of complete details is the de facto purpose of the interview.”  This seems to argue against their first core hypothesis. If this is being presented as an alternate theory that didn’t fully come across.
 
 
1C. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis or alternative sampling plans where applicable). 
 
Positives: The rules for data exclusion are stated; it’s noted that coders will be kept naïve to hypotheses; the order of scenarios will be randomized; participation in R1 and R2 will happen concurrently; there are attention and manipulation checks; R2 clearly follows up on a potential explanation for findings in the past work while R1 seeks to replicate past work with a similar design; the sample size, power, and sampling plan are acceptable
 
Concerns:
1.        It’s not clearly stated that random assignment will be used for the between subjects manipulations (I rather assumed it would be, but it should be stated in the text/procedure).
2.        How the manipulation check regarding disposition would be scored wasn’t stated, and I did not find it to be obvious. Further, it wasn’t clear if failure to answer this “correctly” would result in exclusion or not. Presumably not, as other exclusion criteria were noted.
3.        It wasn’t clear what the “decision making” manipulation check was (in Exclusion Criteria section) but I assume it refers to that “instructional” manipulation check (in Appendix B).
4.        The textbox prompt “The police-contact wants to know if…” seemed odd to me. Specifically, the word “if.” That seems like it would generate answer like “…if I know what brand of drugs the gang is selling” as opposed to answer like “…the gang is selling off brand oxy” (I assume the latter is the type of answer the authors’ are seeking. I realize the authors piloted this, so I’m willing to defer to them on this point. It just seems very odd to me.
5.        The “optional” wager question comes out of nowhere – why is it included? It’s also unclear what is meant by optional. They have the option to skip it and not indicate what/if they would wager? Or it’s optional to wager something (i.e., they have the 0% option)? 
6.        The authors state that in their past study, and in R1 and R2, pragmatic correspondence was designed to be equivalent to complete details – so high specificity questions specifically request complete details. I thought I understood this, and it made sense to me. But once I got to the Appendices and saw the scenario information and the high and low specificity questions, I had some concerns. In fact, this is my biggest concern regarding the proposed studies. 
 
Let’s take the first scenario:

One day after work, on your bus ride home, you recognized one of the KET22
members. You were sitting just behind him, and he was talking on the phone.
He tried to be quiet, but you heard him say: “It is better to sell the off-brand
green-star oxycodone.”
a. Have you discovered the particular brand of narcotics KET22 sells? [HIGH]
b. Have you discovered anything about the gang’s narcotics sales lately? [LOW]
 
I would argue that the first question could be answered with “they are selling off brand drugs” – I don’ t how it’s seeking complete information. In contrast, the second, low specificity, question seems to be seeking complete information.
 
This can be contrasted with the second scenario, where the high specificity question is clearly asking for full details (and questions like this are in line with what I was expecting):
 
You always come to work earlier than your colleagues because you supervise
the cleaners. You’ve realized that the KET22 gangsters usually arrive shortly
after you in a blue Nissan Qashqai. By paying more attention, you’ve
memorized the license plate number: FBT038.
a. Do you know the full details about the vehicle the KET22 gangsters
usually arrive in at the park? [HIGH]
b. Do you have any information about KET22’s transportation in the park? [LOW]
 
Turning to third scenario, this seems more like the first, where a very specific piece of information is requested by the high specificity question: “EXIT 7F.” In contrast, the low specificity question seems to invite any and all information.  
 
Lately, you have noticed a particular spot at the park where the KET22
gangsters deal drugs in the evenings. The spot is one of the park’s exits, EXIT
7F. All the exits are located at different edges of the park, but 7F is rather
discreet.
a. Have you spotted the exact location at the park where KET22 deals
drugs? [HIGH]
b. Have you spotted anything about where KET22 deals drugs? [LOW]
 
I will not go through each scenario and its corresponding question. These differences need to be addressed. Either the authors think that asking for a very specific detail will elicit all relevant information (i.e., scenarios 1 and 3), or they think that asking for full information will elicit all relevant information (i.e., scenario 2). But those seem like very different things, and either way, this should be clear in the manuscript, and the scenarios should be consistent in how this is approached.
 
I also have concerns with the fact that scenarios contain such limited information. To me this makes it reasonably easy for all participants to choose to provide all information. Unless I’m missing something, it seems like participants don’t even need to be presented with any information. They could just be presented with questions and ask what information they think the interviewer would want them to find out. This is less leading as there are many potential options, not a couple details. Indeed, some of the low specificity questions are so vague that there are a huge number of details that an interviewee might suggest the interviewer was interested in, if they were not confined to 2 or 3 pieces of information.  (e.g., “Have you discovered anything about the gang’s narcotics sales lately?: This could be getting at whether sales good or bad; is one product selling better than another; sales are initiated via text messages; sales are primarily conducted by person X and person Y). As the proposed studies are more of an initial preliminary test, this is less of a concern than the previous point I made, but something to consider moving forward.
 
Note: I must apologize, but I do not feel I have the relevant statistical expertise to provide a helpful evaluation of the statistical analyses proposed. (Nothing struck me as incorrect or concerning.)
 
1D. Whether the clarity and degree of methodological detail is sufficient to closely replicate the proposed study procedures and analysis pipeline and to prevent undisclosed flexibility in the procedures and analyses. 
 
I appreciate the authors inclusion of the study materials; these were very helpful. That said, without providing the materials in the appendices I don’t think the methods would be sufficiently clear.
 
A few places I was still a bit unclear, even with the appendices: 
-The presentation of the control questions was not entirely clear to me. This should be better incorporated into the procedure.
-The “disposition” manipulation was introduced in such a way on page 8 that I didn’t realize it was a manipulation. I thought at first that participants could choose their disposition (also the term disposition was not introduced at that time, so it took me by surprise later).
-Coding: I was quite confused here. I think part of the is that perhaps where the authors said “choice’ they really intended to say “response.” If that is the case, I think I understand what the authors are proposing, but, it doesn’t quite make sense to me. I think this is related to my major concern above. My understanding is that the authors are proposing that coder should code more “complete” answer on the “high specificity” side of the scale. But that assumes that the high specificity questions really were in fact seeking complete details. Which, as I note above, I don’t think they (all) are. 
 
1E. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the obtained results are able to test the stated hypotheses or answer the stated research question(s). 
 
As noted above, there are manipulation checks included. Overall I don’t have concerns with this criteria beyond anything I have already mentioned.