Dear authors,

Thank you very much for your thorough work in revising this registered report. As it stands now, the manuscript addresses most of the concerns that I outlined (and I can mostly follow your rebuttals on the points that you did not make changes, with the remnant points being minor and not worth of further delay). I especially appreciate the updated analyses and updated scales/items used in the DV. As it stands now, I am quite confident that this study will meaningfully contribute to the literature and I’m looking forward to seeing the results!

Best,

Philipp

Dear authors, 

I have (finally, my apologies) read your revised protocol, as well as your responses to the reviewers. I am very happy with how my own suggestsions were handled (the addition of Bayesian analysis and MI in particular), and have seen that most of the other comments were also handled proactively. The ones relating to planned contrasts and one- versus two-tailed tests were, in my opinion, quite important to address.

My one comment regards this text on p. 18 of the preprint: "Bayes factors will indicate the strength of evidence for each hypothesis, with values greater than 1 supporting the corresponding hypothesis. Rather than relying on fixed thresholds, we will interpret the Bayes factors contextually following the guidance of Hoijtink et al. (2019b)." I have a few thoughts about this, and would like the authors to consider this before collecting data, perhaps even in a new, minor revision or even just in the manuscript.

• First, why talk about a threshold (in this case of 'greater than 1') if you then say that you won't rely on fixed thresholds? Second, why >1? a BF10 of 1.1 is within that range, but is practically the same as saying that both hypotheses are equally likely given the data... Why not greater than or equal to 3 (or the inverse), which is a commonly accepted threshold. Again, only if you're talking about thresholds which you claim then not to be. Do I misunderstand perhaps? If so, my apologies - it might help the reader to make this a little clearer? 
• The Hoijtink article you mention in this same text passage is huge - nearly 50 pages - because it presents several options for reporting Bayesian statistics. From what you write, it's difficult to know what you might mean when you say you'll follow their guidance, because they offer a lot of guidance and for a lot of difference use cases. I suggest that because you're interested in using Bayesian inference to help contextualise the findings, it might be an idea to be specific about how Bayesian analysis can support this. For instance, by interpreting posterior probabilities which are based on the prior distribution and the likelihood of the observed data. They can be useful for parameter estimation and understanding uncertainty, which I think might be useful here (more so than a close-to-1 BF10 which isn't likely to give much information). Hoijtink's article provides much help on how to report and interpret these, so this is likely something you're already aware of but I have suggested it, just in case this is something that you hadn't considered. 

I have nothing further to mention other than this. I am happy to see this be moved on in the process, and wish the authors luck with the data collection and preparation of the stage 2 RR manuscript! 

I always sign my reviews, 

Dr. Sarahanne M. Field

The Stage 1 report proposes to run an important further analysis of the Bayesian Truth Serum. Overall, the question asked is novel and interesting, irrespective of result. However, there are a number of important concerns (1-5) I would like the authors to address, as well as some smaller general issues (6 and onwards) where I invite the authors to address/clarify/add/remove/change something but where I am also happy for the authors to provide a simple explanation for their current choice instead. As such, I would like to see the authors resubmit a revised version of this report.

First, with respect to participants, is there a specific reason why the authors aim to collect responses from six nations instead of one? Prolific will easily be able to provide you with the required sample from just one country without introducing this potential (albeit unlikely) confound. Additionally, within Prolific, are you going to use further selection criteria? E.g., with respect to the number of successfully completed tasks, language competency, etc.

Second, I would ask the authors to clarify the hypotheses and the tails of their analysis. I think stating the null hypotheses (instead of the directional hypotheses) should improve clarity and also set up the authors to say more about what kind of conclusions they would draw in the case of a backfire effect (i.e., if the BTS makes responses less ‘honest’). The paper can still include directional predictions, of course, but as it currently stands there seem to be a couple of potential patterns of data that are not accounted for.

Third, I strongly suggest you adjust your p-values in a way that’s distinct from your current approach, where the adjusted level is still at 5%. I leave it up to the authors to decide an adjustment procedure that they think is appropriate, but I strongly suggest you pick a method that does not result in de facto non-adjusted p-values, as I am not convinced by Rubin (2021) and its application to your design.

Fourth, while I respect the authors’ choice in focusing on mean differences as their main outcome variable, I would suggest using at least one test of differences in response distributions as an exploratory additional analysis (that need not play into the main hypotheses but should be reported). I leave it up to the authors to pick whatever test they think is most appropriate, but I think some type of analysis may help provide a better understanding of a potential null effect.

Fifth, as far as I can tell, the authors do not provide a plan on how they deal with null effects. The preregistered outcome interpretations talk of a failure of hypothesis support (which is framed as a directional effect), so a backfire and a null effect would be treated identically, which doesn’t seem right to me. This gets back to the framing of hypothesis as directional  having them stated as ‘no difference’ null hypotheses allow you to also consider scenarios where effects do not work (established, for example, by some type of equivalence tests with the bounds drawn from the effect size used for the power calculation) or actually backfire.

Sixth, you say that “the BTS will be tested for its ability to elicit truthful responses across various dimensions”. I might have missed this, but won’t you be aggregating across all question types? If so, how will you test for the effects in subsets of these questions? Will these be additional analyses? If so, these p-values also have to be adjusted and their relationship to the main hypotheses stated.

Seventh, why are you preregistering a Welch’s t-test without knowing the variances in both groups? Would it not be better to preregister the criterion by which you will decide on the test (which may well end up being a Welch’s t-test)?

Eighth, how are the questions presented to participants? Are they randomised?

Ninth, how will you include demographic variables in your analyses (if at all)?

Tenth, I would like the authors to also reflect on their choice of items, as some seem to be from work published in the 90s (and as such conducted prior to that). Is it not very plausible that some of the uncomfortable and/or sensitive topics of the 90s are quite distinct from the ones today? For example, discourses of gender and race/ethnicity have changed dramatically, and so has what is seen as sensitive, potentially putting into question your assumptions for the outcome variables.

Eleventh, some small things. I don’t think abbreviations in the abstract are necessary (page 1). Early on (page 3), you state that the BTS is ‘novel’, though I am not sure this is entirely true given the amount of work that has been done on it already. What do you mean by ‘overlapping coefficient’ (page 6)?


I look forward to seeing the revised report!

Best,
Philipp Schoenegger

Dear authors, 

In general I like the idea of this study - it is an interesting topic, and seems to represent a good way of testing the BTS. I find the protocol to be clearly and thoroughly written. In particular, I appreciate the attention that has been paid to the sample size justification. There are, however, a few things I am less sure about. 

1. I'm not convinced by your explanation for choosing a deletion procedure rather than multiple imputation procedure to handle missing data. While I think it's quite likely that missing data will be minimal, and that deletion is unlikely to introduce bias, why not just choose a method that is more methodologically robust? Can you explain your choice a little more, or change the strategy? 

2. I'm wondering about the payment amounts - they seem so small that I am wondering if they will be enough for the purpose. They may well be, of course, but I am not convinced because you (unless I missed something) do not provide a justification for why you have chosen these amounts and why you think they will be sufficient for their purpose. Please provide a clear motivation for the amounts chosen.

3. I did not find a clear justification for 10 questions. You say that you chose this number to keep it manageable, but why do you think 10 will be manageable, and not, say 15 or 20? Have you run a pilot to actually test the time cost to participants? 10 seems to be a low number, and without sufficient motivation about why you have chosen this number, I am left wondering if that will be enough questions. 

4. What will you do if you get p-values greater than your cutoff alpha? Do you just shrug and go "well, we don't really know..."? Because that's about the extent of the information you will get if you use frequentist statistics. I *strongly* recommend using a Bayesian approach along side of frequentist (or even in place of!), because it allows us to deal with pro-null evidence and because Bayes factors can provide more information value than p-values can.

Best of luck with the study going forward!

Sarahanne M. Field