Different disciplines and research areas that rely on participants’ self-reports to accrue data on participants’ true preferences are faced with the question to what extent the former can be equated with the latter. Using monetary incentivisation for study participation may influence this relationship, and researchers, especially in economics, have been discussing how to develop and implement incentive-compatible research designs, i.e., those in which the incentivisation yields the best payoff for the participant if they report their true preferences (Hertwig & Ortmann, 2011; Baillon, 2017). The Bayesian Truth Serum, first introduced by Prelec (2004), according to which participants are rewarded based on how surprisingly common their own answers are relative to the actual distribution of answers, has been proposed as a possible incentive-compatible design for survey studies that rely on participants’ self-reports about their true preferences (Schoenegger, 2021).

 

In this study, Schoenegger and Verheyen (2022) ran a replication of the study by Schoenegger (2021) and assessed whether the effect elicited by the manipulations known as the Bayesian Truth Serum is distinct from its separate constituent parts. The authors report that the manipulation did not yield a significant difference compared to control conditions, which they interpret as a failure to replicate the original results. At the same time, the authors are careful in drawing conclusions as to the usefulness of the Bayesian Truth Serum for self-report studies using Likert-scale items in general, as they emphasise that smaller effect sizes may be of interest and that the results may differ when different items are used. 

 

The Stage 2 manuscript was evaluated by two reviewers, one of whom reviewed the first Stage 1 submission, and the other one of whom reviewed the manuscript specifically to assess statistical questions.

 

Following a careful revision by the authors, the recommender judged that the manuscript meets the Stage 2 criteria and awarded a positive recommendation.

 

URL to the preregistered Stage 1 protocol: https://osf.io/dkvms

 

Level of bias control achieved: Level 6. No part of the data or evidence that was used to answer the research question existed prior to Stage 1 in-principle acceptance. 

 

 

 

1. Baillon, A. (2017). Bayesian markets to elicit private information. Proceedings of the National Academy of Science, 114(30), 7985-7962. https://doi.org/10.1073/pnas.1703486114

 

2. Hertwig, R. & Ortmann (2001). Experimental practices in economics: a methodlogical challenge for psychologists? Behavioral and Brain Sciences, 24(3), 383-403. https://doi.org/10.1037/e683322011-032

 

3. Prelec, D. (2004). A Bayesian Truth Serum for Subjective Data. Science, 306(5695), 462-466. https://doi.org/10.1126/science.1102081

 

4. Schoenegger, P. (2021). Experimental Philosophy and the Incentivisation Challenge: a Proposed Application of the Bayesian Truth Serum. Review of Philosophy and Psychology https://doi.org/10.1007/s13164-021-00571-4 

 

5. Schoenegger, P., & Verheyen, S. (2022). Taking A Closer Look At The Bayesian Truth Serum: A Registered Report. Stage 2 Registered Report, in principle acceptance of Version 3 by Peer Community in Registered Reports. https://doi.org/10.31219/osf.io/9zvqj