Close printable page
Recommendation

Reassessing the use of the Bayesian Truth Serum as an incentive-compatible design for self-reports

based on reviews by Joël van der Weele
A recommendation of:
toto

Taking A Closer Look At The Bayesian Truth Serum: A Registered Report

Abstract
Submission: posted 11 June 2022
Recommendation: posted 16 September 2022, validated 16 September 2022

This is a stage 2 based on:

Recommendation

Different disciplines and research areas that rely on participants’ self-reports to accrue data on participants’ true preferences are faced with the question to what extent the former can be equated with the latter. Using monetary incentivisation for study participation may influence this relationship, and researchers, especially in economics, have been discussing how to develop and implement incentive-compatible research designs, i.e., those in which the incentivisation yields the best payoff for the participant if they report their true preferences (Hertwig & Ortmann, 2011; Baillon, 2017). The Bayesian Truth Serum, first introduced by Prelec (2004), according to which participants are rewarded based on how surprisingly common their own answers are relative to the actual distribution of answers, has been proposed as a possible incentive-compatible design for survey studies that rely on participants’ self-reports about their true preferences (Schoenegger, 2021).
 
In this study, Schoenegger and Verheyen (2022) ran a replication of the study by Schoenegger (2021) and assessed whether the effect elicited by the manipulations known as the Bayesian Truth Serum is distinct from its separate constituent parts. The authors report that the manipulation did not yield a significant difference compared to control conditions, which they interpret as a failure to replicate the original results. At the same time, the authors are careful in drawing conclusions as to the usefulness of the Bayesian Truth Serum for self-report studies using Likert-scale items in general, as they emphasise that smaller effect sizes may be of interest and that the results may differ when different items are used. 
 
The Stage 2 manuscript was evaluated by two reviewers, one of whom reviewed the first Stage 1 submission, and the other one of whom reviewed the manuscript specifically to assess statistical questions.
 
Following a careful revision by the authors, the recommender judged that the manuscript meets the Stage 2 criteria and awarded a positive recommendation.
 
URL to the preregistered Stage 1 protocol: https://osf.io/dkvms
 
Level of bias control achieved: Level 6. No part of the data or evidence that was used to answer the research question existed prior to Stage 1 in-principle acceptance. 
 
List of eligible PCI RR-friendly journals:
 
References
 
1. Baillon, A. (2017). Bayesian markets to elicit private information. Proceedings of the National Academy of Science, 114(30), 7985-7962. https://doi.org/10.1073/pnas.1703486114
 
2. Hertwig, R. & Ortmann (2001). Experimental practices in economics: a methodlogical challenge for psychologists? Behavioral and Brain Sciences, 24(3), 383-403. https://doi.org/10.1037/e683322011-032
 
3. Prelec, D. (2004). A Bayesian Truth Serum for Subjective Data. Science, 306(5695), 462-466. https://doi.org/10.1126/science.1102081
 
4. Schoenegger, P. (2021). Experimental Philosophy and the Incentivisation Challenge: a Proposed Application of the Bayesian Truth Serum. Review of Philosophy and Psychology https://doi.org/10.1007/s13164-021-00571-4 
 
5. Schoenegger, P., & Verheyen, S. (2022). Taking A Closer Look At The Bayesian Truth Serum: A Registered Report. Stage 2 Registered Report, in principle acceptance of Version 3 by Peer Community in Registered Reports. https://doi.org/10.31219/osf.io/9zvqj
Cite this recommendation as:
Ljerka Ostojic (2022) Reassessing the use of the Bayesian Truth Serum as an incentive-compatible design for self-reports. Peer Community in Registered Reports, . https://rr.peercommunityin.org/articles/rec?id=218
Conflict of interest:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.

Reviews

Evaluation round #3

DOI or URL of the report: https://osf.io/9zvqj

Version of the report: V4

Author's Reply, 16 Sep 2022

We have now added the Conflict of Interest statement and have also moved the Author Note and the Open Practices Statement in front of the References.

Decision by , posted 16 Sep 2022

Dear Philipp, 

Many thanks for the careful revision. I am happy to now recommend this Stage 2 report, following a final minor revision.

I would just ask you to add a conflict of interest statement before the Reference section in line with the PCI RR guidelines (see text below):

Authors must have no financial conflict of interest relating to the article. The article must contain a "Conflict of interest disclosure" paragraph before the reference section containing this sentence: "The authors of this article declare that they have no financial conflict of interest with the content of this article.";

Once this is done we will proceed immediately with the final Stage 2 recommendation.

All the best, 

Ljerka

 

 

Evaluation round #2

DOI or URL of the report: https://osf.io/9zvqj

Version of the report: V3

Author's Reply, 10 Sep 2022

Decision by , posted 31 Aug 2022

The stage 2 report has now been seen by one of the original reviewers as well as Zoltan Dienes, who served as a statistical reviewer during your stage 1 revision process and whose comments I am pasting here. Unfortunately I was not able to contact the other original reviewer who had reviewed your stage 1 report during the first submission stage.

As you will see, there are some changes that are required: 1. a clearer separation between the registered and non-registered analyses, such that the different analyses are clearer for readers to follow, and 2. an adjustment to your conclusions based on the non-significant results you report. 

One additional revision for the authors to make is in Footnote 3:

"This manuscript is a Stage 2 Registered Report of this Stage 1 Registered Report https://osf.io/xw6hn"
Replace that with the following (noting the change of URL to the official PCI RR registration)
“This manuscript is a Stage 2 Registered Report of this Stage 1 Registered Report: https://osf.io/ztucr (date of in-principle acceptance: 23/04/2022)”

 

Comments by Zoltan Dienes:

I didn't read the final Stage 1, so this is based on my reading of the Stage 2. It is clear and concise.  However, the registered and non-registered analyses should be in separate sections to more clearly separate them. Further, their use of the obtained effect size from their previous study as a basis for the power calculation for the current study has led to a non-signfiicant result with an effect size V = .09, with the original significant effect size being V = 0.1. I realize already vast numbers of subjects were needed for the RR as it is; but still the upshot is, the study was not powered to detect all effects of interest (if V = 0.1 was of interest, then so is V = .09). Thus, the abstract and discussion need to conclude more along the lines of "reserve judgment" rather than "we failed to replicate". The abstract and discussion should point out that although the current study was non-significant, it was not powered to detect all effects of interest.

Reviewed by , 11 Jul 2022

Dear editor

my review of the paper is complicated by the fact that I missed the second review moment (finalization of stage 1). I also did not see a way to access the comments and replies pertaining to that round.

I note that the authors didn't follow all of my suggestions (the use of regressions to look at treatment differences, and explicit  hypotheses), but because I don't have (or am not able to find) access to all the materials, it may be that this issue was discussed and the suggestion discarded for good reasons. I also don't know what final analysis was agreed upon.

However, looking at the track changes document, it appears to me that the authors followed their proposed methodology, and clearly flag preregistered and non-preregistered analysis. Since they find no effect, some of their analysis (and my criticism of it) has become less relevant. In particular, the decomposition of the effect was the source of some criticism of mine in the first round (I thought they should use more directional tests in disaggregating any effects), but given the null result, the issue is largely moot.

I thus think the paper meets the criteria for the registered report. My apologies again for not going through all stages of the report, and hence this somewhat handicapped final evaluation.

Best wishes

Evaluation round #1

DOI or URL of the report: https://osf.io/9zvqj

Author's Reply, 15 Jun 2022

Decision by , posted 14 Jun 2022

Dear Philipp,

Thank you for submitting your Stage 2 Registered Report. We have now looked carefully at your manuscript and before we proceed with in-depth review, we would like you to address some comments/questions from the recommender Ljerka Ostojic (see attached).

In revising, please include a letter that responds point-by-point to each issues raised. As well as updating the clean Stage 2 preprint, please also include a tracked changes version of the manuscript. Add the tracked changes resulting from this pre-review revision to the previous tracked changes (between Stage 2 and Stage 1) keeping all tracked changes visible. To ensure that this latest round of tracked changes can be clearly distinguished from the previous changes between stages, flag the additional set of tracked changes in the manuscript as being specifically in response to this pre-review decision (e.g. using an additional comment in the margin associated with each update).

Once you have submitted your revised version and response we will commence the Stage 2 review process.

Many thanks,

PCI RR Managing board on behalf of Ljerka Ostojic

Download recommender's annotations