Recommendation
DOI or URL of the report: https://osf.io/4ps8m/
Version of the report: v2
Please see detailed review in attached file.
We believe that we were able to answer the questions and address the concerns without requiring further changes to the manuscript. Therefore, no changes have been made to the manuscript, and we did not update the preprint.
, posted 16 May 2022Thank you for your careful and thorough revision and response. One of the reviewers from Stage 1 was available to evaluate the revised manuscript within a short timeframe. As you will see, the reviewer is largely satisified, with just a few remaining clarifications to make in a minor revision. We should then be in a position to offer Stage 1 in-principle acceptance (IPA) without further in-depth review.
For the reviewer who was unable to assess the manuscript within 14 days, I have considered your revisions myself and found them sufficient to proceed. In the event that IPA is awarded, I will invite that reviewer back in due course to evaluate the Stage 2 manuscript.
, 09 May 2022I would like again to thank the authors to take the time to review the manuscript based on the comments. In general, I believe the authors sufficiently addressed most of my concerns. Below I only focus on the comments where I feel that further changes would improve the quality and impact of the proposed study.
You suggested the following improved sentence to be included in the abstract:
“Out of the 17 mental accounting hypotheses, we found empirical support for X with effect sizes ranging from X.XX [X.XX, X.XX] to X.XX [X.XX, X.XX], and no empirical support for Y with effect sizes ranging from X.XX [X.XX, X.XX] to X.XX [X.XX, X.XX].”
Although I understand the reasoning behind the sentence, to me it is somewhat strange to dichotomize the results and the effect size ranges that way. Is it interesting what are the exact ranges for the different groups? I think rather the number of hypotheses with support and no support, and the general effect size range is interesting, but independently. However, I have no strong opinion on that.
You write the following sidenote response regarding how big a 0.23 effect size is.
“Sidenote: From Cohen (1988) to the more recent and social-psychology specific Lovakov and Agadullina (2021) a large effect for Cohen’s d is 0.65-0.8, a medium effect 0.36-0.5, and a weak effect 0.15-0.2. A Cohen’s d of 0.23 is considered a weak effect (35th percentile in the entire literature). The newly targeted 0.29/0.36 is considered a weak to medium effect.”
It does not concern the acceptability of the present paper, but I cannot stop myself from saying that I still disagree and leaning toward the effect size interpretation of https://journals.sagepub.com/doi/full/10.1177/2515245919847202, that is 0.29/0.36 is rather a medium to large effect size.
To me the last part of this sentence seems to be confusing:
“Our second goal was to examine several predictions made by Thaler regarding mental accounting behaviors that the review did not cover empirical tests for.“
You write that “ In a pre-registered study with an American online Amazon Mechanical Turk sample (N = 1000)”. Is that n=1000 the number of people starting the experiment OR participants providing a full response to all questions OR the target number after exclusions?
Again, thank you for the interesting submission!
Best regards,
Barnabas Szaszi
DOI or URL of the report: https://osf.io/2g6a8/
Revised manuscript: https://osf.io/4ps8m/
All revised materials uploaded to: https://osf.io/v7fbj/ , updated manuscript under sub-directory "PCI-RR submission following R&R"
, posted 24 Mar 2022Two reviewers have now evaluated the Stage 1 manuscript, and at the outset I would like to thank them for providing such generously detailed and constructive assessments. As you will see, the reviewers are broadly positive about the prospects of the replication but also raise a lot of major concerns that will need to be resolved to meet the Stage 1 criteria. The majority of issues focus on the rationale and level of detail surrounding the predictions and analysis plans, methodological concerns surrounding bias control and sample quality, and overall coherence of the presentation.
Reviews of this quality represent the RR format operating at its best because they provide authors with the opportunity to clarify and correct major issues before it is too late. In this case, a significant amount of work lies ahead to achieve in-principle acceptance (and you may decide it is too much); however I would nevertheless like to offer you the opportunity to address the concerns in full.
A revised manuscript will be returned to the reviewers for further in-depth evaluation.
, 23 Mar 2022Dear authors,
Please find my review attached. I hope you find the comments constructive.
Download the review