Mixed evidence for the hypothesis that sunk cost effects are weaker for time than money
Sunk cost effects for time versus money: Replication and extensions Registered Report of Soman (2001)
Abstract
Recommendation: posted 21 May 2023, validated 21 May 2023
Chambers, C. (2023) Mixed evidence for the hypothesis that sunk cost effects are weaker for time than money. Peer Community in Registered Reports, 100452. 10.24072/pci.rr.100452
This is a stage 2 based on:
Nikolay Petrov, Wenkai Song, Yin Kan (Megan) CHAN, Cheuk Nam (Chris) LAU, Tin Ho (Donald) KWOK, Lok Ching (Estelle) CHOW, Wai Yan (Vivian) LO, Gilad Feldman (gfeldman@hku.hk)
Scheduled submission: 2022-04-29
Recommendation
URL to the preregistered Stage 1 protocol: https://osf.io/65htv
List of eligible PCI RR-friendly journals:
- Experimental Psychology
- F1000Research
- Meta-Psychology
- Peer Community Journal
- PeerJ
- Royal Society Open Science
- Swiss Psychology Open
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.
Evaluation round #1
DOI or URL of the report: https://osf.io/wb2vy
Version of the report: 5
Author's Reply, 17 May 2023
Revised manuscript: https://osf.io/q9s6p
All revised materials uploaded to: https://osf.io/pm264/ updated manuscript under sub-directory "PCIRR Stage 2\PCIRR-S2 submission following R&R"
Decision by Chris Chambers, posted 14 May 2023, validated 14 May 2023
I have now received evaluations of your Stage 2 manuscript from two of the reviewers who assessed the Stage 1 manuscript. As you can see, Soman is fully satisfied with your submission in its current state, while Olivola lists several points of concern. Given the specific and limited criteria by which Stage 2 submissions are assessed at PCI RR, I want to provide some guidance on how to respond to these points.
Concerning the reviewer's point 2a (table that summarising the subset of prior studies that examined sunk costs involving time), this was not a required element at Stage 1; therefore I am not going to require this at Stage 2, although I fully appreciate the reviewer's point that it would be a useful synthesis for readers; I will therefore leave this at your discretion to consider.
Concerning point 2b, factual accuracy and clarity is vital so please ensure that this issue is fully considered and revised as necessary (even if it involves some minor changes to the Introduction).
Concerning point 3: I agree with the reviewer that this would be an interesting exploratory analysis. Since it was not part of the preregistered and approved Stage 1 proposal, and since these analyses are not (in my view) required for the conclusions to be sufficiently supported by the evidence (Stage 2 criterion 2E), it is at your discretion to include these or not. Please consider the point carefully and include in your response to the reviewer a clear explanation for your decision (one way or the other).
Concerning point 4: There seems to be a general tension with online studies between favouring participants with greater experience vs. favouring participants with a lower chance of prior exposure to key interventions; therefore I think this could be a useful point to consider in the Discussion (particularly if there is evidence that can inform the debate).
I will evaluate your revision and responses to the above points at desk before issuing a final decision.
Reviewed by Christopher Olivola, 04 May 2023
Unfortunately, the authors have (again) failed to address many of my comments and concerns:
Comment 2a: I still don't fully understand why the authors refuse to provide a table that summarizes the (relatively small) subset of prior studies that examined sunk-costs involving time. Doing so is not a huge endeavor (there are far fewer studies involving investments of time), yet it would provide a summary of all previous attempts to test time sunk-cost effects, and give us an idea of how replicable those effects are. I won't insist on this point, though I do think the authors are doing their paper (and readers) a disservice by limiting the scope of their paper to a replication of just a single paper examining sunk-time effects.
Comment 2b: The authors incorrectly claim that Olivola (2018) did not contain any sunk time cost scenarios. To the contrary, that paper contained *multiple scenarios* involving sunk time costs (e.g., the "cello lessons" and "potluck cake" scenarios), including scenarios used in papers that the authors *did* choose to cite. Therefore, there is no reason for the authors to not cite that paper as demonstrating a sunk-cost effect for time. Moreover, that paper demonstrates both intra- and inter-personal versions of time sunk-cost effects (i.e., that we also honor *other people's* time investments), which is notable. This omission by the authors is far less justifiable (and requires even less work to correct) than their decision not to include a summary table.
Comment 3: I’m glad that the authors have agreed to increase (specifically: double) their sample sizes. However, it seems the authors are resisting my suggestion that they *also* (not solely) carry out *additional* purely between-subjects analyses that only compare the first study/condition that participants were exposed to. The reasons for these *additional* between-subjects analyses were clearly explained in my prior review(s) and I don’t see a reason not to include them as well (in addition to the analyses the authors had in mind). The authors’ argument that “[their] experience has been that order had very little to no implications in [their] other PCIRRs and replications” is a poor one, and I can quickly counter it with my own experience, which has shown that exposure to prior studies (especially those examining the same concepts) can often impact subsequent responses.
Comment 4: The authors are wrong to claim that participants with 100 studies have no commitment and little reputational concerns. I’ve carried out several MTurk studies in which I limit the sample to workers with 100 studies or less (but with a high reputation score from those studies they did complete), and found that they are no less attentive than MTurkers who have completed many hundreds of studies. I won’t insist on this point, though I think the authors risk not being in a strong position to claim their study is a proper replication of Soman (2001), since his studies involved participants who were likely totally unfamiliar with sunk-cost scenarios, whereas experienced MTurk workers are likely to have been exposed to those scenarios (and debriefed about them).
Reviewed by Dilip Soman, 28 Apr 2023
The Stage 2 paper is ready for prime time! There were a lot of intriguing results - perhaps none more so than the differences in patterns of results across replicated Study 1 and Study 2. It does appear that time is a lot more malleable that most other resources, so the call for more research on what factors might drive the evaluation of time as a resource is a good one.
Thank you to the authors for their excellent work throughout the process!