First of all, thank you very much again for submitting your paper and for the opportunity to review it. Despite this being a Stage 2 review, and have read through the whole paper and provided minor suggestions. In sum, my read of the paper that it followed the plan accepted at Stage 1 and the general content of the paper is acceptable. Most of my comments concern the way the results are presented, but  I think that the way the results are presented should be improved. Again, you need to note that I’m not an expert on mental accounting, so I could not properly review the validity of the statements concerning different theoretical aspects of the paper.  You can find below my specific comments: 

Abstract:

In XX “a” missing XXX Replication Registered Report with an American online Amazon Mechanical Turk sample using CloudResearch (N = 1007).

“Systematic replications and extensions of review articles using a single data

collection are a promising direction in revisiting seminal findings, mapping and examining

untested assumptions and predictions, comparing different designs and effects, and identifying possible links, gaps, and future directions“.

I would be happy to read some specifics in the abstract. Would be great to read about the conclusions. E.g., in the PCIRR study design table one main question is the link between different mental accounting problems. Or any results that the authors think 

Introduction:

“The earliest empirical evidence on mental accounting behaviors dates back to Tversky and Kahneman’s (1981) famous theater-ticket experiment (one of our replication problems).” Would be great to shortly explain what this experiment is, because the average reader probably wont know what it is, or alternatively just delete the sentence. 

“In Thaler (1985) and our target article-Thaler (1999),” is this correct? Shouldn’t this be “In Thaler (1985) and in our target article (Thaler, 1999)”

Instead of this “The two recent review papers cited very similar research to Thaler (1999), such as Heath and Soll (1996), Tversky and Kahneman (1981), Thaler (1980), and Thaler and Johnson (1990). This further exemplifies the necessity in…” I would suggest to change the wording, because it is hard to follow the logic: “A very similar set of papers formed the basis of these previous review papers, exemplifying the necessity ….”

“We also recognized the potential for improving on both transparency and methods.” To “We also recognized the need to improve both the transparency and method of some problems reviewed by Thaler”

Methods: 

In table 3 your write (and at other places you refer to as well) that the sample size in the study is 1007. Which is factually correct, but it is misleading because it ic compared with the sample size for different problems. But for the case of the present study, the true sample size is half of the study sample size (500 instead of 1000). I would suggest to use 500 as the reference number at each place suggesting that this is the sample who completed the each of the problems. 

Results:

Table 9 please put the original and the replication percentages next to each other to help the comparison of the two. (Apply the same logic for the other tables as well please).

In some of the table the font seems to be changing. (Even within the table).

Please provide at least a summary textual description of all the results main results for each of the problems, or some way help the reader to understand the results. Now, the results part basically is a few sentences and a lot of tables without context which is very hard to follow.

I would also add a summary table which shows which of the problems were replicated / or not. (So the content of the results are OK, but the way it is presented is not). 

“Therefore, we only compared the direction and relative magnitude of the mental accounting effects in some of the problems where it seemed to be meaningful.” Please describe your criteria for seem to be meaningful (or not meaningful) was.

“This exploratory analysis was an innovative and preliminary attempt to study the

connections among different subsets of the mental accounting framework. The results indicated that further explorations hold some promise.” 

I would take out the word innovative and leave it to the reader to make this judgment. It would be great if you could interpret the results at least to some basic extent beyond this very short and vague sentence. 

Discussion:

“Among the Problems, Problems 1, 14,…” -> ‘Among the problems’ is not needed. 

“…and the results were in the expected direction. The earlier loss could not induce risk-seeking in both Problems”. Is the expected direction that earlier loss could not induce risk seeking? 

“ … may cost more than $5 to drive 20 minutes to the other store with the increasing cost of driving. Therefore, the inconsistency may be due to participants' awareness of driving costs rather than a lack of mental accounting effect. It is also possible that over time since

it was conducted, the value of $5 in relation to transport costs has changed dramatically enough to shift participants’ preferences entirely.” To me this a very plausible explanation, as driving 20 minutes for sure costs at least 5 dollar these days. However, if this is true, I would not call this a non-successful replication, I would categorize this as miscellaneous as potential methodological concerns were raised after the data collection. 

“Taking into account the replication success of Problem 3, these together revealed that the sunk cost effect might be context-based.” What do you mean by context-based? I have a strong opinion that in psychology everything is context based so this statement is very vague. 

“.. sizes, and to try and map links between the different studies. We provided one such initial analysis focusing… ” See my related comments above. The conclusion is missing and the point why this is useful. 

“We recruited a much larger and more diverse sample than the original studies”… I would only say that “we recruited a larger sample.” I am not very much convinced that the sample is more diverse. 

Thank you again for your work!
Barnabas Szaszi

Dear authors,

Thank you for the opportunity to engage with your research again. My Stage 2 review is below. I am also attaching a pdf with some minor comments, mostly on wording but also on some specific interpretations. 

Review

The primary purpose of a Stage 2 review is to ensure that the pre-approved method and analysis plan was adhered to, and to assess that the interpretation of the results. Where fundamental flaws were missed at Stage 1, they can be highlighted in Stage 2; this is not relevant to this manuscript, where the reviewers’ concerns at Stage 1 were taken onboard by the authors and appropriate changes made.

Method

The authors provided a tracked-changes document. There are barely any differences from the IPA document, and where these differences appear they are incidental. For example, the additional changes to the Problem 10 design are minor and justified. 

Results

The pre-approved analysis plan has been followed. I will not comment on the significance of specific Problem replications because that is not the purpose of a Stage 2 review. The analysis procedure is sound and the description of results is comprehensive and transparent. Interpretation in the results section is kept to a minimum and deferred to the discussion.

In terms of departures, The IPA stated that LeBel’s interpretation criteria would be applied. In the end, the authors noted “we only compared the direction and relative magnitude of the mental accounting effects in some of the problems where it seemed to be meaningful.” A partial application of LeBel’s interpretation criteria is a sensible approach. 

A minor suggestion is that the authors develop a Figure to illustrate the change in effect size between the replication of each Problem and its original counterpart. For readability, these could be ordered from most positive change to most negative, like in a butterfly bar chart.

In terms of additional analysis – and feel free to ignore this suggestion – it would be interesting to see at the individual level how many participants could be described as ‘mental accounters’ i.e. their responses to at least 5 of the 9 Problems were aligned with Thaler (1999). In this study design, 9 problems were randomly chosen from a set of 18 for each participant, meaning there are 48620 possible combinations. This variation is helpful in attributing any ‘mental accounting’ tendency to the individual rather than there being particular sequences of questions that might amplify or nullify a tendency to display mental accounting traits. In fact, it is likely that few of the 1007 participants completed the same subset of questions.

Discussion

The characterisation of the replication as being mostly successful is accurate in my mind. Also, the authors show admirable restraint in bounding the discussion of implications. One suggestion is to add some specifics as to what avenues of research these results point to as holding promise. For instance, I was struck by the higher proportion of participants who selected the ‘indifferent’ response option. Indifference is the ‘rational’ prediction, so it would be theoretically interesting to pursue the causes of this change, if it occurs across different samples and contexts. However, the authors rightly warn against reading too much into specific differences between this replication and the original study, and perhaps I have just strayed into doing exactly that. 

Overall, this research is a valuable contribution to the body of knowledge on mental accounting, and it has been conducted in adherence to best practice in open science.  

Féidhlim McGowan