When do perceptions of wastefulness affect how people make choices?

Dear Dr. Feldman,

Thank you for submitting your revisions and response to the reviews for the Stage 1 registered report entitled “Revisiting the Psychology of Waste: Replication and Extensions Registered Report of Arkes (1996)”.

Overall the revisions were comprehensive and highly responsive to the points raised in the reviews. There are two remaining minor issues that I’d ask you to address:

1. In my view the procedure for attention checks should be documented in the manuscript well enough for a reader to judge the level of protection against low effort or automated responses, especially when it results in the exclusion of some participants. You point out in your response that failing the attention checks causes the survey to end and the participant is prevented from starting over. I would ask that you simply add this detail to the Procedure section rather than require a reader to consult the survey itself. The description also does not mention the “copy-paste” question, which I would note *does* allow a participant to correct their response if they carelessly skip past it or don’t follow the instruction, so it’s not clear that this question is serving the same purpose as the others in screening out inattentive participants.
2. The other remaining issue is the plan to conduct analyses of order effects only in the case that you fail to find evidence for the hypotheses. I don’t think order effects are akin to other extraneous factors like age and education—indeed you anticipated the possibility of order effects impacting the replication outcome by planning for follow-up analyses. I do, however, appreciate the concern that reporting all of these additional analyses from the outset would hurt the readability and interpretability of the paper, and agree that these costs may not be worth it considering that this is a nuisance factor unrelated to the theory being tested. But part of the reason for raising this issue is that, just as you mention in your response, these analyses bring some additional analytic flexibility, and the current description of the "pre-registered" plan in the Order Effects section doesn’t do much to limit that flexibility (specifically on the question of how a moderating effect of order would be tested). If you can provide more details about the plan for these tests at this stage then I would recommend including it now. If you choose not to, the expectation would be that these analyses are appropriately caveated as being exploratory in the stage 2 submission.

After these final points are addressed I expect to be able to quickly move on to a IPA for this submission.

Best wishes,
Doug Markant

Dear Dr. Feldman,

Thank you for submitting your Stage 1 registered report entitled “Revisiting the Psychology of Waste: Replication and Extensions Registered Report of Arkes (1996)” to PCI: Registered Reports. I have now received comments from two expert reviewers and have also read the report myself. Overall, we’re in agreement that your submission has several strengths, including a good justification for conducting a replication of the target article, well-documented plans for the study and clear criteria for evaluating the outcome of the replication, well-justified modifications to the original article, and a number of proposed extensions that improve upon the original study’s methodological rigor.

Based on my own reading and reviewers’ comments, I’ve summarized below the main points that should be addressed in a revision prior to an IPA. 

Major points

1. Some aspects of the planned analyses need to be clarified. The description of the analyses starting on pg. 34 does not entirely match the simulated results. As reviewer TC notes, it’s not clear how you’ll use the ANOVA to draw conclusions about the role of the different reasons (relative to utility maximization) in wastefulness judgments. In general, the “Data analysis strategy” section would be improved by reiterating the hypotheses that are targeted by each set of tests (especially for the proposed extensions). The Bayesian analyses need more explanation about the approach and justification for the choice of prior (see also reviewer QA’s suggestion about an alternative that doesn’t rely on Bayes Factors). Reviewer TC also points out some inconsistencies between descriptions of the analyses of perceived wastefulness.
2. Given that the within-subjects design is a major deviation from the target article, I agree with reviewer TC’s suggestion that an evaluation of order effects should be carried out regardless of the outcome of any other tests. If there is an effect of order, examining the scenario that is presented first in a follow-up analysis seems sensible but will change the sensitivity to detect the target effect sizes and may leave you with a smaller sample than the minimum of 240, so an alternative analysis approach that accounts for order might be preferable.
3. The purpose of the attention checks at the beginning of the survey is somewhat unclear. It sounds like participants will have to answer Yes to each of them in order to proceed. My impression is that if participants fail the attention checks they are given a chance to correct their response (although I couldn’t quite tell from the survey flow). Perhaps this prompts people to pay closer attention at least momentarily, but it seems there’s no plan to exclude people based on whether they initially fail these questions, correct?
4. A similar comment applies to the comprehension checks (see point by reviewer QA). It appears that the plan is not to exclude anyone based on these checks either, but rather to give them as many opportunities as they need to answer correctly in order to move on. Given the complexity of the scenarios I would be concerned that some participants are going to simply cycle through the options without understanding the scenario (and the risk of this might increase with later scenarios). This is a concern especially with the change from an in-person study in the target article to an online replication—if the replication fails, a natural question will be whether it’s due to the online setting. In follow-up analyses it would be reasonable to use responses on the comprehension questions to divide the sample into groups. If you foresee going down that route, I’d recommend specifying at this point how you would approach that (e.g., deciding now what would be a reasonable number of incorrect responses where you would classify someone as an inattentive respondent). If not, please address how you will otherwise guard against low-effort or inattentive responses of the kind commonly seen in online samples.

Minor points

1. The Introduction is generally well-written and clearly organized, but there are a few points that are a bit too terse and need some clarification.
   1. Pg. 8: “Our secondary goal was to build on the target’s design and add extensions to refine the target’s methods and gain further insights.” — Although this information comes at a later point, it would be helpful to give a little more of a preview here of the extensions and the overall motivation for them.
   2. Pg. 15: “Given that the coding procedure was unclear and the process noisy…” — In what way? The preceding paragraph doesn’t seem to explain these limitations, apart from reasons not being measured in Study 2.
   3. Pg. 17: “we were concerned about a possible discrepancy between the Arkes’s (1996) conceptualization of the concept of wastefulness, and the laypersons’ perspective of wastefulness.” — What was this concern?

2. Clarify early on that the study will use a within-subjects design (see point by reviewer TC). I too found the phrasing about a “unified” design to be ambiguous and to cause some confusion at some points. In addition, I’d strongly recommend that for the planned experiment the authors describe the three tasks as separate “scenarios” (or something similar) rather than “studies” (“studies” is appropriate when describing the target article, but is confusing here given the within-subjects design).
3. Please address the question by reviewer TC about the “Likelihood” measure, where the simulated mean appears to fall outside the range of response.
4. Assuming that the red text is meant to be deleted for the Stage 2 submission, I’d suggest including some or all of the text at the top of pg. 20 about pretesting and incentive pay as part of the main text (e.g., the planned pay rate and survey duration). The two sections also seem to repeat some of the same statements and could be consolidated.
5. Comprehension checks are described in two places (first on pg. 26 then on pg. 28), but they appear to be describing the same questions. I recommend consolidating these sections so they are described in one place.
6. See the suggestions for improving the figures made by reviewer TC. NOTE: Although I agree with both points, given that the figures will be updated in the Stage 2 submission along with the rest of the results I don’t view these as necessary changes prior to IPA.

Best wishes,
Doug Markant