Putting the Expected Value of Control (EVC) theory to the test in explaining habitual action
Motivational Control of Habits: A Preregistered fMRI Study
Recommendation: posted 08 February 2022, validated 08 February 2022
What are the neurocognitive mechanisms underlying the formation of habits? In this Stage 1 Registered Report, Eder and colleagues propose an fMRI study to test a key prediction of the Expected Value of Control (EVC) theory: that the dorsal anterior cingulate cortex (dACC) – a region heavily implicated in reward processing, cognitive control, and action selection – will show increased activity during the presentation of Pavlovian cues that are associated with devalued outcomes. In combination with a series of behavioural positive controls, this observation would provide evidence in support of EVC theory, whereas failure to do so may support alternative accounts that propose independence of habits from the representations of outcomes.
The Stage 1 manuscript was evaluated over two rounds of in-depth specialist review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA). This IPA recommendation was issued on 8 Feb 2022, and was initially provisional due to lack of ethics approval. The recommendation was then updated and confirmed on 21 Feb 2022 following confirmation that ethics approval had been granted.
URL to the preregistered Stage 1 protocol: https://osf.io/k8ygb
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.
List of eligible PCI RR-friendly journals:
Eder, A. B., Dignath, D. & Gamer, M. (2022). Motivational Control of Habits: A Preregistered fMRI Study. Stage 1 preregistration, in principle acceptance of version 3 by Peer Community in Registered Reports. https://osf.io/k8ygb
Chris Chambers (2022) Putting the Expected Value of Control (EVC) theory to the test in explaining habitual action. Peer Community in Registered Reports, . https://rr.peercommunityin.org/articles/rec?id=140
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.
Evaluation round #2
DOI or URL of the report: https://osf.io/52xpg/?view_only=551d1c39ed1f41549cbd35ec6b8bbed6
Version of the report: Prereg fMRI PIT revised_final
Author's Reply, 27 Jan 2022
Decision by Chris Chambers, posted 07 Jan 2022
The manuscript was returned to the two original reviewers, whose evaluations you will find below. As you will see, both reviewers continue to harbour major doubts concerning the theoretical framing of the study and severity of controls (Reviewer 1) and the sufficiency of the sample size (Reviewer 2).
In relation to the concern about sample size, I completely agree with the reviewer that point estimates of effect sizes from small pilot studies are likely to be unreliable, and thus the proposed sample size could very well generate inconclusive findings. On the other hand, I also do not see this as a fatal block for the current work. Given the costs involved, a sample size of N=38 is within conventional standards for fMRI studies (perhaps even toward the upper end), even if the "unit" of evidence produced by each individual study may turn out to be statistically weak. Indeed, due to the high risk of bias with fMRI studies (e.g. due to undisclosed analytic flexibility), in my view the RR model provides an especially important route for ensuring unbiased publication and a "course correction" for the neuroimaging literature.
I am, however, more worried about the sample sizes for the latter three hypotheses (manipulation checks), all of which are N=11 or less. That said, since they are manipulation checks, it is in some ways your risk to take as to whether you genuinely believe these samples are sufficiently large, because failure of critical manipulation checks is one of the few grounds for rejection at Stage 2 based on outcomes (see criterion 2A here). If you feel there is risk of failure in any of these checks, I would strongly advise increasing the sample size to avoid a rare Stage 2 rejection.
The concerns of Reviewer 1 are more fundamental to the acceptability of the current proposal, and I recognise they represent a strong difference of opinion between yourselves and the reviewer. In essence, the reviewer does not believe the design is capable of answering the research question, in part due to lack of an appropriate control but also a broader misalignment between the question/theory and the proposed methodology. It is difficult to see a way forward for this manuscript without resolving this disagreement in one way or another, and I am also keen to avoid overly burdening reviewers, especially when a discussion reaches a stalemate. Instead, I am going to offer you the opportunity to revise again. If you choose to simply rebut Reviewer 1's point rather than revise the design, I will seek additional specialist input to determine whether to accept or reject the proposal as submitted (with no further revision). However, if you believe you can sensibly revise the design to address the reviewer's concern once and for all, then I will invite Reviewer 1 back for a look before issuing a final decision. In either case, the next revision is pivotal and will determine whether in-principle acceptance is achievable.
Reviewed by anonymous reviewer, 06 Jan 2022
Reviewed by anonymous reviewer, 07 Jan 2022
Evaluation round #1
DOI or URL of the report: https://osf.io/dyt96/?view_only=5414b4189b2e4880ac614ec9a27807bf
Author's Reply, 17 Dec 2021
Decision by Chris Chambers, posted 29 Oct 2021
Two expert reviewers have now evaluated the Stage 1 manuscript. As you will see, the assessments are constructive but also very critical of a range of aspects of the proposed study. The most serious concerns are the lack of an appropriate control condition to be able to answer the proposed research question (Reviewer 2), insufficient methodological detail, and lack of clear links between the hypotheses, sampling plans, and analysis plans (Reviewer 1).
Concerning the remarks from Reviewer 1 about statistical power and evidence thresholds, due to its broad disciplinary remit, PCI RR does not set a minimum power requirement (see policy here) but please be aware that several of our PCI RR-friendly journals do set such requirements (e.g. Cortex requires a minimum power of 0.9 with alpha=.02 for all preregistered hypothesis tests; see full list of journals here). Therefore, if you want to preserve the option to eventually publish your RR in a PCI RR-friendly journal, over and above achieving a positive PCI RR recommendation, I suggest consulting the requirements of the potential outlet journals carefully. For the purposes of achieving a PCI RR recommendation, the main requirement is that the planned evidence strength is sufficiently strong to be able to provide a sensitive test of the hypothesis, and that the power analysis is linked precisely to the analysis plan. Reviewer 1 has significant concerns in this area which you will need to address in revision.
The major shortcoming identified by Reviewer 2 is the lack of a suitable control condition. As a non-specialist in this particular field, it struck me that addressing this rather subtle but central concern may require additional pilot data, but I will leave this with you to consider. Addressing the issue to the reviewer's satisfaction will be critical to achieving Stage 1 in-principle acceptance (IPA).
For a regular completed manuscript, concerns of this magnitude would lead to outright rejection. But of course the benefit of the Registered Reports process is that we have the opportunity to make critical design improvements before they become outright roadblocks. For this reason I hope you will find the reviews helpful in updating your proposal and progressing toward IPA.