We are almost there, but there is an inferential problem in your Design Table. For H2 (and similarly H3) you say "Therefore, not obtaining a significant effect will not be treated as evidence against the theory. " but this does not follow from your claim to have determined a minimally theroetically interesting effect. The point arises because you have not determined a minimally interesting effect, but rather an effect you happened to be powered to detect (which does seem very small, so it seems quite relevant to inference). The way to put it, if you are basing inferences on power, is that you will report the minimal effect you are powered to detect without claiming evidence against the theory as a whole, precisely because you have not determined a theoretically relevant minimal effect. The next column switches to a different inference procedure, what I call "inference by intervals". Namely you say "Conversely, if the lower bounds of confidence intervals are below the minimal interesting effect, then this indicates that performing the behaviour consistently is irrelevant for automatization." If you did have a minimally interesting effect, it would be if the upper bound of the CI was below the minimally interesting effect that the effect could not be larger than the miimally interesting; if the lower bound is below, your CI may span interesting and uninteresting effects, so one would withold judgment. But as you don't have a  theoretically justified minimally interesting effect, this does not quite work anyway. If you stick to power you can say simply a non-significant result does not count against the theory as a whole, but you were powered to pick up effects larger than X units difference.  (Incidentally I prefer inference by intervals, as you have in the interpretation column rather than simply relying on power, because it makes use of the data you actually obtained - the CI tells you what effects you are allowed to rule out in the light of the data, no matter what you were originally powered to do, without considering the data. But then inference hinges on the minimally interesting effect size, so for conclusions to related to theory, that minimal effect better depends on theory rather than on the theory-irrelevant issues such as a researcher's resources.)

Thank you for your excellent revision. Some of my points have not been yet fully taken on board; this is not surprising as RRs often require thinking in ways people are rarely used to.

1) minor point: p 4 " even more powerful"Given what has been said so far the issue is not power which is about testing hypotheses, but the precision of estimation (i.e. narrower confidence intervals). So it would be better to say your study will have more precision.

2) The main issues concern control of multiple testing; but especially how one draws inferences without justifying what minial effect size is of scientific relevance, for rows 2 and 3 of the Design Table. 

But first row 1. The final column of the first row is no longer accurate; you will not be testing Hull's theory.

In terms of "interpretation" in the first row, you can leave it as it is. But whether the best estimate is really exactly 66 is not really the point.  I would think if the overall confidence interval included any values in say the range 55 - 75, the original estimate was really pretty good. (Imagine you ran a million subjects and no CIs included 66, but they were all tightly formed around 61 days. One would still conclude that the original study had done a fine job of estimating the time taken.) So you mghit want to rephrase along these lines.

Second row. Three tests are suggested here. Will you conclude consistent performance is important if any one of them are significant? If so you should use a bonferroni or other familywise error correction. If not specify how you will draw conclusions - will you infer consistency is important only if all three tests are significant, for example? You do not justify a minimally interesting effect size i.e. give reasons why there is minimally interesting effect relevant to this scientific problem. Thus, a non-significant result does not count against any theory. Thus, the inference in the final column is incorrect.

Third row.  Be clear about which family you're correcting for: Are you correcting for the fact you are looking at 5 DVs? Or for pairwise comparisons for each of those ANOVAs? State how you will correct for each.  The same issue mentioned for row 2 also arises here, you find it hard to give reasons for why some effect is of minimal interest. That is not surprising, but it does mean a non-significant result is not support for any conclusion.  What if an effect were interesting that were smaller than even the small ones you are powered to detect? Then you haven't got evidence against any theory.

So one has two choices: either give scientific reasons for why an effect size is relevant (in the case of the significance testing you are doing, that means a minimal interesting effect; see https://psyarxiv.com/yc7s5/); or else forgo any claims that one can find evidence against an effect existing. In the latter case one could simply estimate relevant mean differences and draw no existential conclusions. So for your second and third rows you could just estimate relevant mean differences and make claims like: given there is an effect, it is between these bounds (i.e. a 95% CI).  The CIs would still be corrected for multiple testing. If you go this way, do not slip in any claims of having shown there is a an effect or not effect, just stick to saying how big it is.

A  third option is that one suspends judgment for all non-significant results. This was a position Fisher himself sometimes took. I personally don't like taking this option because it means there is no way of getting evidence against a theory that predicts an effect. Many RR journals will not accept it either.
 

Dear Lukas

I now have three positive reviews back for your submission, which are as a whole enthusiastic about the planned research.  In addition to other points raised by the reviewers, also address the following in your revision:

1)  Wood raises the issue of whether data has already been collected. You say in your cover letter that it has not, but the verb tense used in other places implies it might be otherwise e.g. "The detailed study protocol, materials, anonymized raw data, code used in the analyses and output are permanently stored on Open Science Framework (https://osf.io/n6srx/)" (which I didn't have access to), and the frequent use of past tense for Method. To be clear, assuming data have not been collected, for Stage 1 use future tense throughout for things that have not happened, including Method (and e.g. "the data will be permanently stored.."). Future tense can then be changed to past tense in all cases when the Stage 2 is submitted. (Of course if data have been collected, than clarify this as well, and also therefore what precautions are in place for bias control.)

2) In the results section indicate *exactly* what analyses you will do. The specification should be so clear that there is no analytic flexibility left. Make sure for example that anyone could fit the curves exactly the same way you will, and so they could precisely reproduce your results with your data. Be clear what the other "plausible curve" is that you will fit in addition (I know you refer to the original authors, but this information should be in your mansucript if it figures in your analytic pipeline.). be clear exactly what comparisons are done with what error control (more on this below). Another example "We will also perform multiple regression analyses to determine whether impulsivity, personal need for structure and conscientiousness were related to curve parameters and performance variables." State how many regressions you will perform. Specify the variables for each regression. If you mention analyses in the results section also put them in the Design Table and justify power for each analysis and what conclusions hang on each. Alternatively, leave out mentioning any analyses you do not want to tie down exactly (nor make sure are properly powered) and put them in a non-pre-registered section in the Stage 2. (There are other cases where the analysis needs to be tied down I have not mentioned.)

3) Relatedly, in comparing linear and s-shaped functions you say you will do a sign test on R2's. But you also provide cogent reasons for why this is a flawed strategy, and hence say you will also use BIC and AIC to aid the reader. This leaves a lot of inferential wriggle room. A possible outcome is higher R2 for the s-shaped than linear function because of the difference in parameter numbers, but BIC comes down the other way. Has your manipulation check on the form of the function failed or not? Justify one measure as most suitable  - and I presume it will be either BIC or AIC or related, e.g. an informed Bayes factor - and test the form of the function with that measure, stating your criteria for good enough evidence for choosing s-shape over linear functions (or one s-shape over another).

4) As per my previous point, it seems to me that providing evidence for your exponential function is more an outcome neutral test than a test of theory. You point out that the original authors were motivated by Hull's theory of habit; and thus you frame the choice of function as testing Hull's theory, by testing the predicted shape of the function defining the increase of habit strength over time. However, strength will be measured on Likert scales, therefore with fixed minimum and maximum values. A linear function is therefore a priori ruled out if testing continues long enough. Something at least approximating an s-shape is guaranteed by the nature of the measurement. Therefore no theory can be at stake depending on the outcome of this test. Rather, obtaining something like an s-shape is a necessary precondition for your study in order to estimate when a habit has formed. 

5) I take it you plan to perform all pairwise comparisons between the 5 data sets (your 4 plus original) with Kolmogorov-Smirnov tests, which is 10 tests. How will you control familywise error rate?  Given that corrected alpha, determine N to control power to detect a difference you are interested in. How far apart should median number of days be before it is interesting? As Gardner points out, 66 days is just a rough figure; 59 is in the same ball park. Is there any way (other than reaching deep in one's soul) for specifying how far away from 66 would start to be interesting? (e.g. https://psyarxiv.com/yc7s5/ ) Once you have justified an interesting difference, use simulations to determine the power of KS to detect the effects that are just interesting. (And you should note in the paper that KS is sensitive to more than location differences, as a proviso on your analysis.)

6) For RQ3, specify an interesting effect size in raw units:  What difference in rated automaticity would be just interesting? (When several LIkert ratings are combined, I find using an average rather than sum over number of ratings useful to put the final number on the same scale as the rating itself, so one has a more intuitive grasp of what one unit is.) Otherwise we just have "medium effect size" plucked out of the air; and being standardized it depends on measurement noise and reliability. But presumably what is interesting is the actual difference in automaticity.

7) In terms of what defines an interesting effect, Takacs asks "for RQ4, would a non-significant result prove that “complexity is irrelevant for automatization”?" You can just qualify the conclusion that it applies to this particular difference in complexity. (If in addition you could qunatify or measure the complexity difference (and I am not requiring you do) it would help place the conclusion in perspective too.)  How will you control familywise error rate for number of Dvs? Determine interesting differences in raw units, then determine power for those diferences, taking into account the corrected alpha. Specify the IV and its levels - is it more than 2 as you are performing a KW? Will there be post hocs? What conclusions follow from different patterns? There is also inferential flexibility in specifying both ANOVA and KW. Justify one, or provide a decision procedure for choosing between them (one that does not allow wriggle room).

best

Zoltan

Dear Dr de Wit

Thank you for your clearly written submission "the shape of habits". Before I send to review, one point needs to be addressed. Namely, you should demonstrate sensitivity to answer each question in your Design Table; or put another way, you should be able to, or at least know whether you can, obtain evidence against any existential claims made. At the moment, N is determined with respect to one issue, namely the precision with which the median number of days is estimated. But you also ask a number of other questions, and it needs to be shown whether you have the power or sensitivity to answer each of them, considered on their own terms. Some journals require a certain power or Bayes factor threshold for any hypothesis testing; and for those that have no set requirement, it should be known whether you did or did not e.g. have the power to justify asserting H0 (or do or do not have the resources to achieve good Bayesian evidence). Some advice is given here for approaching the problem: https://psyarxiv.com/yc7s5/ For each row of your Design Table, show that you will (or will not) have the means to obtain evidence that would show the claim wrong if it were wrong.

best

Zoltan