I would like to thank the authors for their detailed responses and useful clarifications. I understand that they have to make some choices, given the many parameters that might impact on the reliability of test-retest fMRI data. I look forward to reading the main results in the next stage.  

The authors have addressed my questions well. I have no further comments. 

I appreciate the authors’ revisions of their registered report. I generally found them quite receptive and responsive to the suggested adjustments made by the reviewers, and that the report is substantially improved. I am eager to see the results of this work!

My only remaining comment regards the generalizability of the results from the current study and how the authors plan to address this issue. I though the authors’ motivation to limit analyses to the MID task was reasonable (given the broad range of analyses already proposed and the ubiquity of this task in the field). However, I also think that it is very unclear whether the results of this study are likely to generalize to other task contrasts in very different domains. As such, it seems important to me that the authors acknowledge their specific focus on the MID task further in their abstract and introduction. Currently both read as very generic investigations of reliability in task fMRI, and the specific task focus is only mentioned quite briefly/late (pg. 7 in the introduction and in parentheses in the abstract). A clearer and earlier statement regarding the focus on the MID task and why would be helpful for appropriately contextualizing the findings.

The aim of this registered report is to investigate how certain analytic decisions affect individual and group-level reliability in task fMRI using three independent datasets and 360 different modeling permutations. While the report is well-written, I have a few minor comments.

For estimating reliability at the group level, the authors plan to use "significance thresholded (p< 0.001, uncorrected) group [binary] estimates." This is not a common threshold strategy reported in publishable studies. To enhance the applicability of the results, the authors may consider using a more commonly used group analysis threshold strategy e.g., one that combines an uncorrected p threshold (e.g. 0.001) and a cluster size threshold. 

The authors may clarify why two metrics (Jaccard's similarity coefficient and tetrachoric correlation) are planned to use and whether they provide complementary information on reliability results.

“the Aim 2 Hypothesis is that, on average, there will be higher between-subject than within-subject variance across the three samples within task-relevant regions.” The wording of the Aim 2 hypothesis is a bit confusing. E.g., we may never expect between-subject variance to be lower than within-subject. The authors may consider rephrasing it to make it clearer. Additionally, it would be better to clarify how the "task-relevant regions" will be defined. 

“Aim 3 evaluates at what sample the ICC stabilizes using the most optimal pipeline (e.g., highest ICC) from Aim 2.” Is it correct that all samples will be used for Aim 2, and only smaller subsets of the samples will be used for Aim 3? 

Year 1 and year 2 data were used for each of the three datasets. As reliability estimates may decrease with the time lag between two scans, I would suggest the authors report the exactly time lag in each dataset. 

“Structural and functional MRI preprocessing is performed using (?) 23.0.0rc0”. The package name is missing. Is it fMRIPrep? 

The meaning of A/B/C/D in Figure 1c and Equation 3 is unclear. In addition, there seems to be some confusion regarding the two "Ses-1" in the first row. The authors may consider clarifying these points to enhance the readability of the manuscript. 

In this Stage 1 registered report, Demidenko and colleagues propose to examine how different options in the fMRI analysis pipeline affect reliability of task activation results. Specifically, the authors propose to examine the effects of different levels of smoothing, forms of motion correction, methods of task parameterization, and types task contrasts influence the reliability of results. The authors state they will examine this question using two runs of Monetary Incentive Delay task data (anticipatory period), from 3 different datasets, with sample sizes of 65, 150, and 2000. A combination of ICC and Jaccard overlap similarity will be used to assess similarity. Aim 1 proposes to determine which “multi-verse” analytical set of options produces the most reliable results, Aim 2 proposes to separately examine how analysis steps affect the size of within and between subject variance, and Aim 3 proposes to investigate how sample size affects the reliability of the optimal analysis pipeline.

In the past 5 years or so, a number of papers have noted the low reliability of many fMRI results. This low reliability likely limits the ability to interpret fMRI data and use this data to assess brain-behavior relationships, a topic that has also received recent scrutiny. The current planned project represents an important contribution in this domain, in terms of addressing the extent to which different analysis options will influence reliability. I generally found the report to be well written and analyses to be carefully explained, with hypotheses and tests well laid out. I have a few suggestions, however, that I think would increase the impact of this work.

Major suggestions

-       My biggest concern is that the current work is based on only a very narrow set of data: anticipatory cue periods from the monetary incentive delay task. This will strongly limit the generalizability of the current results – it is unclear if the findings from this particular work would have any bearing on many other potential tasks that could be run. Given the premise of the article, it’s not clear why the authors chose to limit themselves in this way. This work would have a much bigger impact if the authors were to test the pipelines across a range of tasks (e.g., each of the tasks in the ABCD, HCP). It may not be possible to apply all of the combinations of analysis steps in the new tasks (e.g., the different versions of task parametrization), but even if only a subset of analysis options were repeated, it would be very helpful for understanding the degree of generalizability of the current results to other contexts. If the authors choose to keep their focus narrowly on the MID, this should be reflected more clearly in the title and abstract, and in discussing the types of conclusions that can be drawn from the current work.

-       The authors discussed reliability issues at length, but did not touch on the topic of validity. These usually go hand in hand, as methods to increase reliability at times reduce the validity or utility of a dataset – for example, consistency of datasets is likely to increase monotonically with smoothing, although this will come at the cost of being able to distinguish meaningful/valid results from distinct areas of the cortex. Similarly, different task contrasts will have different levels of validity in terms of the underlying cognitive constructs that they can be connected with. At a minimum, I think this bears additional discussion in the final manuscript. If the authors could also propose a confirmatory test of validity (e.g., using a motor task to find distinct components of the motor system), that would also help with the interpretation of the results.

-       I was surprised that the authors did not consider the amount of per subject data (rather than the total number of subjects) in their evaluation of the reliability, given that recent reports have consistently found this to be a primary way of improving reliability of the findings. This might make sense if the focus of the work is only on analytic means to improve reliability, but given the proposed analyses in Aim 3, it seems reasonable to ask whether the amount of data per subject or number of subjects makes a bigger difference for reliability (I would guess the first).

-       In a similar vein, the current work is based on a very small amount of data from each participant (comparison of single runs), and for a set of contrasts that likely have relatively small effects (e.g., relative to sensory/motor stimulation). I am concerned that the authors will be at floor in terms of reliability, preventing them from finding meaningful differences across analysis strategies. Is there any evidence that this will not be the case? Including a positive control dataset with more data per participant would help to alleviate this issue. The hypothesis that between subject variance is larger than within subject variance may not be true in the context of this very small amount of data.

Minor suggestions

-       Effects of younger vs. older samples may be influenced by study design (e.g., fMRI imaging parameters, block lengths, etc.) that are not matched across datasets. How will this be addressed? It seems like it may be challenging to interpret differences across datasets.

-       Table 1: #5 says “censor high motion frames” but not what the cut off for these will be? This should be specified in the report.

Title: Test-Retest Reliability in Functional Magnetic Resonance Imaging: Impact of Analytical Decisions on Individual and Group Estimates

By: Demidenko and colleagues

This is an interesting and well-written report that addresses an important question about the reliability of fMRI-based brain mapping with respect to the relative size of intra-subject (inter-session) and inter-subject variability. The main strength of this paper is the systematic comparison of multiple combinations of different methodological choices (almost 360 different parametrizations) in fMRI data analysis while task is held constant. The authors also considered the use of different datasets with variable sample sizes. Overall, the topic of this paper is important and relevant to current debate(s) about the consistency of fMRI data analysis procedures.

I have the following comments about the current version.

First, while the topic is important, the novelty of this work is not clear. There is already a massive literature about test-retest reliability in fMRI in different domains (visual, motor, language, memory, decision making…etc.). Many previous test-retest studies are not cited in the current version. I believe it would be useful to discuss reliability with respect to its origin/source and its implications in fMRI: e.g. differences due to situational factors (e.g. Dubois and Adolphs 2016 TICS) or ‘meaningful’ differences that reflect inherent cognitive and behavioral differences (Seghier and Price 2018 TICS). For instance, if task performance would vary across sessions, should fMRI activations vary as well to reflect such differences in task performance? My point here is that test-retest reliability in task performance (behavioral data) should not be overlooked when explaining test-retest reliability in fMRI (as long as there is variability in behavioral data there will be variability in fMRI data). 

In the same way, performance during the selected MID task varied across runs and subjects. The targeted accuracy was set to 60%, which is relatively low (is the task challenging?). How the first-level GLM models will be defined? The authors mentioned in Page 12 that the design matrix included 15 task-relevant regressors (5 cue and 10 feedback types), but it is not clear if all trials will be included or not? For example, it is likely that correct trials might show different activations (and different reliability) than incorrect trials. Again, task performance is a huge confounder in such test-retest type of analysis and thus the authors should clarify how correct versus incorrect trials will be modelled.

As the authors know, test-retest reliability may vary with task. For instance, it is likely that tasks involving primary sensory cortices might show higher reliability than tasks activating high-level processing regions. Even for the same domain, reliability might vary with activation condition (e.g. see for the language domain: Otzenberger et al. 2005). In this context, I’m not sure what is the exact impact of the expected findings on current practices; i.e. the best parameterization (or combination of methodological choices) might not generalize to other contexts that (1) use different tasks and cognitive domains, (2) different populations, (3) different time points between repeated sessions, (4) different magnetic fields, (4) different paradigms (block, event-related, mixed…etc), (5) different acquisition protocols (high temporal or spatial resolution EPI), (6) variable run/session length (e.g. Genovese et al. 1997 MRM) …etc. There are so many factors that can impact upon the BOLD signal and its reliability. Also, there are already many guidelines (recommendations) about optimal pipelines and analysis protocols to improve reliability and thus it would be nice that the authors could spell out the expected implications to current practices. 

I find the inclusion of task-irrelevant voxels slightly confusing. Basically, task-irrelevant voxels are just the complement set of task-relevant voxels (assuming relevant voxels + irrelevant voxels = the whole brain as a constant). So, I’m not sure what would be gained by that inclusion, in particular when using binary maps to calculate the similarity metrics like Jaccard. Note also that the definition of task-irrelevant voxels is contingent to the selected threshold and the used baseline/control condition. 

Some statements need to be integrated in a more coherent way; e.g. Line 161 “there will be higher between-subject than within-subject variance across the three samples within task-relevant regions” and Line 329 “the reliability within sessions would be hypothesized to be greater than between sessions” versus Line 395 “To test whether the between-subject variance is lower than within-subject variance consistently across the three samples”. 

In Line 735, I didn’t get what the authors meant by this statement: “so it could be the case that a less efficient model captures the data variability better and the estimated variance of contrast estimates is reduced.” How this statement should be understood with respect to modelling efficiency of Figure S2?

In the same way, Line 134 “However, there is little empirical research on whether the culprit in the reportedly low reliability of fMRI signal across measurement occasions is a decreased between-subject and/or an increased within-subject variability.” If there is a decrease in the between-subject variability, it would then make sense to expect a subsequent increase in reliability. I feel there is a confusion in the current version (at least to me) about the exact impact of low/high variability toward low/high reliability that should explained beyond the issue of reliability quantification with ICC. Put another way, ICC is just one type of objective measures used for the assessment of reliability, but the relationships between intra/inter-subject variability and reliability in general is much broader than that. 

In the study of Kennedy et al. (2022) with the ABCD dataset, the authors assessed reliability and stability for short term (within-session) and long-term (between-session) and reported overall poor reliability and stability. Please add a statement about how the current study can help make sense of Kenedy et al.’s findings.    

Jaccard similarity index might show low values for small activated volumes, which means that Jaccard values can decrease with more conservative thresholds. The issue of thresholding is thus critical when calculating Jaccard.   

Equations (1), (2) and (3): explain what these abbreviations/parameters stand for? (In addition to what is already mentioned in Figure 1).

Line 456: how the sample size N=60^5 was defined? 

Explain what the tetrachoric correlation will add on top of what one can get with Jaccard (both indices use binary data). 

Figure 1b: Jaccard index (i.e. Intersection over Union) can be simplified as Red divided by the sum of Red, Blue and Green (no need for the two extra Red squares in the denominator of 1b). 

Table S3: TE of the BOLD run for MLS/GE Sigma looks odd (3.6ms !).