Testing the replicability of dynamic functional connectivity correlates of cerebral small vessel disease in the Hamburg City Health Study

I read the authors responses with interest and just have a couple of additional thoughts, which do not necessarily require a response.

In response to the second point about how missing data will be handled, the authors state that they will be carrying out a complete-case analysis. In the final report from this work, it would be good to see a justification of this choice. I would suggest that the authors eventually report how many participants were excluded from the sample due to incomplete data and consider any biases that this might introduce (if any).

In response to the third point regarding additional exclusion criteria, the authors state that they will not exclude participants with dementia. It would be beneficial to explicitly state this in the final report, and state the proportion of the sample with a dementia diagnosis (if available, if not, this is a limitation), in order to better characterise the sample and contextualise the findings.

Best of luck to the authors with the submission of their work!

The authors have addressed my previous comments. The pre-registration has been improved as a result of these and the other modifications.

The author presents a pre-registration of a replication study to examine associations between dynamic resting-state fMRI, small vessel disease (WMH), and executive function (TMT-B). The research question is scientifically valid, but the theoretical rationale for the hypothesis (specifically regarding the focus on dynamic FC and high-occupancy states) requires further clarification and justification. The sample and proposed methods are mostly appropriate and feasible, but I offer some suggestions for improved clarity and rigor.

1.     Line 47: The author suggests that limited reproducibility of functional connectivity “in the presence of cSVD” might contribute to “heterogeneous” findings in the prior literature. I believe it is important to acknowledge here that functional connectivity measurements are generally quite noisy with low reliability, even in healthy control samples, especially when measured with short (~5 minutes) resting-state scans (e.g., Laumann et al., 2015, Figure 4), which are quite common in clinical studies, including the present study. This is not a problem specific to cSVD or any clinical population.

2.     The justification for choosing to focus on dynamic FC as opposed to traditional FC is not clear. Given the context of the paragraph beginning in line 40, it seems to imply that dynamic FC might be more “reproducible” or less “arbitrary” than traditional FC. However, this is counter-intuitive, as dynamic FC has not been demonstrated to lead to either of these improvements. If anything, dynamic FC is likely less reproducible than traditional FC in the sense that resting-state scans are broken into even shorter “state” bins. If the argument is that dynamic FC is more likely to be relevant for cognition, this claim has also been disputed in the literature, as dynamic shifts in FC have been suggested to arise from non-cognitive processes, which should be acknowledged as a potential interpretation of the data (Laumann et al., 2016; Laumann & Snyder, 2021).

3.     What is the expected final sample size after excluding participants who were already included in the prior study?

4.     The Trail Making Test - part B is selected as a measure of executive function. However, unless performance is adjusted for part A, performance on part B alone may be driven in large part by psychomotor processing speed (Arbuthnott & Frank, 2000).

5.     The proposed clustering analyses will use a default k of 5, presumably based on the result from the author’s previous paper. However, I believe it is important to test whether this result replicates in the proposed analyses. A range of k values should be tested and compared to find the best fitting number of clusters, rather than selecting a value a priori.

6.     In addition to the demographic variables, variables relating to resting-state fMRI data quality and motion (e.g., mean FD, % frames retained in scrubbing) should be reported and included as statistical covariates.

7.     Table 2. It is not clear which pipelines will include GSR, e.g., GSR should be included in the Power and Satterthwaite pipelines if consistent with the references, but it is not listed in the table.

8.     Hypotheses regarding the dynamic FC are mostly framed in terms of high vs. low occupancy. For instance, the separation of 2 high-occupancy and 3 low-occupancy “states” is shown to be robust in the multiverse analyses, but are the regional network configurations of these states also consistent? Is the optimal solution of k = 5 consistent? These questions seem to have more importance for theoretical interpretation, whereas the observation of greater occupancy in states defined as being “high-occupancy” is nearly circular. Why is meaningful that WMH would be associated with dwell time in a state defined by its occupancy rate, as opposed to a state defined by its pattern of network configuration?

References

Arbuthnott, K., & Frank, J. (2000). Trail Making Test , Part B as a Measure of Executive Control : Validation Using a Set-Switching Paradigm. Journal of Clinical and Experimental Neuropsychology, 22(4), 518–528. https://doi.org/10.1076/1380-3395(200008)22

Laumann, T. O., Gordon, E. M., Adeyemo, B., Snyder, A. Z., Joo, S. J. un, Chen, M. Y., Gilmore, A. W., McDermott, K. B., Nelson, S. M., Dosenbach, N. U. F., Schlaggar, B. L., Mumford, J. A., Poldrack, R. A., & Petersen, S. E. (2015). Functional System and Areal Organization of a Highly Sampled Individual Human Brain. Neuron, 87(3), 657–670. https://doi.org/10.1016/j.neuron.2015.06.037

Laumann, T. O., & Snyder, A. Z. (2021). Brain activity is not only for thinking. Curent Opinion in Behavioral Sciences, 40, 130–136. https://doi.org/10.1016/j.cobeha.2021.04.002

Laumann, T. O., Snyder, A. Z., Mitra, A., Gordon, E. M., Gratton, C., Adeyemo, B., Gilmore, A. W., Nelson, S. M., Berg, J. J., Greene, D. J., McCarthy, J. E., Tagliazucchi, E., Laufs, H., Schlaggar, B. L., Dosenbach, N. U. F., & Petersen, S. E. (2016). On the Stability of BOLD fMRI Correlations. Cerebral Cortex, 1–14. https://doi.org/10.1093/cercor/bhw265

Reviewer comments on registered report entitled “Functional MRI brain state occupancy in the presence of cerebral small vessel disease-pre-registration for a multiverse replication analysis of the Hamburg City Health Study”. 

N.B. I do not have sufficient practical knowledge of the imaging methodology described in this report to be able to assess its suitability for the intended application. However, the proposed study is a replication of a previously published article (Schlemm, 2022) and will use the same imaging pipeline, so I assume the proposed method has already passed peer review at Biological Psychiatry. Still, the handler at PCIRR might wish to recruit another reviewer for this article who can better cover the imaging aspects.  

I enjoyed reading this well-written and well-thought out registered report. The author proposes to replicate their own previous findings (Schlemm, 2022) in a sample from the Hamburg City Health Study:  

• Greater white matter hyperintensity (WMH) volume is associated with fMRI-derived brain states of high fractional occupancy (i.e. the proportion of BOLD volumes assigned to each brain state) 
• Less time spent in high-occupancy states is associated with poorer scores on the trail making task part B (a test of executive function). 

The analysis plan appears sound. The methods seem appropriate to be able to test the stated hypotheses and seem sufficiently detailed that the study might be replicated (with a reminder of the above caveat that I am not able to assess the imaging parameters).  

Main comments/questions:

1. The original study (Schlemm, 2022) tested associations between brain dynamics and multiple domains of cognitive ability. Would it not be beneficial to test the same associations in the present replication study, as were carried out in the original article, rather than to attempt replication of a single significant result? The increased statistical power in the present study might reveal associations that were not detected previously (which would be very interesting in itself), or might return null results, which would be in line with previous the previous article.  
2. The results of this work might be more compelling if the replication were to be carried out in a different dataset entirely. Did the author consider carrying out these replication analyses on any alternative samples (i.e. not the HCHS)?  
3. The authors note that missing data patterns will be reported. How will missing data be handled? Apologies if I’ve missed this. 
4. Are there any additional exclusion criteria beyond not having imaging data, failure of pre-processing, or inclusion in Schlemm et al., 2022? I’m wondering whether participants with non-SVD-related brain/cognitive disorders will be excluded? 
5. The author states that participants will be excluded from analysis if automated processing using Freesurfer or fmriPrep fails. Failure to pre-process through these automated pipelines might be more likely for those with more severe SVD (due to the presence of additional visual SVD markers such as lacunes, microbleeds etc) and may risk biasing the sample (towards milder cases of SVD). Do the imaging/data team perform any manual checks on any of the imaging data at any point, or on those that fail the automated pipelines? If so, it would be useful to detail this in the report – apologies if I’ve missed this.  
6. I note that this registered report only has one author. From the previously published article (Schlemm, 2022), I can see that other researchers have likely been involved in the preparation of this work and should probably also be credited as authors on this report.  
    
   Minor comments: 
7. The author mentions that they will exclude the sample who were included in the previous (Schlemm, 2022) analyses. As the sample size of the previous study (Schlemm, 2022) is known, it would be helpful to include it in the current report. 
8. It would be helpful if fractional occupancy was defined in the introduction, to save the reader having to hunt for it further down in the article. 
9. The author states that ‘gender’ is included as a covariate. Do you perhaps mean sex, rather than gender? See this guidance from the ONS

Additional question: 
Is the author planning to test potential mediation of the association between WMH and cognitive ability, via brain dynamics? I’m not suggesting that they include this as part of this report – I’m just interested.