Testing the replicability of dynamic functional connectivity correlates of cerebral small vessel disease in the Hamburg City Health Study
Functional MRI brain state occupancy in the presence of cerebral small vessel disease -- pre-registration for a replication analysis of the Hamburg City Health Study
Abstract
Recommendation: posted 26 June 2023, validated 27 June 2023
McIntosh, R. (2023) Testing the replicability of dynamic functional connectivity correlates of cerebral small vessel disease in the Hamburg City Health Study. Peer Community in Registered Reports, . https://rr.peercommunityin.org/PCIRegisteredReports/articles/rec?id=341
Related stage 2 preprints:
Thies Ingwersen, Carola Mayer, Marvin Petersen, Benedikt M. Frey, Jens Fiehler, Uta Hanning, Simone Kühn, Jürgen Gallinat, Raphael Twerenbold, Christian Gerloff, Bastian Cheng, Götz Thomalla, Eckhard Schlemm,
https://github.com/csi-hamburg/HCHS-brain-states-RR/blob/f9d00adbbcf9593d8d191bf5b93912141b80ab1b/manuscript/build/main.pdf
Recommendation
Level of bias control achieved: Level 2. At least some data/evidence that will be used to answer the research question has been accessed and partially observed by the authors, but the authors certify that they have not yet observed the key variables within the data that will be used to answer the research question AND they have taken additional steps to maximise bias control and rigour.
List of eligible PCI RR-friendly journals:
- Brain and Neuroscience Advances
- Imaging Neuroscience
- In&Vertebrates
- NeuroImage: Reports
- Peer Community Journal
- PeerJ
- Royal Society Open Science
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.
Evaluation round #2
DOI or URL of the report: https://raw.githubusercontent.com/csi-hamburg/HCHS-brain-states-RR/main/manuscript/build/main.pdf
Version of the report: v1.3
Author's Reply, 26 Jun 2023
Dear Rob,
thanks for your message and inviting further minor revisions to our manuscript. We agree with all suggestions and have updated the report accordingly.
Please find attached detailed reponses to your and the reviewer's comments, as well as the current tracked-changes document.
Thanks again and kind regards,
Eckhard
Decision by Robert McIntosh, posted 20 Jun 2023, validated 20 Jun 2023
Thank you for the revisions made to this Stage 1 Registered Report. As you will see, the reviewers are now generally happy with the plan, although Reviewer#1 (Olivia Hamilton) has a couple more suggestions for aspects of the protocol that should be made explicit. It is probably best to include these details in the Stage 1 plan, rather than adding them at Stage 2, and you have the opportunity to do so, because there is one other small matter that needs to be attended to before IPA can be issued.
Specifically, you have two main hypotheses, as stated in your Intoroduction, the second of which concerns the relation of fractional occupancy to the time to complete TMT-B. A summary specification of this hypothesis should be included in the study design table (Table 1). You should also make it clear in the text (paragraph beginning line 63, page 3) whether this finding was reported by the previous analysis of Schlemm et al (2021). It would also be helpful if this paragraph could be expanded beyond simply stating that this previous study 'found associations', to providing some quantitative clarification of the associations that were found.
Once these comments have been addressed, I anticipate being able to issue IPA.
Reviewed by Olivia Hamilton, 13 Jun 2023
I read the authors responses with interest and just have a couple of additional thoughts, which do not necessarily require a response.
In response to the second point about how missing data will be handled, the authors state that they will be carrying out a complete-case analysis. In the final report from this work, it would be good to see a justification of this choice. I would suggest that the authors eventually report how many participants were excluded from the sample due to incomplete data and consider any biases that this might introduce (if any).
In response to the third point regarding additional exclusion criteria, the authors state that they will not exclude participants with dementia. It would be beneficial to explicitly state this in the final report, and state the proportion of the sample with a dementia diagnosis (if available, if not, this is a limitation), in order to better characterise the sample and contextualise the findings.
Best of luck to the authors with the submission of their work!
Reviewed by anonymous reviewer 1, 17 Jun 2023
The authors have addressed my previous comments. The pre-registration has been improved as a result of these and the other modifications.
Evaluation round #1
DOI or URL of the report: https://raw.githubusercontent.com/csi-hamburg/HCHS-brain-states-RR/main/manuscript/build/main.pdf
Version of the report: v1.1
Author's Reply, 31 May 2023
Dear Rob,
thanks again for handling my manuscript and recruiting two very constructive and helpful reviewers. We have now revised the manuscript based on your and the reviewers' comments and suggestions and feel that it has much improved.
Detailed point-by-point responses are contained in the project's GitHub repository at:
https://github.com/csi-hamburg/HCHS-brain-states-RR/blob/main/manuscript/R1_response.md
I hope that the manuscript will now be suitable for acceptance-in-principle and am looking forward to carrying out the proposed analysis.
Many thanks again and best regards,
Eckhard
Decision by Robert McIntosh, posted 04 Apr 2023, validated 04 Apr 2023
Dear Dr. Schlemm,
Thank you for patience, and I apologise that it has taken so long to return reviews for your manuscript. It proved rather tricky to find reviewers, but once suitable and willing people were found, the reviews were both prompt and thorough, and I think you will find them very helpful in guiding a revision of this Stage 1 plan.
Both reviewers are generally positive about the proposed study, but have substantive suggestions for improvement. Amongst other points, Reviewer #1 emphasises the need for development of the theoretical rationale for the study (and the decision to focus on dynamic FC). Reviewer #2 is Dr. Olivia Hamilton, who has a number of queries and suggestions for improved clarity (including the possibility of seeking additional suitable datasets in which to test the robustness of your findings).
As I am not an expert in the topical area, my own reading was focused primarily on the aspects of the plan related specifically to the RR format. The main points that I would draw your attention to are listed below, the most fundamental of which is probably (4).
1) Testing against a null hypothesis of no association is arguably relatively uninformative, given a sample size of ~1500, because even a very weak relationship could be statistically significant at this sample size. It would good to include a more nuanced a priori discussion of what size of effect would be of theoretical and/or practical importance in the present context.
2) The power analysis is predicated directly upon the central estimate of effect size from Schlemm et al (2022). It seems likely that this effect size will overestimate the true population effect, because it has been selected non-randomly as the target of this replication (i.e. you chose to propose a replication for this effect, and not for effects that did not come out as significant in the previous analysis). It would be more realistic to use a lower-bound estimate of the effect size or, to motivate the smallest effect size of interest in the present context a priori (as noted above).
3) It is stated that the multiverse analysis will be run if the hypothesised association can be replicated using the primary analytical choices. If the point is to test the robustness of the outcome to different analytical choices, it is not clear to me why this is only relevant if the main hypothesis test is significant.
4) This leads into a deeper conceptual issue for your Stage 1 plan. You wish to preregister a multiverse analysis, but you state that “For each combination… we will quantify the association between WMH load and average time spent in high-occupancy brain states... no hypothesis testing and therefore no adjustment for multiple comparisons will be carried out.” (The statement that no adjustment for multiple comparisons will be made may not be necessary here, because – as you note – it is entailed by the fact that no hypothesis testing is being done.)
The problem here is that you do not specify how the results of the multiverse analysis will determine your conclusions. The point of preregistration (at least in a formal registered Report) is not just to say what analyses you will do, but to specify unambiguously all of the logical linkages between possible outcomes and theoretical conclusions. The title of your manuscript suggests that the multiverse analysis is the focus of your study; but, within the methods, it is an additional exploratory step which is contingent on the outcome of the main hypothesis test and will serve a descriptive and not an inferential purpose.
So, if your main purpose is your inferential replication, then the multiverse analysis should probably be dropped from the Stage 1 plan, unless it can be sufficiently thoroughly specified to elevate it to a proper part of the registered plan (which could be challenging, given the vast range of possible outcomes). It could still be added at Stage 2 as an additional exploratory analysis, but the format would require that your conclusions could not be substantially driven by non-registered parts of your analysis.
On the other hand, if your main purpose is the multiverse analysis (as your title suggests), then there may be relatively little benefit in the Registered Reports format for your study, because your approach is to cover the range of possible analysis pipelines rather than nail a specific one down. In a sense, Registered Reports and multiverse analyses can be considered tio represent alternative strategies to achieve analytic transparency (notwithstanding that, in principle, it is possible to preregister a multiverse analysis).
5) As a minor point, the legend for Figure 2 is insufficient for me to understand the Figure (e.g. how should we know which effects in this plot are significant? I do not see 36p in the symbol key...), and should be improved. In passing, I note that (if I interpret correctly) the vast majority of analyses return results that are smaller than the effect size reported by Schlemm et al (2022).
Reviewed by anonymous reviewer 1, 07 Mar 2023
The author presents a pre-registration of a replication study to examine associations between dynamic resting-state fMRI, small vessel disease (WMH), and executive function (TMT-B). The research question is scientifically valid, but the theoretical rationale for the hypothesis (specifically regarding the focus on dynamic FC and high-occupancy states) requires further clarification and justification. The sample and proposed methods are mostly appropriate and feasible, but I offer some suggestions for improved clarity and rigor.
1. Line 47: The author suggests that limited reproducibility of functional connectivity “in the presence of cSVD” might contribute to “heterogeneous” findings in the prior literature. I believe it is important to acknowledge here that functional connectivity measurements are generally quite noisy with low reliability, even in healthy control samples, especially when measured with short (~5 minutes) resting-state scans (e.g., Laumann et al., 2015, Figure 4), which are quite common in clinical studies, including the present study. This is not a problem specific to cSVD or any clinical population.
2. The justification for choosing to focus on dynamic FC as opposed to traditional FC is not clear. Given the context of the paragraph beginning in line 40, it seems to imply that dynamic FC might be more “reproducible” or less “arbitrary” than traditional FC. However, this is counter-intuitive, as dynamic FC has not been demonstrated to lead to either of these improvements. If anything, dynamic FC is likely less reproducible than traditional FC in the sense that resting-state scans are broken into even shorter “state” bins. If the argument is that dynamic FC is more likely to be relevant for cognition, this claim has also been disputed in the literature, as dynamic shifts in FC have been suggested to arise from non-cognitive processes, which should be acknowledged as a potential interpretation of the data (Laumann et al., 2016; Laumann & Snyder, 2021).
3. What is the expected final sample size after excluding participants who were already included in the prior study?
4. The Trail Making Test - part B is selected as a measure of executive function. However, unless performance is adjusted for part A, performance on part B alone may be driven in large part by psychomotor processing speed (Arbuthnott & Frank, 2000).
5. The proposed clustering analyses will use a default k of 5, presumably based on the result from the author’s previous paper. However, I believe it is important to test whether this result replicates in the proposed analyses. A range of k values should be tested and compared to find the best fitting number of clusters, rather than selecting a value a priori.
6. In addition to the demographic variables, variables relating to resting-state fMRI data quality and motion (e.g., mean FD, % frames retained in scrubbing) should be reported and included as statistical covariates.
7. Table 2. It is not clear which pipelines will include GSR, e.g., GSR should be included in the Power and Satterthwaite pipelines if consistent with the references, but it is not listed in the table.
8. Hypotheses regarding the dynamic FC are mostly framed in terms of high vs. low occupancy. For instance, the separation of 2 high-occupancy and 3 low-occupancy “states” is shown to be robust in the multiverse analyses, but are the regional network configurations of these states also consistent? Is the optimal solution of k = 5 consistent? These questions seem to have more importance for theoretical interpretation, whereas the observation of greater occupancy in states defined as being “high-occupancy” is nearly circular. Why is meaningful that WMH would be associated with dwell time in a state defined by its occupancy rate, as opposed to a state defined by its pattern of network configuration?
References
Arbuthnott, K., & Frank, J. (2000). Trail Making Test , Part B as a Measure of Executive Control : Validation Using a Set-Switching Paradigm. Journal of Clinical and Experimental Neuropsychology, 22(4), 518–528. https://doi.org/10.1076/1380-3395(200008)22
Laumann, T. O., Gordon, E. M., Adeyemo, B., Snyder, A. Z., Joo, S. J. un, Chen, M. Y., Gilmore, A. W., McDermott, K. B., Nelson, S. M., Dosenbach, N. U. F., Schlaggar, B. L., Mumford, J. A., Poldrack, R. A., & Petersen, S. E. (2015). Functional System and Areal Organization of a Highly Sampled Individual Human Brain. Neuron, 87(3), 657–670. https://doi.org/10.1016/j.neuron.2015.06.037
Laumann, T. O., & Snyder, A. Z. (2021). Brain activity is not only for thinking. Curent Opinion in Behavioral Sciences, 40, 130–136. https://doi.org/10.1016/j.cobeha.2021.04.002
Laumann, T. O., Snyder, A. Z., Mitra, A., Gordon, E. M., Gratton, C., Adeyemo, B., Gilmore, A. W., Nelson, S. M., Berg, J. J., Greene, D. J., McCarthy, J. E., Tagliazucchi, E., Laufs, H., Schlaggar, B. L., Dosenbach, N. U. F., & Petersen, S. E. (2016). On the Stability of BOLD fMRI Correlations. Cerebral Cortex, 1–14. https://doi.org/10.1093/cercor/bhw265
Reviewed by Olivia Hamilton, 03 Apr 2023
Reviewer comments on registered report entitled “Functional MRI brain state occupancy in the presence of cerebral small vessel disease-pre-registration for a multiverse replication analysis of the Hamburg City Health Study”.
N.B. I do not have sufficient practical knowledge of the imaging methodology described in this report to be able to assess its suitability for the intended application. However, the proposed study is a replication of a previously published article (Schlemm, 2022) and will use the same imaging pipeline, so I assume the proposed method has already passed peer review at Biological Psychiatry. Still, the handler at PCIRR might wish to recruit another reviewer for this article who can better cover the imaging aspects.
I enjoyed reading this well-written and well-thought out registered report. The author proposes to replicate their own previous findings (Schlemm, 2022) in a sample from the Hamburg City Health Study:
- Greater white matter hyperintensity (WMH) volume is associated with fMRI-derived brain states of high fractional occupancy (i.e. the proportion of BOLD volumes assigned to each brain state)
- Less time spent in high-occupancy states is associated with poorer scores on the trail making task part B (a test of executive function).
The analysis plan appears sound. The methods seem appropriate to be able to test the stated hypotheses and seem sufficiently detailed that the study might be replicated (with a reminder of the above caveat that I am not able to assess the imaging parameters).
Main comments/questions:
- The original study (Schlemm, 2022) tested associations between brain dynamics and multiple domains of cognitive ability. Would it not be beneficial to test the same associations in the present replication study, as were carried out in the original article, rather than to attempt replication of a single significant result? The increased statistical power in the present study might reveal associations that were not detected previously (which would be very interesting in itself), or might return null results, which would be in line with previous the previous article.
- The results of this work might be more compelling if the replication were to be carried out in a different dataset entirely. Did the author consider carrying out these replication analyses on any alternative samples (i.e. not the HCHS)?
- The authors note that missing data patterns will be reported. How will missing data be handled? Apologies if I’ve missed this.
- Are there any additional exclusion criteria beyond not having imaging data, failure of pre-processing, or inclusion in Schlemm et al., 2022? I’m wondering whether participants with non-SVD-related brain/cognitive disorders will be excluded?
- The author states that participants will be excluded from analysis if automated processing using Freesurfer or fmriPrep fails. Failure to pre-process through these automated pipelines might be more likely for those with more severe SVD (due to the presence of additional visual SVD markers such as lacunes, microbleeds etc) and may risk biasing the sample (towards milder cases of SVD). Do the imaging/data team perform any manual checks on any of the imaging data at any point, or on those that fail the automated pipelines? If so, it would be useful to detail this in the report – apologies if I’ve missed this.
- I note that this registered report only has one author. From the previously published article (Schlemm, 2022), I can see that other researchers have likely been involved in the preparation of this work and should probably also be credited as authors on this report.
Minor comments: - The author mentions that they will exclude the sample who were included in the previous (Schlemm, 2022) analyses. As the sample size of the previous study (Schlemm, 2022) is known, it would be helpful to include it in the current report.
- It would be helpful if fractional occupancy was defined in the introduction, to save the reader having to hunt for it further down in the article.
- The author states that ‘gender’ is included as a covariate. Do you perhaps mean sex, rather than gender? See this guidance from the ONS
Additional question:
Is the author planning to test potential mediation of the association between WMH and cognitive ability, via brain dynamics? I’m not suggesting that they include this as part of this report – I’m just interested.