Recommendation
based on reviews by 1 anonymous reviewer
DOI or URL of the report: https://github.com/csi-hamburg/HCHS-brain-states-RR/blob/d5a42f66c833cc023323ea9ee32448bb1ea71720/manuscript/build/main.pdf
Version of the report: v2.0.1
, posted 09 Jan 2024, validated 09 Jan 2024Thank you for submitting your Stage 2 Registered Report. The manuscript has been assessed by one of the original (Stage 1) reviewers, who has provided some insightful comments, to which you should respond with appropriate revisions (or rebuttals).
Whilst you should do your best to fully consider reviewer comments, you should not change your Stage 1 material without further consultation, and you are not obliged to follow reviewer suggestions for additional exploratory analyses at this stage. That said, I think it would be acceptable to add information about achieved sample size into Figure 1.
I have been unable to obtain comments from a second reviewer and, given the time already elapsed, have decided not to wait any longer. I append some comments of my own, though these are more focused on RR requirements than on the specific scientific topic.
First, you have removed from the Stage 1 manuscript your previous pilot analysis, and also moved the timeline section (describing Stage 1 state of knowledge of the data) to the end of the manuscript. The timeline section should be reinstated within the Methods for correspondence with the approved Stage 1 plan. If you wish to remove the pilot analysis from the Stage 2 manuscript, you should explain your reasoning in your response, so that it can be evaluated, and you should at least add a footnote to the Stage 2 Methods to inform the reader that a pilot analysis included at Stage 1 has been omitted for brevity but can be found in the archived Stage 1 manuscript, providing a link to that document.
In passing, I note two very minor typographical/stylistic points: (1) please regularise 'subjects' to 'participants'; (2) there seems to be a word or two missing from the following: "network activation profiles were computed for brain states estimated Schaefer parcellations..."
The authors present a phase 2 pre-registered replication study to examine associations between dynamic resting-state fMRI, small vessel disease (WMH), and cognition. The research question is scientifically valid, but the theoretical rationale requires some additional clarification and justification. The sample and methods are mostly appropriate, but I offer some suggestions for improved rigor. Results are presented clearly, but I offer some suggestions for additional transparency. My strongest critique is that the authors’ characterization of “robustness” in the behavioral association does not appear to be supported by the data.
1. Figure 1 should be updated to report the achieved sample rather than expected.
2. What is the justification for focusing on average fractional occupancy in either DMN+ or DMN- clusters? How is occupancy in these two clusters related across individuals? Are similar associations with WMH or cognition observed for DMN+ or DMN- occupancy individually?
3. “49/81 (39/81) negative and 8/81 (0/81) associations of nominal statistical significance” I assume this sentence is missing the word “positive” after (0/81)?
4. The authors acknowledge that the TMT-B results are “somewhat less robust” than the WMH results, but this wording seems too generous given that the effect nominally replicates in less than 20% of the analyses. At the very least, they should remove the word “somewhat” as this effect is clearly less robust than the WMH effect. It is also misleading to state that both effects are robust in line 301 and line 344 without qualification. Overall robustness should be assessed in a meta-analysis-like approach by calculating the average effect size and CI across the multiverse analyses. Is the average effect significantly different from 1? How does it compare to the observed effect size in the 2022 paper?
5. Forest plots of the multiverse analyses for periventricular and deep WMH volumes should be provided as supplementary material.
6. Figure 6. Do these spider plots align with network patterns identified in the 2022 paper?
7. The spider plots do not characterize regional patterns of high vs. low activation in the clusters. Please provide brain images as well.
8. It does not appear that the tests of additional cognitive relationships were corrected for multiple comparisons.
9. “all reported associations were robust to additional, unplanned adjustments for DVARS, RMSD or mean 361 framewise displacement.” - please provide the details of these analyses as supplementary material.