Recommendation
based on reviews by Marietta Papadatou-Pastou
DOI or URL of the report: https://doi.org/10.31222/osf.io/s5xd4
Version of the report: v2
, posted 04 Dec 2023, validated 04 Dec 2023One of the reviewers from Stage 1 kindly returned to evaluate your Stage 2 manuscript; the other was unavailable, but I have decided that we can proceed with an interim Stage 2 decision based on this reviewer's comments and my own reading.
Overall I think the Stage 2 manuscript is promising, with the caveat that there needs to a lot more clarity over the scope and reasons for the numerous text deviations between the approved Stage 1 manuscript and corresponding components within the Stage 2 manuscript. As the reviewer notes (and I picked up myself), there are several points where deviations are made but not explained, and some where they are made but not flagged (e.g. the tracked-changes version of the Stage 2 manuscript didn't flag some text changes in the Introduction, such as the statement about conducting an umbrella review.
In revising, please revert all changes to the text of the Introduction unless they are necessary to correct an error of fact, to clarify an ambiguity, or to appropriately alter the tense. Beyond these conditions, no new literature, rationale or justification (or redescription of unchanged procedures/rationale) should be made. Elsewhere, other tracked changes are not fully explained, such as the alterations to the Data Synthesis section, leaving the reader (and the reviewer in this case) unclear as to how much is purely descriptive change vs. how much is actual changes to the methods. As a result, the description of deviations from protocol appears to be incomplete.
I want to be clear that there is no accusation of impropiety here; the requirements of RRs are somewhat different (and more strict) than with regular papers, so some confusion is understandable particularly for authors who are new to RRs. But please attend carefully to this general point in revising, along with the reviewer's many additional points.
, 30 Nov 2023I would first like to congratulate the authors for completing their review in a timely manner (esp. since their searches were based on full text; this is a good practice by the way).
When I started reading the Stage 2 manuscript, I noticed a lot of differences between Stage1 and Stage 2 (obviously I am not referring to the results and discussion section here, but the earlier sections), starting with the title, which was changed from "Genetically-modified animals as models of neurodevelopmental conditions: an umbrella review" to "Review evaluating the reporting quality of systematic reviews 1 describing animal studies of neurodevelopmental conditions: a Registered Report". This is not just a slight change of wording. For instance, the new title does not describe "genetically-modified" animals nor an umbrella review.
Another example of a difference is that the aim is now worded differently in the introduction. The Stage 1 aim was: "Here, we aim to conduct an umbrella review to identify...." and the Stage 2 is: "Here, we aim to identify....". I still wonder why this work is not termed an umbrella review in Stage 2 (a review technique with specific characteristics), but more importantly why it was not an umbrella review as accepted in Stage 1. Also, the "full text retrieval" section does not appear in Stage 1. There are also a lot of minor changes, new sentences, and up-to-date references in the intro and methods section. The most important change is that from "Data extraction" onwards, all titles have been modified (the title "Data extraction" has been removed altogether - I suggest it is reinstated) and there are subtle changes in the text too.
The data synthesis paragraph has been changed from
"We will not conduct a meta-analysis. However, we will present a descriptive, tabular summary of the bibliographic, characteristics, and reporting quality data extracted from each included systematic review. We will score each included systematic review using the PRISMA-Pre checklist and provide a summary table detailing which items of the checklist each review met. Additionally, the checklist will allow us to assess which tools are currently being used to conduct systematic reviews in this field, including the tools used to screen studies (Checklist Item 6a), extract numerical data (Checklist Item 17a), and measure study quality or risk of bias (Checklist Item 19)."
to
"The purpose of this work is not to conduct a meta-analysis based on the findings of included systematic reviews, but rather to identify the quality and reporting quality of these reviews. All data annotated are presented as descriptive summaries, and no formal statistical analysis was undertaken."
Here, there is an important change, namely that the "tools used to screen studies, extract numerical data, and measure study quality or risk of bias" were not assessed, as promised in Stage 1.
I guess whether such changes are allowed is something to be discussed with the recommender. However, it is my understanding from my experience in publishing Stage 2 RRs that only the verb tense is allowed to change from future tense (Stage 1) to past tense (Stage 2). These changes created more workload for me, as I had to compare the two submissions instead of just evaluating the results and discussion section (maybe the authors could have annotated their manuscript so that these changes were easily spotted?). But most importantly, this practice goes against the rationale of Registered Reports, whereby in principle acceptance is given to a study at Stage 1 and at Stage 2 what was agreed earlier is what should be done and reported. Of course, understandably things don't always go as planned, therefore it is very important for transparency purposes that Stage 2 includes a section entitled: "Deviations from Preregistration" (as is also suggested by PRISMA).
Such a section exists in the manuscript, but the title is "Protocol registration". Actually, "Protocol registration" and "Deviations from Preregistration" should be two different sections. Having said that, this section only reports two changes (the inclusion of studies on the UBE3A gene and the search strategy for PROSPERO) and none of the changes I listed above. Therefore, this section should be more detailed.
Below are some more specific comments:
-Results - systematic search results: "The same systematic review was identified as a conference abstract (Zhang et al., 2022) and a peer-reviewed journal article (Zhang et al., 2021). As our evaluation is primarily concerned with reporting quality these publications, we assessed the conference abstract and journal article as separate publications." (Of note, there is a "the" missing in the second sentence). I am not sure I follow this rationale - researchers and policymakers, who use systematic reviews to make decisions, would read the published paper and not the conference abstract.
-In the "Characteristics of included reviews" I would also like to hear which neurodevelopmental conditions were targeted by the reviews.
-Figures 2 and 3 are not needed, as they duplicate information already presented in the text of the manuscript (only a few words in parenthesis are missing, which can be added). Actually, this also concerns Figures 6 and 7.
-line 348: "neurodevelopmental conditions or autism." Autism is also a neurodevelopmental condition (best use the term "autism spectrum disorder" throughout)
-The flow diagram is fine, but I would suggest that the authors use the PRISMA template (here: http://prisma-statement.org/prismastatement/flowdiagram.aspx?AspxAutoDetectCookieSupport=1)
Minor comments/typos
Since the authors already made some changes in Stage 1, I will take the liberty to make a couple of comments on that part too:
1. First sentence of the abstract: "Using genetically-modified animals to model neurodevelopmental conditions (NDCs) helps better our understanding of their underlying biology". It is not clear what "their" is referring to, maybe some rewriting will make this clearer.
2. Second sentence of the abstract: "In vivo research has unique characteristics not shared with clinical research, meaning approaches to systematic review must be adapted to this context." To "systematically review" what? Again, maybe some rewriting will make this clearer.
3. Abstract, methods, line 3: "is" - should it be "was"?
4. Abstract, methods, line 4: the first comma needs to be removed I think.
5. There are a lot of one-sentence paragraphs. The APA publication manual asks that paragraphs contain more than one sentence. However, maybe this is something that is different from discipline to discipline.
6. Line 354: "all which are ongoing." An "of" is missing here.
7. There are other minor grammatical issues in the manuscript too, but I guess they will be spotted during type-editing.
Overall, if the recommender thinks that the changes made to the first part (corresponding to Stage 1) can be accepted, then I would suggest publication following the minor revisions I list above.