Close printable page
Recommendation

Evidence for mixed quality of systematic reviews in preclinical animal studies of neurodevelopmental conditions

ORCID_LOGO based on reviews by Marietta Papadatou-Pastou
A recommendation of:

Genetically-modified animals as models of neurodevelopmental conditions: a review of systematic review reporting quality

Abstract

EN
AR
ES
FR
HI
JA
PT
RU
ZH-CN
Submission: posted 22 November 2023
Recommendation: posted 28 February 2024, validated 28 February 2024
Cite this recommendation as:
Chambers, C. (2024) Evidence for mixed quality of systematic reviews in preclinical animal studies of neurodevelopmental conditions. Peer Community in Registered Reports, 100600. 10.24072/pci.rr.100600

This is a stage 2 based on:

Recommendation

Single gene alterations have been estimated to account for nearly half of neurodevelopmental conditions (NDCs), providing a crucial opportunity for animal models to understand the underlying mechanisms, causes and potential treatments. The use of systematic reviews (SRs) can, in principle, provide a powerful means to synthesise this evidence-base; however, the reporting quality of previous SRs in preclinical animal research has been found lacking (Hunniford et al., 2021). In the current study, Wilson et al. (2023) will undertook a review of systematic reviews to assess the characteristics and reporting quality of SRs that, in turn, synthesise research in genetically-modified animals to model NDCs. In particular, the authors extracted key features of reviews (including, among others, the aim and primary research questions, relevant animal model, and number of studies in the SR), in addition to quality indicators such as risk of bias and completeness of reporting. In doing so, the authors aimed to enhance guidance on the conduct and reporting of systematic reviews in this area.
 
Of twelve publications that met the preregistered search criteria, the completeness and quality of reporting was variable. Among the better reported characteristics were search strategies (9 of 12 articles), reporting of funding sources (10 of 12 articles) and use of animal data (11 of 12 articles). In contrast, only two articles reported whether the study protocol was preregistered, only three articles reported methods for assessing risk of bias, and just one included methods to analyse publication bias. In addition, the authors identified 19 review registrations via PROSPERO, most of which remained unpublished after their anticipated end dates. Overall, the results highlight the importance of adherence to reporting guidelines for increasing the transparency and reproducibility of SRs in this field.
 
The Stage 2 manuscript was evaluated over one round of in-depth review. Based on detailed responses by the authors, the recommender judged that the manuscript met the Stage 2 criteria and awarded a positive recommendation.
 
URL to the preregistered Stage 1 protocol: https://osf.io/952qk
 
Level of bias control achieved: Level 4. At least some of the data/evidence that was used to answer the research question already existed prior to IPA and was accessible in principle to the authors, but the authors certify that they did not access any part of that data/evidence prior to IPA.
 
List of eligible PCI RR-friendly journals:
 
References
 
1. Hunniford V. T., Montroy J., Fergusson D. A., Avey M. T., Wever K. E., McCann S. K., Foster M., Fox G., Lafreniere M., Ghaly M., Mannell S., Godwinska K., Gentles A., Selim S., MacNeil J., Sikora L., Sena E. S., Page M. J., Macleod M., Moher D., & Lalu M. M. (2021). Epidemiology and reporting characteristics of preclinical systematic reviews. PLOS Biology, 19:e3001177. https://doi.org/10.1371/journal.pbio.3001177
 
2. Wilson, E., Currie, G., Macleod, M., Kind, P. & Sena, E. S. (2023). Genetically-modified animals as models of neurodevelopmental conditions: a review of systematic review reporting quality [Stage 2]. Acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/s5xd4
Conflict of interest:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.

Reviews

Evaluation round #1

DOI or URL of the report: https://doi.org/10.31222/osf.io/s5xd4

Version of the report: v2

Author's Reply, 26 Feb 2024

Decision by ORCID_LOGO, posted 04 Dec 2023, validated 04 Dec 2023

One of the reviewers from Stage 1 kindly returned to evaluate your Stage 2 manuscript; the other was unavailable, but I have decided that we can proceed with an interim Stage 2 decision based on this reviewer's comments and my own reading.

Overall I think the Stage 2 manuscript is promising, with the caveat that there needs to a lot more clarity over the scope and reasons for the numerous text deviations between the approved Stage 1 manuscript and corresponding components within the Stage 2 manuscript. As the reviewer notes (and I picked up myself), there are several points where deviations are made but not explained, and some where they are made but not flagged (e.g. the tracked-changes version of the Stage 2 manuscript didn't flag some text changes in the Introduction, such as the statement about conducting an umbrella review.

In revising, please revert all changes to the text of the Introduction unless they are necessary to correct an error of fact, to clarify an ambiguity, or to appropriately alter the tense. Beyond these conditions, no new literature, rationale or justification (or redescription of unchanged procedures/rationale) should be made. Elsewhere, other tracked changes are not fully explained, such as the alterations to the Data Synthesis section, leaving the reader (and the reviewer in this case) unclear as to how much is purely descriptive change vs. how much is actual changes to the methods. As a result, the description of deviations from protocol appears to be incomplete.

I want to be clear that there is no accusation of impropiety here; the requirements of RRs are somewhat different (and more strict) than with regular papers, so some confusion is understandable particularly for authors who are new to RRs. But please attend carefully to this general point in revising, along with the reviewer's many additional points.

Reviewed by ORCID_LOGO, 30 Nov 2023

I would first like to congratulate the authors for completing their review in a timely manner (esp. since their searches were based on full text; this is a good practice by the way).


When I started reading the Stage 2 manuscript, I noticed a lot of differences between Stage1 and Stage 2 (obviously I am not referring to the results and discussion section here, but the earlier sections), starting with the title, which was changed from "Genetically-modified animals as models of neurodevelopmental conditions: an umbrella review" to "Review evaluating the reporting quality of systematic reviews 1 describing animal studies of neurodevelopmental conditions: a Registered Report". This is not just a slight change of wording. For instance, the new title does not describe "genetically-modified" animals nor an umbrella review. 


Another example of a difference is that the aim is now worded differently in the introduction. The Stage 1 aim was: "Here, we aim to conduct an umbrella review to identify...." and the Stage 2 is: "Here, we aim to identify....". I still wonder why this work is not termed an umbrella review in Stage 2 (a review technique with specific characteristics), but more importantly why it was not an umbrella review as accepted in Stage 1. Also, the "full text retrieval" section does not appear in Stage 1. There are also a lot of minor changes, new sentences, and up-to-date references in the intro and methods section. The most important change is that from "Data extraction" onwards, all titles have been modified (the title "Data extraction" has been removed altogether - I suggest it is reinstated) and there are subtle changes in the text too.


The data synthesis paragraph has been changed from

"We will not conduct a meta-analysis. However, we will present a descriptive, tabular summary of the bibliographic, characteristics, and reporting quality data extracted from each included systematic review. We will score each included systematic review using the PRISMA-Pre checklist and provide a summary table detailing which items of the checklist each review met. Additionally, the checklist will allow us to assess which tools are currently being used to conduct systematic reviews in this field, including the tools used to screen studies (Checklist Item 6a), extract numerical data (Checklist Item 17a), and measure study quality or risk of bias (Checklist Item 19)." 

to 


"The purpose of this work is not to conduct a meta-analysis based on the findings of included systematic reviews, but rather to identify the quality and reporting quality of these reviews. All data annotated are presented as descriptive summaries, and no formal statistical analysis was undertaken." 


Here, there is an important change, namely that the "tools used to screen studies, extract numerical data, and measure study quality or risk of bias" were not assessed, as promised in Stage 1. 


I guess whether such changes are allowed is something to be discussed with the recommender. However, it is my understanding from my experience in publishing Stage 2 RRs that only the verb tense is allowed to change from future tense (Stage 1) to past tense (Stage 2). These changes created more workload for me, as I had to compare the two submissions instead of just evaluating the results and discussion section (maybe the authors could have annotated their manuscript so that these changes were easily spotted?). But most importantly, this practice goes against the rationale of Registered Reports, whereby in principle acceptance is given to a study at Stage 1 and at Stage 2 what was agreed earlier is what should be done and reported. Of course, understandably things don't always go as planned, therefore it is very important for transparency purposes that Stage 2 includes a section entitled: "Deviations from Preregistration" (as is also suggested by PRISMA). 


Such a section exists in the manuscript, but the title is "Protocol registration". Actually, "Protocol registration" and "Deviations from Preregistration" should be two different sections. Having said that, this section only reports two changes (the inclusion of studies on the UBE3A gene and the search strategy for PROSPERO) and none of the changes I listed above. Therefore, this section should be more detailed.


Below are some more specific comments:


-Results - systematic search results: "The same systematic review was identified as a conference abstract (Zhang et al., 2022) and a peer-reviewed journal article (Zhang et al., 2021). As our evaluation is primarily concerned with reporting quality these publications, we assessed the conference abstract and journal article as separate publications." (Of note, there is a "the" missing in the second sentence). I am not sure I follow this rationale - researchers and policymakers, who use systematic reviews to make decisions, would read the published paper and not the conference abstract. 


-In the "Characteristics of included reviews" I would also like to hear which neurodevelopmental conditions were targeted by the reviews. 


-Figures 2 and 3 are not needed, as they duplicate information already presented in the text of the manuscript (only a few words in parenthesis are missing, which can be added). Actually, this also concerns Figures 6 and 7.


-line 348: "neurodevelopmental conditions or autism." Autism is also a neurodevelopmental condition (best use the term "autism spectrum disorder" throughout)


-The flow diagram is fine, but I would suggest that the authors use the PRISMA template (here: http://prisma-statement.org/prismastatement/flowdiagram.aspx?AspxAutoDetectCookieSupport=1)

 

Minor comments/typos


Since the authors already made some changes in Stage 1, I will take the liberty to make a couple of comments on that part too:


1. First sentence of the abstract: "Using genetically-modified animals to model neurodevelopmental conditions (NDCs) helps better our understanding of their underlying biology". It is not clear what "their" is referring to, maybe some rewriting will make this clearer.
2. Second sentence of the abstract: "In vivo research has unique characteristics not shared with clinical research, meaning approaches to systematic review must be adapted to this context." To "systematically review" what? Again, maybe some rewriting will make this clearer.
3. Abstract, methods, line 3: "is" - should it be "was"?
4. Abstract, methods, line 4: the first comma needs to be removed I think. 
5. There are a lot of one-sentence paragraphs. The APA publication manual asks that paragraphs contain more than one sentence. However, maybe this is something that is different from discipline to discipline. 
6. Line 354: "all which are ongoing." An "of" is missing here. 
7. There are other minor grammatical issues in the manuscript too, but I guess they will be spotted during type-editing. 

Overall, if the recommender thinks that the changes made to the first part (corresponding to Stage 1) can be accepted, then I would suggest publication following the minor revisions I list above.