The Peer Community In (PCI) initiative is a non-profit, non-commercial platform that publishes the peer-reviews of preprints. The overarching aim of this researcher-run organisation is to create specific communities of researchers reviewing and recommending, for free, unpublished preprints in their field. PCI communities have so far been created in evolutionary biology, ecology, paleontology, animal science, zoology, circuit neuroscience, metascience, and genomics, with more in the pipeline. Once the submissions are accepted (or, in PCI terms, “recommended”) following peer review, the revised manuscript is posted at the preprint server where the preprint is hosted, and the peer reviews and recommendation of the preprint are posted at the PCI website. Authors then have the option to also publish the preprint in a traditional journal.

 

PCI Registered Reports (PCI RR) is a new community dedicated to receiving, reviewing, and recommending Registered Reports (RRs) across the full spectrum of STEM, medicine, the social sciences and humanities. RRs are a form of empirical article in which study proposals are peer reviewed and pre-accepted before research is undertaken. By deciding which articles are published based on the question and proposed methods, RRs offer a remedy for a range of research biases, including publication bias and reporting bias.

 

Peer review for a RR takes place over two stages.  At Stage 1, authors submit their research question(s), theory and hypotheses (where applicable), detailed methods and analysis plans, and any preliminary data as needed. Following detailed review and revision – usually according to specific criteria – proposals that are favourably assessed receive in principle acceptance (IPA), which commits PCI RR to recommending the final article regardless of the outcomes, provided the authors adhere to their approved protocol and interpret the results in line with the evidence. Following IPA, the approved protocol is then registered in a recognised repository, either publicly or under a temporary embargo. Then, after completing the research, they submit a Stage 2 manuscript that includes the approved protocol plus results and discussion, which may include clearly labelled post hoc analyses in addition to the preregistered outcomes. The reviewers from Stage 1 then return to assess the completed Stage 2 manuscript, focusing on compliance with protocol and whether the conclusions are justified by the evidence. Crucially, reviewers do not relitigate the theory, hypotheses or methods, thus preventing knowledge of the results from influencing recommendations. PCI editors (termed “recommenders”) are similarly forbidden from rejecting a manuscript on the basis of any new concerns about the methodology or rationale, or on the basis of the main outcomes.

 

Following the completion of peer review, authors of RRs that are recommended (i.e. accepted) have the option to publish their articles in the growing list of PCI RR-friendly journals that have committed to accepting PCI RR recommendations without further review. PCI RR thus aims to become the central hub and clearing house for Stage 1 and Stage 2 RRs across all areas of research.

 

2. Submission requirements, review policy and workflow

2.1 Stage 1 and Stage 2 criteria

Reviewers for PCI RR assess Stage 1 and Stage 2 RRs across all areas of quantitative and qualitative research according to the following criteria:

1A. The scientific validity of the research question(s). This criterion addresses the extent to which the research question is scientifically justifiable. Valid scientific questions usually stem from theory or mathematics, an intended or possible application, or an existing evidence base, and should be defined with sufficient precision as to be answerable through quantitative or qualitative research. They should also fall within established ethical norms. Note that subjective judgments about the importance, novelty or timeliness of a research question are NOT a relevant criterion at PCI RR and do not form part of this assessment.

1B. The logic, rationale, and plausibility of the proposed hypotheses, as applicable. This criterion addresses the coherence and credibility of any a priori hypotheses. The inclusion of hypotheses is not required– a Stage 1 RR can instead propose estimation or measurement of phenomena without expecting a specific observation or relationship between variables. However, where hypotheses are stated, they should be stated as precisely as possible and follow directly from the research question or theory. A Stage 1 RR should also propose a hypothesis that is sufficiently conceivable as to be worthy of investigation. The complete evaluation of any preliminary research (and data) in the Stage 1 submission (see Section 2.7) is included within this criterion.

1C. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis or alternative sampling plans where applicable). This criterion assesses the validity of the study procedures and analyses, including the presence of critical design features (e.g. internal and external validity, blinding, randomisation, rules for data inclusion and exclusion, suitability of any included pilot data) and the appropriateness of the analysis plan. For designs involving inferential statistics and hypothesis-testing, this criterion includes the rigour of the proposed sampling plan, such as a statistical power analysis or Bayesian alternative, and, where applicable, the rationale for defining any statistical priors or the smallest effect size of interest. For programmatic RRs (see Section 2.15), this criterion captures the assessment of whether the separate study components are sufficiently robust and substantive to justify separate Stage 2 outputs.

1D. Whether the clarity and degree of methodological detail is sufficient to closely replicate the proposed study procedures and analysis pipeline and to prevent undisclosed flexibility in the procedures and analyses. This criterion assesses the extent to which the Stage 1 protocol contains sufficient detail to be reproducible and ensure protection against research bias, such as analytic overfitting or vague study procedures. In general, meeting this requirement will require the method section(s) of a Stage 1 protocol to be significantly longer and more detailed than in a regular manuscript, while also being clearly structured and accessible to readers. This criterion also covers the extent to which the protocol specifies precise and exhaustive links between the research question(s), hypotheses (where applicable), sampling plans (where applicable), analysis plans, and contingent interpretation given different outcomes. Authors are strongly encouraged to include a design summary table in their Stage 1 protocols that make these links clear (see Section 2.16 for examples). Note that in some circumstances, authors may wish to propose a more general analysis plan involving a blinded analyst rather than a precise specification of data analyses. Such submissions are acceptable and will meet this criterion provided the blinded analysis procedure is specified in reproducible detail, and provided the blinding procedure itself is sufficiently robust.

1E. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the obtained results are able to test the stated hypotheses or answer the stated research question(s). This criterion addresses, where applicable, the extent to which the proposal pre-specifies data quality checks that will reveal whether the results are able to answer the research question(s). Such tests come in many forms and, depending on the design or subject area, are not always possible or appropriate. Where they are proposed they must be pre-specified and “outcome-neutral”, meaning that they are designed prior to knowledge of the results and are independent of the main study hypotheses. In experimental studies, such tests often consist of positive controls (also referred to in different fields as “manipulation checks” or tests of “intervention fidelity”). In observational studies they might include parallel measurements to replicate an established fact, thus confirming that the design or instrumentation is sufficiently sensitive to detect the existence of the phenomenon that forms the main focus of the research. This 1993 study which used the Galileo spacecraft to search for signs of life on Earth provides a signature example of a positive control.

Once a Stage 1 protocol receives IPA, the manuscript is formally registered in a repository (see Section 2.8). The Stage 2 manuscript reporting the completed research is then submitted and reviewers are asked to assess it according to the following criteria:

2A. Whether the data are able to test the authors’ proposed hypotheses (or answer the proposed research question) by passing the approved outcome-neutral criteria, such as absence of floor and ceiling effects or success of positive controls or other quality checks. This criterion addresses whether the data quality is sufficient to be able to test the stated hypotheses, according to the pre-specified conditions in 1E. Since not all protocols are able to pre-specify outcome-neutral tests, this assessment is not relevant to all forms of RRs. Where it is relevant and pre-specified, it is possible that the failure of a crucial outcome-neutral test could, in severe circumstances, lead to the rejection of a Stage 2 manuscript.

2B. Whether the introduction, rationale and stated hypotheses (where applicable) are the same as the approved Stage 1 submission. This criterion assesses whether the authors have remained consistent in their framing of the study at Stage 2. Aside from changes in tense (e.g. future tense to past tense), correction of typographic and grammatical errors, and correction of clear factual errors, the introduction, rationale and hypotheses of the Stage 2 submission must remain identical to those in the approved Stage 1 manuscript. To make any changes clear, authors are required to submit a tracked changes version of the manuscript at Stage 2.

2C. Whether the authors adhered precisely to the registered study procedures. This criterion assesses compliance with protocol. In cases where the preregistered protocol is altered after IPA due to unforeseen circumstances (e.g. change of equipment or unanticipated technical error), the authors must consult the PCI RR recommender immediately for advice, prior to the completion of data collection. Minor changes to the protocol may be permitted per recommender’s discretion. In such cases, IPA would be preserved and the deviation reported in the Stage 2 submission. If the authors wish to alter the study procedures more substantially following IPA but still wish to publish their article as a Registered Report then the manuscript must be withdrawn and resubmitted as a new Stage 1 submission. The outcome of all preregistered analyses must be reported in the manuscript, except in rare instances where a preregistered and approved analysis is subsequently shown to be logically flawed or unfounded. In such cases, the authors, reviewers, and recommender must agree that a collective error of judgment was made and that the analysis is inappropriate. In such cases the analysis would still be mentioned in the Stage 2 method but omitted with justification from the results. Additional unregistered analyses can also be included in a final manuscript (see 2D).

2D. Where applicable, whether any unregistered exploratory analyses are justified, methodologically sound, and informative. This criterion addresses the quality and value of any additional data analyses that are reported at Stage 2 but were not included in the registered Stage 1 submission. Such analyses are often highly valuable. For instance, a new analytic approach might become available between IPA and Stage 2 review, or a particularly interesting and unexpected finding may emerge. Alternatively, some unexpected characteristic of the data might suggest that the preregistered analyses, while bias-free, are not as sensitive as planned, and therefore a more sensitive post hoc analysis could be informative. Such analyses are admissible but must be clearly identified (e.g. through a separate heading in the Results for “Exploratory Analyses” or “Unregistered analyses”), justified, and appropriately caveated. Authors should also be careful not to base their conclusions entirely on the outcome of unregistered analyses.

2E. Whether the authors’ conclusions are justified given the evidence. This criterion addresses whether the claims drawn by the authors in their conclusions (including in the Discussion, Abstract, and anywhere else in the paper) are warranted by the data or evidence in hand. Note that PCI RR recommendation decisions will never be based on the perceived importance, novelty, or conclusiveness of the results.

2.2 Replications and original research

PCI RR welcomes submissions proposing original research and replication studies. All submissions are assessed according to the same criteria, and novelty or originality of the research is irrelevant, provided the research question is judged to be scientifically valid (review criterion 1A).

2.3 Evidence thresholds

Forms and standards of evidence vary widely across quantitative and qualitative research disciplines. Given the broad remit of PCI RR, this disciplinary heterogeneity precludes the setting of any universal standard across all submissions. Instead, the following general policies apply.

For quantitative research fields that do not draw conclusions based on inferential statistical analyses (including many physical sciences), authors should ensure that their Stage 1 submissions plan to obtain a sufficient strength of evidence to draw a clear conclusion (e.g. to confirm/disconfirm a hypothesis), at a threshold considered rigorous within the field. The same general expectation applies to qualitative research areas. Reviewers and recommenders will assess on a case-by-case basis whether the strength of evidence likely to arise from the study is sufficient to meet criterion 1C.

For quantitative sciences that do employ inferential statistics, the specific requirements differ depending on the analytic methods. Studies involving Neyman-Pearson inference should include a priori statistical power analysis, with estimated effect sizes justified with reference to the existing literature or theory. Since publication bias overinflates published estimates of effect size, power analysis should be based on the lowest available or meaningful estimate of the effect size. In the case of highly uncertain effect sizes, a variable sample size and interim data analysis can be undertaken but with inspection points stated in advance, appropriate Type I error correction for ‘peeking’ employed, and a final stopping rule for data collection outlined. A range of freely available general tools are available for implementing power analysis, including G*Power, ANOVAPower (article, Shiny App, source code on Github), and PANGEA, as well as specialised tools for estimating power using particular methods, such as the SIMR package for linear mixed models, and a method for writing your own power analyses (McElreath 2020).

PCI RR recommends that authors who intend to test directly for invariance between conditions (i.e. evidence of absence) should consider using frequentist equivalence testing or Bayesian analyses. For studies involving analyses with Bayes factors, the predictions of the theory must be specified so that a Bayes factor can be calculated. Authors should indicate what distribution will be used to represent the predictions of the theory and how its parameters will be specified.  For example, will the authors use a uniform distribution up to some specified maximum, or a normal/half-normal distribution to represent a likely effect size, or a JZS/Cauchy distribution with a specified scaling constant? For inference by Bayes factors, authors should aim to obtain a strength of evidence that is likely to be useful to readers (e.g., that a Bayes factor of a particular magnitude will be suitably convincing for the hypothesis in question). If the stopping rule is dependent on the Bayes factor, authors should indicate a maximum feasible sample size after which sampling will stop, regardless of the Bayes factor. In this case, reviewers should judge whether the maximum possible sample size is sufficiently large that an inconclusive Bayes factor at that sample size would nevertheless provide an important message for the field.

For further advice on Bayes factors or Bayesian sampling methods, we recommend that authors and reviewers consult this key article by Schönbrodt and Wagenmakers on Bayes Factor Design Analysis. These two articles by PCI RR Managing Board member Zoltan Dienes (here and here) also provide practical advice for specifying theoretically relevant effect sizes for statistical hypothesis testing.

In some fields, custom tools for statistical sample planning are available. For example, in functional magnetic resonance imaging, neuropowertools and fmripower can be used, and authors are welcome to use bespoke methods involving simulations. In place of conventional hypothetico-deductive methods, authors are welcome to propose more innovative analytic approaches such as integration of Bayesian optimisation and hypothesis testing. Authors proposing studies that do not involve hypothesis testing are welcome to propose alternative methods for establishing an evidence threshold, such as planning for a sufficiently precise parameter estimate or confidence interval.

Although PCI RR sets no minimum requirement for statistical power, Bayes factors, or the precision of parameter estimates, submissions that reviewers judge to fall short of sufficient rigour in these areas are likely to be rejected under criterion 1C. Furthermore, PCI RR advises authors intending to have their RR automatically accepted by a PCI RR-friendly journal to ensure that the evidential standard in their submission meets the minimum requirement of their preferred journal (e.g. in terms of power, alpha, Bayes factors, or any other conditions). All such conditions can be found in the current list of journal adopters. Reviewers need not take into account journal requirements when assessing submissions – all that matters for the review process is that the submission meets the requirements set by PCI RR. Finally, we note that these requirements apply strictly to a priori planning – the clarity of evidence actually obtained and reported at Stage 2 is irrelevant both to the final recommendation and to the automatic acceptance of recommended submissions by PCI RR-friendly journals.

2.4 Ethics approval

In fields where research requires approval by an ethics committee or institutional review board, PCI RR expects authors to ensure that the proposed research has received all necessary approvals prior to Stage 1 submission. However, we also recognise that in some cases, obtaining prior ethics approval may not be the most efficient course of action, for instance where the ethics committee will only review and approve a fixed, detailed protocol (as can be the case in clinical settings). Therefore, in specific circumstances, PCI RR will perform Stage 1 review prior to ethics approval. In such cases, reviews and feedback will be conveyed to authors, but any IPA recommendation will be provisional, pending confirmation from the authors that the provisionally accepted protocol has received the necessary ethics approval. Note that this provisional IPA will not be sufficient to trigger automatic acceptance by PCI RR-friendly journals, all of which require the necessary ethics approval to be in place. Reviewers are welcome to raise any ethical concerns with submissions, which will be assessed by PCI RR recommenders under criterion 1A.

2.5 Funding approval

In general, authors should ensure that sufficient funding and resources are in place to support the proposed research prior to Stage 1 submission. There are, however, two exceptions to this rule. First, under certain circumstances (such as for proposals tackling issues of major public importance or as part of targeted funding initiatives), PCI RR may agree to undertake Stage 1 review prior to a project being funded. Any IPA recommendation issued by PCI RR under such circumstances will be provisional, pending confirmation that the Stage 1 protocol has received sufficient funds to be undertaken. Authors considering a Stage 1 manuscript prior to receiving funding are advised to consult the Managing Board for advice prior to submission (contact@rr.peercommunityin.org).

The second exception to this rule is that PCI RR is happy to work directly with funders in support of Registered Reports Funding Partnerships (RRFPs). Regular RRs require authors to already have grant funding in place, but RRFPs take place earlier in the research cycle, before researchers have secured funding. Under our RRFP model, the funder and PCI RR will review a Stage 1 RR (or set of RRs) concurrently (or near concurrently), and if both PCI RR and the funder judge the proposal to be of sufficient quality then the funder will award funding at the same time as PCI RR awards IPA. This model has the advantage of compressing two phases of pre-study review (grant review and Stage 1 RR review) into a single review process, and it can also be integrated with ethics and regulatory review. Existing RRFPs have already been established between organisations including the Children’s Tumor Foundation and PLOS ONE, and Cancer Research UK and the journal Nicotine and Tobacco Research. Authors then have the opportunity to publish their recommended RRs in a wide range of reputable PCI RR-friendly journals (or in no journal, as the authors prefer). We welcome discussion from funders who may be interested in participating in our RRFP programme.

2.6 RRs involving existing data

Choosing a primary data analysis strategy after observing data can increase the risk of error resulting from hindsight bias. For this reason, some journals will only consider Stage 1 RRs for which the data do not yet exist and thus prior observation is impossible (so-called “primary RRs”; see Level 6 in Table 1, below). However, given the broad disciplinary scope of PCI RR, and the consequent need to ensure inclusivity across a wide range of subject areas, our policy is to consider both primary RRs and also “secondary RRs”: submissions where the data that will be used to answer the research question already exist. Where there is a potential risk of bias due to prior observation of the data, achieving Stage 1 IPA will require authors to take active steps to address this risk. Such measures could include providing proof that the authors have not yet accessed the data (Level 5) or, where such proof is unavailable, self-certifying that no such access has occurred (Level 4). Where authors have already accessed the data, then depending on the risk of bias, authors must be able to self-certify that they have not yet observed the data (Level 3), or that they have not observed key variables within the data (Level 2), or that they have not yet performed the proposed analyses on the key variables in the data (Level 1). Submissions at Level 1 or 2 will usually be required to include stringent countermeasures against overfitting, such as the adoption of conservative inferential statistical thresholds, recruitment of a blinded analyst, or multiverse/specification analysis (for further guidance on these approaches see articles here, here, here and here). Note that PCI RR will not consider studies where the authors already know the outcomes of the prospective (planned) analyses at the point of Stage 1 submission. In addition, these levels apply only to data that form the focus of the prospective (planned) analyses in the a Stage 1 manuscript, and do not apply to any completed preliminary studies or pilot data that are reported at Stage 1 (as described in Section 2.7).

As part of the Stage 1 submission process, authors will be asked to identify which level applies to their proposal, and this selection (Level 1-6) will appear next to the abstract on the PCI RR website following a final positive Stage 2 recommendation. Where the prospective component of a Stage 1 proposal includes research in which different parts of the work are undertaken at different levels (e.g. using different datasets) then the article as a whole will be assigned the lowest level among the set. For example, where some data have not yet been collected (Level 6) but other data to be used in the planned (prospective) analysis have already been acquired and partially observed (Level 2) the article as a whole would be designated Level 2. Authors are advised that some PCI RR-friendly journals require a higher level of protection against bias than the minimum level required by PCI RR. Therefore, authors who intend to publish their accepted RR in a journal should ensure that they know which journals will automatically accept and publish their submission following IPA. Details concerning these minimum requirements can be found at the PCI RR-friendly Journal Adopters page.

As part of accepting submissions proposing analyses of existing data, PCI RR welcomes the submission of prospective meta-analyses, systematic reviews, scoping reviews, umbrella reviews, and systematic maps, provided the authors can achieve Level 1 or higher. Note that meta-analyses, systematic reviews, scoping reviews, umbrella reviews, and systematic maps can never achieve Level 6 because the data that are included within them by definition must already exist. Most such submissions achieve Level 3, 2 or 1; however, the requirement for additional steps to ensure stringent bias control for these article types at Level 2 and Level 1 will generally be waived.

A key variable is considered to be observed if the raw data that will go into the analysis that answers the research question has been seen. For example, a Stage 1 RR would be a level 2 if authors have observed some, but not all of the response variables in a given analysis (e.g., the response variable has been observed, but not the independent variables or vice versa). In a meta-analysis example, a key variable would be considered to be observed if the data extracted from the literature were the effect sizes, but it would not be considered observed if the response variable used data collected from the literature (observed), but needed to undergo a calculation to obtain effect sizes (unobserved) as part of the analysis.

Table 1. Levels of bias control recognised by PCI RR for submissions proposing analyses of new data (Level 6) or existing data (Levels 1-5). Submissions from Level 1 upward are eligible for recommendation by PCI RR. However, authors intending to publish their RR in a journal should be aware that many journals set a higher minimum threshold than PCI RR. Authors should therefore ensure that their achieved level meets the minimum requirement of their preferred journal. If authors have an inflexible data collection start date and have not received in principle acceptance (IPA) before this date, they may begin collecting data but must adjust the bias-control level accordingly (e.g., if the initial submission was Level 6, it would then drop to Level 3, 2, or 1). There are several points to consider when dropping to a lower bias-control level. First, there is a greater risk of Stage 1 rejection if concerns with the study procedures raised in the Stage 1 review process can no longer be addressed due to data collection commencing and crucial parts of the methodology being immutable from that point forward. Second, the number of PCI RR-friendly journals that will automatically accept the Stage 2 RR may be reduced because adopting journals can set a minimum bias-control level that exceeds the requirements of PCI RR (for example, the submission would become ineligible for automatic acceptance in a PCI RR-friendly journal that sets a minimum requirement of Level 4 or higher). Third, as explained in the table, reducing the bias-control level increases the stringency of steps required to minimise bias and increase rigour (e.g., through the adoption of a more conservative statistical threshold, or blinded analyst, etc). Finally, it is essential that, despite the drop in bias-control level, the manuscript remains at Level 1 or higher: if authors begin to discover the conclusions (or likely conclusions) of the research prior to IPA then they would risk dropping to Level 0 and the manuscript would no longer be eligible for consideration at PCI RR.

2.7 Preliminary and pilot studies

Authors are welcome to include preliminary studies in their Stage 1 submissions to establish the research question, generate hypotheses, estimate effect sizes for subsequent hypothesis testing, or to demonstrate the feasibility of their proposed methods. Such work can significantly strengthen a Stage 1 submission, for instance by confirming the effectiveness of proposed positive controls or other outcome-neutral checks (criterion 1E). Any preliminary studies will be published in the Stage 2 manuscript and must be clearly distinguished from preregistered protocol. We ask reviewers to assess whether the prospectively proposed research at Stage 1 is sufficiently justified and rigorous that it can provide an answer to the research question, independently of the outcomes of any preliminary studies. In other words, authors should not rely on the outcomes of any preliminary results to establish the utility or overall conclusions of the research. Submissions that fall short in this respect are likely to be rejected under criteria 1A, 1B or 1C.

2.8 Protocol preregistration

At the point of Stage 1 IPA, PCI RR will preregister the approved manuscript on behalf of authors using the dedicated Stage 1 RR preregistration mechanism provided by the Open Science Framework. The Stage 1 manuscript will either be made public or preregistered under a private embargo until submission of the Stage 2 manuscript. As part of the Stage 1 submission checklist, authors can choose a private embargo of up to 4 years from the date of IPA. Note that PCI RR will preregister the protocol only once the Stage 1 manuscript receives IPA, and not if it is rejected or withdrawn by authors prior to being awarded IPA.

The Stage 1 manuscript will be preregistered by PCI RR unchanged from its approved state, and the URL to the preregistered protocol will be included in the Stage 1 acceptance letter.  Authors must confirm at submission that if they withdraw their paper following Stage 1 IPA then they agree to PCI RR (a) lifting any applicable private embargo on the preregistered Stage 1 protocol, thus making the protocol public on the Open Science Framework; and (b) publishing a short summary of the preregistered study under the section Withdrawn Registrations, which will include the abstract of the Stage 1 submission, the URL of the preregistered Stage 1 protocol on the OSF (with any applicable embargo lifted), and a stated reason for the withdrawal (see Section 2.12).

Note that the abstract of the Stage 2 manuscript must contain a direct URL to the preregistered Stage 1 protocol and state the date of preregistration, e.g. “Preregistered Stage 1 protocol: URL (date of in-principle acceptance: DD/MM/YYYY)”. If the authors choose to preregister their accepted Stage 1 manuscript under a private embargo, then the embargo will be lifted by PCI RR and the preregistered protocol made public when any one of the following conditions is met: (a) submission of the Stage 2 manuscript; (b) withdrawal of the submission after in-principle acceptance and consequent triggering of a Withdrawn Registration; or (c) natural expiry of the embargo period.

To be eligible for consideration by PCI RR, authors must agree to these conditions at the point of the Stage 1 submission. Authors can also separately register their protocol in a trusted repository subject to their own discretion or disciplinary requirements; for instance, for clinical trials, authors should proceed with trial registration in the regular way at the point of Stage 1 IPA.

2.9 Inclusion of registered and unregistered analyses at Stage 2

The outcome of all registered analyses must be reported in the manuscript, except in rare instances where a registered and approved analysis is subsequently shown to be so severely flawed that the outcomes it generates are either meaningless or would be grossly misleading. If the authors, reviewers, and recommenders collectively agree that the analysis is inappropriate, then it would still be mentioned in the Methods but omitted with justification from the Results.

It is reasonable that authors may wish to include additional analyses that were not included in the registered submission. For instance, a new analytic approach might become available between IPA and Stage 2 review, or a particularly interesting and unexpected finding may emerge. Such analyses are admissible but must be justified in the text, appropriately caveated, and clearly distinguished from the preregistered outcomes. For example, authors could include such results in a section called “Exploratory analyses”. To ensure that criterion 2E is met, authors should be careful not to base their conclusions entirely on the outcome of unregistered analyses. At the same time, reviewers at Stage 2 may suggest that authors report additional tests on their data; however, authors are not obliged to conduct such tests unless they are necessary to satisfy one or more of the Stage 2 criteria.

Finally, authors should note that while unregistered analyses are acceptable, the inclusion of additional unregistered studies is not permitted at Stage 2. Authors wishing to add further studies at Stage 2 should instead submit an Incremental Registration (see Section 2.13).

2.10 Protocol deviations and changes at Stage 2

Authors are reminded that any deviation from the approved study procedures following Stage 1 acceptance, regardless of how minor it may seem to the authors, could lead to rejection of the manuscript at Stage 2 (criterion 2C). In cases where the preregistered protocol is altered after IPA due to unforeseen circumstances (e.g. change of equipment or unanticipated technical error), the authors must consult the recommender immediately for advice, and prior to the completion of data collection. Changes to the protocol that are scientifically essential and do not significantly increase risk of bias are likely to be swiftly approved. In such cases, IPA would be preserved and the deviation reported in the Stage 2 submission (note that authors are asked to describe what changed, why, when the change occurred in the research process - e.g., after data collection started, but before any data analyses - how scientific integrity was maintained, and to note that the recommender approved of this change). If the authors wish to alter the study procedures more substantially following IPA, but still wish to gain a PCI RR recommendation, then the manuscript must either be readmitted to Stage 1 review or withdrawn and resubmitted as a new Stage 1 submission. Note that registered analyses must be undertaken, but additional unregistered analyses can also be included in the final manuscript (see Section 2.9).

In general, the parts of the Stage 1 manuscript that were approved at the point of IPA (including the Introduction and Method) should remain the same at Stage 2, with the following exceptions:

• Any description of the rationale or proposed methodology that was written in future tense within the Stage 1 manuscript should be changed to past tense.
• The Abstract of the Stage 2 submission should be updated to include an overview of the results and conclusions.
• The title of the manuscript can be changed to reflect the results and conclusion.
• Any typographical or outright factual errors should be corrected.
• Authors can be added or removed without needing to seek the approval of the PCI RR recommender.
• Where there were any deviations from the accepted methodology, the Method section of the Stage 2 manuscript must be altered to state the method that was implemented, accompanied by a footnote to explain the nature of, and reason for, the deviation from the planned methodology.

Reviewers are welcome to recommend relevant new literature at Stage 2; however, note that any such literature will be covered in the Discussion rather than the Stage 2 Introduction. In general, any suggestion from reviewers that authors should update the literature in the Introduction will be overruled by the recommender, and no part of the research question, rationale, or hypotheses (as applicable) can be altered at Stage 2.

PCI RR requires that all textual changes to the parts of the Stage 2 manuscript that were approved at Stage 1 are without exception highlighted as tracked changes in the Stage 2 manuscript. Authors can meet this requirement either by highlighting the tracked changes in the published Stage 2 preprint or, where they prefer to publish a “clean” Stage 2 preprint, by linking to a duplicate version of the manuscript (e.g. in a trusted repository) that highlights the tracked changes.

2.11 Data and materials transparency

PCI RR is a signatory of the Transparency and Openness Promotion (TOP) guidelines, which describe a series of modular standards for transparency and reproducibility in published research. In general, authors are required to make all study data, digital materials, and computer code publicly available (at Stage 2 submission) to the maximum extent permissible by relevant legal or ethical restrictions. Authors are expected to consult the PCI RR TOP guidelines policy (available here) prior to Stage 1 submission. Reviewers are not required to assess whether the manuscript is TOP-compliant, as this assessment is made by PCI RR recommenders.

2.12 Withdrawn registrations

It is possible that authors with IPA may wish to withdraw their manuscript from PCI RR, during or following the research. Possible reasons could include (but are not limited to) a major technical error, inability to complete the study due to unforeseen circumstances, or the authors declining to revise the Stage 2 manuscript as requested by the recommender or Managing Board. In all such cases, manuscripts can of course be withdrawn at the authors’ discretion. However, PCI RR will publicly record each withdrawal by marking the Stage 1 recommendation as "withdrawn" and including a statement in the Stage 1 recommendation explaining the key facts and the reason for the withdrawal. If the recommended Stage 1 protocol was registered under a private embargo then this will also be released and the protocol made public.

Partial withdrawals are generally not possible; i.e. with a regular RR, authors cannot publish part of a registered plan by selectively withdrawing one of the planned studies. Such cases must lead to withdrawal of the entire paper. The exception to this policy is where one component of an accepted programmatic RR is withdrawn (see Section 2.15). For programmatic RRs, entire components that are planned as separate Stage 2 outputs can be withdrawn without affecting the IPA of the remaining components. In such cases, each remaining Stage 2 output must include a URL to the Withdrawn Registration of the withdrawn component.

Should Stage 1 IPA be forthcoming, authors will be asked to provide PCI RR with an estimated submission date for the completed Stage 2 manuscript. This deadline can be readily altered in consultation with the recommender (e.g. in case of delays requiring additional time to complete the research). However, in the event that the authors (a) fail to submit the Stage 2 manuscript within 6 months of the mutually agreed deadline, while also (b) becoming non-responsive during this period to enquiries, then the manuscript will be considered by PCI RR to be withdrawn.

2.13 Incremental registration

Authors may add studies to approved submissions. In such cases the approved Stage 2 manuscript will be recommended, and authors can propose additional studies for Stage 1 consideration. Where these studies extend the approved submission (as opposed to being part of new submissions), the recommender will seek to fast-track the review process. This option may be particularly appropriate where an initial study reveals a major serendipitous finding that warrants follow-up within the same article. In cases where an incremented submission is rejected (at either Stage 1 or 2), authors will retain the option of having the recently approved version of the manuscript recommended. For further advice on specific scenarios for incremental registration, authors are invited to contact the PCI RR Managing Board (contact@rr.peercommunityin.org).

2.14 Qualitative research

PCI RR recognises the value of qualitative research and commits to treating qualitative and quantitative submissions with equal priority. Qualitative studies will be assessed according to the same Stage 1 and Stage 2 criteria, with the exception that certain requirements (e.g. concerning statistical methods) that apply only to quantitative research will not apply to qualitative submissions. Reviewers assessing a qualitative submission can consult published guidance for preregistration under these circumstances. This template, which is based on this additional guidance, also provides a useful starting point.

2.15 Programmatic RRs for large or long-term research programmes

To make RRs accessible to a wide variety of fields and working styles, PCI RR offers authors the option to publish multiple Stage 2 articles from a single “programmatic” Stage 1 RR. The programmatic RR track can be useful for large or long-term projects where multiple studies contribute to a broader goal, and where the size or scope of the project is too great for a single published output. Authors of a programmatic Stage 1 RR should ensure that all the usual criteria for a RR are met, including detailed specification (where applicable) of theory, hypotheses, procedures, and analysis plans (see review criteria in Section 2.1). In addition, authors should ensure that the manuscript makes clear how the different elements of the programme – including different sets of research questions, hypotheses, and methods – fit together to form a cohesive package.

A Stage 1 programmatic RR must also prespecify which parts of the protocol will eventually produce separate Stage 2 articles, and as part of the Stage 1 review process, reviewers and recommenders will evaluate of the validity and substantive contribution of each component (as noted in criterion 1C; see Section 2.1). These prespecified boundaries are effectively treated as design elements; therefore, like any other design element, Stage 1 IPA will be contingent on authors adhering to the prespecified and approved article boundaries at Stage 2.

Following completion of each Stage 1 component, authors will submit a separate Stage 2 manuscript to PCI RR that includes the introduction, methods, results, and conclusions of each relevant component of the Stage 1 protocol. Because each Stage 2 submission relates only to a portion of the approved Stage 1 protocol, it is understandable that authors may need to make greater structural amendments to the introduction and method sections of the Stage 2 manuscripts compared with a regular RR (including only the relevant content). Nevertheless, to ensure that Stage 2 criterion 2B is met (see Section 2.1), authors are reminded to make as few changes as necessary, being sure to keep the hypotheses, rationale and methodological descriptions of the relevant component in each Stage 2 manuscript as similar as possible to its corresponding component in the approved Stage 1 manuscript.

Because programmatic RRs are designed to support long-term projects, authors have the option to submit Stage 2 manuscripts corresponding to the relevant Stage 1 components as they go, receiving positive recommendations and publishing the outputs serially in journals (including any PCI RR-friendly journals). In each case, the Stage 2 manuscript must include a section that (a) states that the current manuscript is one part of the larger protocol (listing the public URL to the protocol), (b) cites all previously published Stage 2 outputs arising from the same protocol, and (c) notes which, if any, Stage 2 components arising from protocol are either awaiting completion or have either been rejected at Stage 2 by PCI RR or formally withdrawn by the authors.

Where authors elect for their approved programmatic RR to be registered at the point of IPA under a temporary private embargo, please note that the embargo will be released - and the entire protocol made public - at the point that the first Stage 2 manuscript of the planned series of outputs is received by PCI RR. This policy means that parts of the Stage 1 protocol will be made public that are not (yet) linked to a completed Stage 2 manuscript. If authors do not want this to happen then they should submit their programme of research as a series of separate regular RRs rather than as a single programmatic RR.

In case of a Withdrawn Registration, entire components that are planned as separate Stage 2 outputs can be withdrawn without affecting the IPA of the remaining components (see Section 2.12). However, just as part of a regular RR cannot be withdrawn (without withdrawing the entire RR), neither can part of a single component within a programmatic RR.

2.16 Study design template

Depending on the nature of the research, PCI RR either encourages or requires authors to include a study design template in their Stage 1 submissions to ensure clear linking between the research question, hypotheses (where applicable), analysis plans, and prospective interpretation given different outcomes. For full Stage 1 manuscripts (not RR  snapshots) describing quantitative research that involves the testing of hypotheses, inclusion of a study design template is a mandatory requirement; for all other quantitative and qualitative research modes it is strongly encouraged but not required.

Not all parts of the template will be relevant or interpreted equivalently for all fields – for example, the “hypothesis” column can be omitted where the study is not hypothesis-driven, and, for fields that do not employ inferential statistical analyses, the “sampling plan” column could describe the quantity of the data/evidence that will be collected rather than a statistical sampling plan. For published examples of study design templates, authors are pointed to the following pre-accepted Stage 1 RRs at Royal Society Open Science in the fields of chemistry, immunology, virology and psychology.

A blank PCI RR study design template is shown below with guidance notes. Each row should correspond to a different entry, and cells can be vertically merged to create a nested structure, e.g. where one research question (column 1) will be answered by multiple hypotheses. An editable version of the template may be downloaded here.

Guidance Notes

• Question: articulate each research question being addressed in one sentence.
• Hypothesis: where applicable, a prediction arising from the research question, stated in terms of specific variables rather than concepts. Where the testability of one or more hypotheses depends on the verification of auxiliary assumptions (such as positive controls, tests of intervention fidelity, manipulation checks, or any other quality checks), any tests of such assumptions should be listed as hypotheses. Stage 1 proposals that do not seek to test hypotheses can ignore or delete this column.
• Sampling plan: For proposals using inferential statistics, the details of the statistical sampling plan for the specific hypothesis (e.g power analysis, Bayes Factor Design Analysis, ROPE etc). For proposals that do not use inferential statistics, include a description and justification of the sample size.
• Analysis plan: For hypothesis-driven studies, the specific test(s) that will confirm or disconfirm the hypothesis. For non-hypothesis-driven studies, the test(s) that will answer the research question.
• Rationale for deciding the sensitivity of the test for confirming or disconfirming the hypothesis: For hypothesis-driven studies that employ inferential statistics, an explanation of how the authors determined a relevant effect size for statistical power analysis, equivalence testing, Bayes factors, or other approach.
• Interpretation given different outcomes: A prospective interpretation of different potential outcomes, making clear which outcomes would confirm or disconfirm the hypothesis.
• Theory that could be shown wrong by the outcomes: Where the proposal is testing a theory, make clear what theory could be shown to be wrong, incomplete, or otherwise inadequate by the outcomes of the research.

2.17 Submission workflow

Figure 1 (below) shows the PCI RR submission workflow for Stage 1 and 2 manuscripts.  At Stage 1, authors can choose between a fully public track or a temporarily embargoed track. Both tracks require authors to deposit their manuscript in a recognised repository (e.g. OSF, Zenodo), but the embargoed track permits authors to keep the manuscript private and share a view-only URL for peer review.

Note that PCI RR does not separately publish recommended RRs. Instead, PCI RR recommends manuscripts deposited on a preprint server (e.g., ArXiv, BioRXiv, Zenodo) or repository (e.g. OSF). Authors who intend to publish their recommended RR in a journal should, therefore, note that publication of the Stage 1 and/or Stage 2 manuscript will fall within the terms of the journal’s preprint policy (viewable here) and does not constitute dual publication. Where authors intend to eventually publish their RR in a journal, they need not commit to a specific journal until the final Stage 2 manuscript is recommended by PCI RR.

Figure 1. Submission and review workflow for Stage 1 and Stage 2 submissions at PCI RR.

2.18 Open review and anonymous review

PCI RR publishes all reviews of recommended manuscripts, with reviewers retaining the choice to sign their reviews or remain anonymous (unless one of the authors is a PCI RR Managing Board member, in which case all reviews must be signed. This is because Managing Board members have access to confidential information about all submissions, which poses an ethical problem if a reviewer wishes to remain anonymous from the authors). All Stage 1 and 2 reviews and recommender decision letters are published on the PCI RR platform no later than the point of Stage 2 acceptance. Reviews of rejected submissions are not published and we ask that authors and reviewers refrain from self-publishing (e.g. on the OSF) the content of the reviews, either during the review process or following rejection. Reviewers and recommenders are expected to adhere to the PCI RR code of conduct, avoiding abusive or discriminatory language in their comments. Reviews that are judged to violate the code of conduct may be returned to the reviewer for editing, or discarded.

PCI RR also offers authors the opportunity to submit anonymously at Stage 1 (but not at Stage 2). To submit anonymously, authors should remove all nominative information from the manuscript and supplementary files (including the file properties). Authors can then post their article and supplementary files on a server or repository (e.g. OSF - following the instructions here) which allow authors to share the files via an anonymous URL. When the article has appeared on this server, authors can submit it to PCI RR. See the How To...? section for further details.

Reviewers are welcome to co-review registered reports with a junior academic in their group. To credit the co-reviewer, please email the recommender and ask them to invite the junior academic as a reviewer. After they have accepted the invitation and the joint review has been written, paste the complete review at the junior academic's review space. In the senior academic's review space, please state "Joint review: see review by X for details". If the reviews are anonymous, then state "Joint review: see review below for details" and note in the junior academic's review text that this is the joint review.

2.19 Recommender triage

As shown in Figure 1 (see Section 2.17), all submissions are evaluated by one or more recommenders prior to in-depth peer review as part of the triage process. Only submissions that PCI RR recommenders judge to meet, or nearly meet, the respective review criteria will be sent for specialist peer review. Submissions that fail to meet the review criteria, but which the recommenders judge have the potential to do so, will be desk-rejected with the invitation to resubmit, including specific suggestions for revision. Submissions that fall substantially short of meeting the review criteria will be desk-rejected outright.

2.20 Standard review and scheduled review

PCI RR offers authors two tracks for Stage 1 peer review. Standard review (Figure 2A) is the typical track in which authors submit a complete and detailed Stage 1 manuscript for evaluation. Authors can expect the first round of Stage 1 review via this track to take ~4-8 weeks. Scheduled review (Figure 2B) aims to significantly accelerate this process by performing key tasks in parallel. Rather than submitting a full manuscript, authors initially submit a one-page, template-based RR “snapshot” which undergoes triage by a PCI RR recommender. If deemed suitable (which could require revision of the RR snapshot), the recommender then sends the RR snapshot to potential reviewers and arranges for the review process to take place during a narrow range of dates (1 week) at least 6 weeks in the future, with the submission date nominated by the authors. During the intervening time the authors prepare the full manuscript. Although this process can only feasibly expedite the first round of Stage 1 review (and not the re-review of a revised Stage 1 submission), the overall time-saving is substantial because the first round of Stage 1 assessment is usually the most onerous. Authors intending to submit to the scheduled review track must do so using the PCI RR 1-page snapshot template available here.  Submissions that do not use this template will not be considered. The Standard and Scheduled review tracks are available for both Regular and Programmatic RR submissions.

Figure 2. The submission timelines for (A) standard review and (B) scheduled review. Scheduled review, where feasible for authors, can significantly reduce the duration of the first round of Stage 1 review. Figure adapted from Chambers & Tzavella (2020).

2.21 Appeals process

Authors of rejected manuscripts may appeal the PCI RR recommendation by emailing the PCI RR Managing Board or the recommender who handled the manuscript. Appeals must be received within 30 days of PCI RR issuing a decision. The email must clearly state the case for why the decision should be changed. Appeals will only be considered if the authors a) identify factual errors made by the reviewers or recommender that had a major impact on the decision, or b) can provide a substantiated claim of unfair treatment and/or bias in the review process. Appeals for any other reason will be denied without further consideration. Appeals that meet the identified criteria will be discussed among a panel that includes the recommender and two members of the Managing Board. The panel will review the appeal and vote to uphold or reverse the original decision. Final decisions will be based on majority vote (i.e., decisions need not be unanimous). Decisions following appeal are final.

2.22 Word limits and formatting requirements

PCI RR imposes no word limits or specific formatting requirements on submissions. Manuscripts should be as concise as possible but as long as necessary to ensure that the description of rationale and methods is clear and comprehensive, and that all methods are reproducible. Authors should ensure that they follow established formatting conventions for articles in their discipline. Authors who intend to publish their RR in a journal should also take note of any sectional or overall word limits or other formatting requirements that apply to specific PCI RR-friendly journals (see here). Adhering to such conditions is not required for a PCI RR recommendation but may be necessary to ensure that the PCI RR recommendation is adopted by the intended journal.

---

3. Top tips for reviewers during Stage 1 and Stage 2 assessment

Because conventional peer review is so results-oriented, reviewing a RR can often feel unfamiliar at first. Even so, most reviewers quickly adjust to the requirements. Many reviewers find RR review to be a collegial and constructive process because there is still the opportunity at Stage 1 for expert feedback to help authors strengthen or rethink their study rationale and design. Reviewers who are already familiar with related forms of pre-study review, such as grant review and ethics review, will find RR review to be a similar process.

Sections 3.1 and 3.2 offer some suggestions for the types of questions reviewers should consider when assessing a Stage 1 or Stage 2 RR. For a high quality submission, the answer to each question (where the question is applicable) will usually be YES. Section 3.3 suggests some additional do’s and don’ts as a reviewer.

3.1 Key issues to consider at Stage 1

• Does the research question make sense in light of the theory or applications? Is it clearly defined? Where the proposal includes hypotheses, are the hypotheses capable of answering the research question?
• Is the protocol sufficiently detailed to enable replication by an expert in the field, and to close off sources of undisclosed procedural or analytic flexibility?
• Is there an exact mapping between the theory, hypotheses, sampling plan (e.g. power analysis, where applicable), preregistered statistical tests, and possible interpretations given different outcomes?
• For proposals that test hypotheses, have the authors explained precisely which outcomes will confirm or disconfirm their predictions?
• Is the sample size sufficient to provide informative results?
• Where the proposal involves statistical hypothesis testing, does the sampling plan for each hypothesis propose a realistic and well justified estimate of the effect size?
• Have the authors avoided the common pitfall of relying on conventional null hypothesis significance testing to conclude evidence of absence from null results? Where the authors intend to interpret a negative result as evidence that an effect is absent, have authors proposed an inferential method that is capable of drawing such a conclusion, such as Bayesian hypothesis testing or frequentist equivalence testing?
• Have the authors minimised all discussion of post hoc exploratory analyses, apart from those that must be explained to justify specific design features? Maintaining this clear distinction at Stage 1 can prevent exploratory analyses at Stage 2 being inadvertently presented as pre-planned.
• Have the authors clearly distinguished work that has already been done (e.g. preliminary studies and data analyses) from work yet to be done?
• Have the authors prespecified positive controls, manipulation checks or other data quality checks? If not, have they justified why such tests are either infeasible or unnecessary? Is the design sufficiently well controlled in all other respects?
• When proposing positive controls or other data quality checks that rely on inferential testing, have the authors included a statistical sampling plan that is sufficient in terms of statistical power or evidential strength?
• Does the proposed research fall within established ethical norms for its field? Regardless of whether the study has received ethical approval, have the authors adequately considered any ethical risks of the research?

3.2 Key issues to consider at Stage 2

• Have the authors provided a direct URL to the approved protocol in the Stage 2 manuscript? Did they stay true to their protocol? Are any deviations from protocol clearly justified and fully documented?
• Is the Introduction in the Stage 1 manuscript (including hypotheses) the same as in the Stage 2 manuscript? Are any changes transparently flagged?
• Did any prespecified data quality checks, positive controls, or tests of intervention fidelity succeed?
• Are any additional post hoc analyses justified, performed appropriately, and clearly distinguished from the preregistered analyses? Are the conclusions appropriately centered on the outcomes of the preregistered analyses?
• Are the overall conclusions based on the evidence?

3.3 Do’s and don’ts

Do suggest additional exploratory analyses at Stage 2, but don’t expect the recommender to necessarily require the authors to conduct them. Remember that authors are not obliged to conduct any unregistered analyses unless such tests are necessary to support conclusions that go beyond the preregistered analyses – and these must be conclusions that the authors (not reviewers or recommenders) wish to draw. This protection exists to prevent goal-post shifting and subtle forms of publication bias from creeping into the Stage 2 process.

If you find a flaw in the protocol at Stage 2 that was missed or unaddressed at Stage 1, do mention it but don’t expect the manuscript to be rejected on that basis. It is important to remember that the protocol cannot be relitigated at Stage 2 and recommenders cannot require authors to do extra studies. Barring very rare cases in which the authors, reviewers and recommenders collectively agree that a fatal flaw was missed, the most that will likely happen is that the authors will be asked to address potential design limitations in the Discussion. 

Do reflect on your potential biases to help make academia more equitable. Please read the Bias Reflection Guide (Foster et al. 2021) before deciding whether to accept a review invitation and before writing a review. There is also guidance on how to write a fair and respectful review by Foster et al. (2021).

 

4. Pathways for RRs at PCI RR

Manuscripts submitted to PCI RR follow different routes as shown below:

The manuscript undertakes Stage 1 and Stage 2 peer review at PCI RR, obtains Stage 2 recommendation by a recommender, and is...

1. maintained as a preprint as is.
2. published in our own Peer Community Journal.
3. published in any PCI RR-friendly journal.

4a. The manuscript undertakes Stage 1 peer review at PCI RR, obtains Stage 1 recommendation by a recommender, and undertakes further peer review for publication in any PCI RR-interested journal.
4b. The manuscript undertakes Stage 1 and Stage 2 peer review at PCI RR, obtains Stage 2 recommendation by a recommender, and undertakes further peer review for publication in any PCI RR-interested journal.

Reviewers should carefully note 4a and 4b here. Under these routes, the entity handling the manuscript will be switched from PCI RR to any PCI RR-interested journal in the course of its publication. During this process, the PCI RR-interested journal the authors have selected may contact the PCI RR reviewers to ask for continued review, to ensure consistency in the peer review process. This offer does not force the reviewers to make the switch, but their cooperation will help facilitate the publication of the manuscript more quickly and with higher quality.