PCI Registered Reports

4. Recommender training and certification

1. Introduction

This document provides guidance for recommenders in the handling of Stage 1 and Stage 2 Registered Reports for PCI RR.

The Peer Community In (PCI) initiative is a non-profit, non-commercial platform that publishes the peer-reviews of preprints. The overarching aim of this researcher-run organisation is to create specific communities of researchers reviewing and recommending, for free, unpublished preprints in their field. PCI communities have so far been created in evolutionary biology, ecology, paleontology, animal science, zoology, circuit neuroscience, metascience, and genomics, with more in the pipeline. Once the submissions are accepted (or, in PCI terms, “recommended”) following peer review, the revised manuscript is posted at the preprint server where the preprint is hosted, and the peer reviews and recommendation of the preprint are posted at the PCI website. Authors then have the option to also publish the preprint in a traditional journal.

PCI Registered Reports (PCI RR) is a new community dedicated to receiving, reviewing, and recommending Registered Reports (RRs) across the full spectrum of STEM, medicine, the social sciences and humanities. RRs are a form of empirical article in which study proposals are peer reviewed and pre-accepted before research is undertaken. By deciding which articles are published based on the question and proposed methods, RRs offer a remedy for a range of research biases, including publication bias and reporting bias.

Peer review for a RR takes place over two stages. At Stage 1, authors submit their research question(s), theory and hypotheses (where applicable), detailed methods and analysis plans, and any preliminary data as needed. Following detailed review and revision – usually according to specific criteria – proposals that are favourably assessed receive in principle acceptance (IPA), which commits PCI RR to recommending the final article regardless of the outcomes, provided the authors adhere to their approved protocol and interpret the results in line with the evidence. Following IPA, the approved protocol is then registered in a recognised repository, either publicly or under a temporary embargo. Then, after completing the research, the authors submit a Stage 2 manuscript that includes the approved protocol plus results and discussion, which may include clearly labelled post hoc analyses in addition to the preregistered outcomes. The reviewers from Stage 1 then return to assess the completed Stage 2 manuscript, focusing on compliance with protocol and whether the conclusions are justified by the evidence. Crucially, reviewers do not relitigate the theory, hypotheses or methods, thus preventing knowledge of the results from influencing recommendations. PCI editors (termed “recommenders”) are similarly forbidden from rejecting a manuscript on the basis of any new concerns about the methodology or rationale, or on the basis of the main outcomes.

Following the completion of peer review, authors of RRs that are recommended (i.e. accepted) have the option to publish their articles in the growing list of PCI RR-friendly journals that have committed to accepting PCI RR recommendations without further review. PCI RR thus aims to become the central hub and clearing house for Stage 1 and Stage 2 RRs across all areas of research.

This document provides specific advice for recommenders. Before continuing, please ensure that you have read Section 2 onwards of the PCI RR Policies and Procedures manual, the specific guidance for reviewers in Section 3 of the Guide to Reviewers, and this overview of the history, development, and future prospects of the RR format. The current document supplements the information provided in these materials.

2. Role of the recommender

PCI RR recommenders oversee the peer review process, deciding which RR submissions will ultimately achieve Stage 1 and Stage 2 acceptance. This section provides guidance on the roles and responsibilities of a recommender and provides advice for dealing with common challenges. To ensure a minimum understanding of the role and responsibilities, prospective PCI RR recommenders are required to pass a multiple-choice test and receive certification as a PCI RR recommender before handling their first assignment. Recommenders must also complete a declaration of interests indicating any other editorships or professional roles that could be perceived to conflict with their role at PCI RR.

A PCI RR recommender is analogous to an action editor at a peer reviewed journal. The recommender invites and assigns expert reviewers, assesses submissions and the reviews against the assessment criteria, and issues Stage 1 and Stage 2 recommendations (including writing a brief recommendation at the point of Stage 1 IPA; see Section 2.7 for details). Recommenders typically handle submissions either within or close to their specialism.

We ask our recommenders to reflect on their potential biases to help make academia more equitable. Please read the Bias Reflection Guide (Foster et al. 2021) before deciding whether to accept an invitation to be a recommender for a registered report, before writing a response to the authors, and before writing the recommendation text.

2.1 Triage assessment

Triage assessment refers to the phase of evaluation prior to in-depth peer review. When a submission is received by PCI RR it is assigned to the Managing Board, which includes researchers with significant journal editorial experience and/or expertise in RRs. At least one Managing Board member then assigns the submission to a specialist recommender, and together, the Managing Board member(s) and specialist recommender perform a pre-review triage assessment, judging the submission against the Stage 1 or 2 review criteria (as applicable). The Managing Board member may also handle a submission themselves, effectively serving both roles. There are four possible outcomes of this triage process:

Outright desk reject. Where a submission falls a long way short of at least one of the review criteria, it will be rejected outright by the Managing Board and specialist recommender with no opportunity for the authors to submit a revised version.

Desk rejection with the invitation to submit a revised manuscript. Where a submission falls significantly short of at least one of the review criteria but is deemed to be sufficiently promising, it will be desk rejected (without in-depth review) but with the invitation to submit a revised version that addresses specific points raised by the Managing Board and recommender.

Immediate in-depth review. Where a submission comes sufficiently close to meeting the review criteria, the submission will be sent for specialist review.

Immediate desk accept. Where the Managing Board and specialist recommender judge that a revised submission has sufficiently addressed the concerns of the reviewers, and thus satisfied the review criteria, it will be accepted without requiring further in-depth review.

The Managing Board is responsible for overseeing and confirming the decisions of the recommender. The Managing Board is ultimately responsible for all recommendations and, where necessary, can overrule a recommender's decision.

2.2 Review criteria

Submissions that are judged to be promising are sent for in-depth peer review, at both Stage 1 and Stage 2. Reviewers for PCI RR assess Stage 1 and Stage 2 RRs across all areas of quantitative and qualitative research according to the following criteria:

1A. The scientific validity of the research question(s). This criterion addresses the extent to which the research question is scientifically justifiable. Valid scientific questions usually stem from theory or mathematics, an intended or possible application, or an existing evidence base, and should be defined with sufficient precision as to be answerable through quantitative or qualitative research. They should also fall within established ethical norms. Note that subjective judgments about the importance, novelty or timeliness of a research question are NOT a relevant criterion at PCI RR and do not form part of this assessment.

1B. The logic, rationale, and plausibility of the proposed hypotheses, as applicable. This criterion addresses the coherence and credibility of any a priori hypotheses. The inclusion of hypotheses is not required– a Stage 1 RR can instead propose estimation or measurement of phenomena without expecting a specific observation or relationship between variables. However, where hypotheses are stated, they should be stated as precisely as possible and follow directly from the research question or theory. A Stage 1 RR should also propose a hypothesis that is sufficiently conceivable as to be worthy of investigation. The complete evaluation of any preliminary research (and data) in the Stage 1 submission is included within this criterion (see Section 3.7 of Policies and Procedures).

1C. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis or alternative sampling plans where applicable). This criterion assesses the validity of the study procedures and analyses, including the presence of critical design features (e.g. internal and external validity, blinding, randomisation, rules for data inclusion and exclusion, suitability of any included pilot data) and the appropriateness of the analysis plan. For designs involving inferential statistics and hypothesis-testing, this criterion includes the rigour of the proposed sampling plan, such as a statistical power analysis or Bayesian alternative, and, where applicable, the rationale for defining any statistical priors or the smallest effect size of interest. For programmatic RRs (see Section 3.15 of Policies and Procedures), this criterion captures the assessment of whether the separate study components are sufficiently robust and substantive to justify separate Stage 2 outputs.

1D. Whether the clarity and degree of methodological detail is sufficient to closely replicate the proposed study procedures and analysis pipeline and to prevent undisclosed flexibility in the procedures and analyses. This criterion assesses the extent to which the Stage 1 protocol contains sufficient detail to be reproducible and ensure protection against research bias, such as analytic overfitting or vague study procedures. In general, meeting this requirement will require the method section(s) of a Stage 1 protocol to be significantly longer and more detailed than in a regular manuscript, while also being clearly structured and accessible to readers. This criterion also covers the extent to which the protocol specifies precise and exhaustive links between the research question(s), hypotheses (where applicable), sampling plans (where applicable), analysis plans, and contingent interpretation given different outcomes. Authors are strongly encouraged to include a design summary table in their Stage 1 protocols that make these links clear (see Section 3.16 of Policies and Procedures for examples). Note that in some circumstances, authors may wish to propose a more general analysis plan involving a blinded analyst rather than a precise specification of data analyses. Such submissions are acceptable and will meet this criterion provided the blinded analysis procedure is specified in reproducible detail, and provided the blinding procedure itself is sufficiently robust.

1E. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the obtained results are able to test the stated hypotheses or answer the stated research question(s). This criterion addresses, where applicable, the extent to which the proposal pre-specifies data quality checks that will reveal whether the results are able to answer the research question(s). Such tests come in many forms and, depending on the design or subject area, are not always possible or appropriate. Where they are proposed they must be pre-specified and “outcome-neutral”, meaning that they are designed prior to knowledge of the results and are independent of the main study hypotheses. In experimental studies, such tests often consist of positive controls (also referred to in different fields as “manipulation checks” or tests of “intervention fidelity”). In observational studies they might include parallel measurements to replicate an established fact, thus confirming that the design or instrumentation is sufficiently sensitive to detect the existence of the phenomenon that forms the main focus of the research. This 1993 study which used the Galileo spacecraft to search for signs of life on Earth provides a signature example of a positive control.

Once a Stage 1 protocol receives IPA, the manuscript is formally registered in a repository (see Section 3.8 of Policies and Procedures). The Stage 2 manuscript reporting the completed research is then submitted and reviewers are asked to assess it according to the following criteria:

2A. Whether the data are able to test the authors’ proposed hypotheses (or answer the proposed research question) by passing the approved outcome-neutral criteria, such as absence of floor and ceiling effects or success of positive controls or other quality checks. This criterion addresses whether the data quality is sufficient to be able to test the stated hypotheses, according to the pre-specified conditions in 1E. Since not all protocols are able to pre-specify outcome-neutral tests, this assessment is not relevant to all forms of RRs. Where it is relevant and pre-specified, it is possible that the failure of a crucial outcome-neutral test could, in severe circumstances, lead to the rejection of a Stage 2 manuscript.

2B. Whether the introduction, rationale and stated hypotheses (where applicable) are the same as the approved Stage 1 submission. This criterion assesses whether the authors have remained consistent in their framing of the study at Stage 2. Aside from changes in tense (e.g. future tense to past tense), correction of typographic and grammatical errors, and correction of clear factual errors, the introduction, rationale and hypotheses of the Stage 2 submission must remain identical to those in the approved Stage 1 manuscript. To make any changes clear, authors are required to submit a tracked changes version of the manuscript at Stage 2.

2C. Whether the authors adhered precisely to the registered study procedures. This criterion assesses compliance with protocol. In cases where the preregistered protocol is altered after IPA due to unforeseen circumstances (e.g. change of equipment or unanticipated technical error), the authors must consult the PCI RR recommender immediately for advice, prior to the completion of data collection. Minor changes to the protocol may be permitted per recommender’s discretion. In such cases, IPA would be preserved and the deviation reported in the Stage 2 submission. If the authors wish to alter the study procedures more substantially following IPA but still wish to publish their article as a Registered Report then the manuscript must be withdrawn and resubmitted as a new Stage 1 submission. The outcome of all preregistered analyses must be reported in the manuscript, except in rare instances where a preregistered and approved analysis is subsequently shown to be logically flawed or unfounded. In such cases, the authors, reviewers, and recommender must agree that a collective error of judgment was made and that the analysis is inappropriate. In such cases the analysis would still be mentioned in the Stage 2 method but omitted with justification from the results. Additional unregistered analyses can also be included in a final manuscript (see 2D).

2D. Where applicable, whether any unregistered exploratory analyses are justified, methodologically sound, and informative. This criterion addresses the quality and value of any additional data analyses that are reported at Stage 2 but were not included in the registered Stage 1 submission. Such analyses are often highly valuable. For instance, a new analytic approach might become available between IPA and Stage 2 review, or a particularly interesting and unexpected finding may emerge. Alternatively, some unexpected characteristic of the data might suggest that the preregistered analyses, while bias-free, are not as sensitive as planned, and therefore a more sensitive post hoc analysis could be informative. Such analyses are admissible but must be clearly identified (e.g. through a separate heading in the Results for “Exploratory Analyses” or “Unregistered analyses”), justified, and appropriately caveated. Authors should also be careful not to base their conclusions entirely on the outcome of unregistered analyses.

2E. Whether the authors’ conclusions are justified given the evidence. This criterion addresses whether the claims drawn by the authors in their conclusions (including in the Discussion, Abstract, and anywhere else in the paper) are warranted by the data or evidence in hand. Note that PCI RR recommendation decisions will never be based on the perceived importance, novelty, or conclusiveness of the results.

Note: recommenders should check the submission's report survey to see whether the authors stated that they shared their data and/or code and, if so, confirm that these were actually shared.

2.3 Assigning reviewers

Recommenders are responsible for inviting and assigning reviewers. Recommenders should aim to assign 3 field-specialist reviewers (and never less than 2) to assess the first submission of a new Stage 1 manuscript. Assigning 3 field-specialist reviewers will generally require the recommender to invite 5-10 reviewers to achieve a sufficient number of agreements. Where possible, recommenders should avoid inviting reviewers with a clear conflict of interest such as a close relationship to the authors that includes recent co-publication, joint funding, or other obvious collaborations; or being at the same institution. As part of the submission process, authors can recommend up to five independent expert reviewers. Where a recommender intends to invite a reviewer suggested by the authors, it is the recommender’s responsibility to ensure that the suggested reviewer is independent and does not have a conflict of interest.

Where the recommender is also a field-specialist in the topic of the Stage 1 submission, and has no conflict of interest, the recommender can also serve as one of the 3 specialist reviewers, in addition to adjudicating on the submission as recommender. For programmatic submissions describing long-term, multi-arm projects that aim to produce multiple Stage 2 outputs (see Section 2.11), recommenders should anticipate the loss of reviewers between Stage 1 and Stage 2, and should recruit more reviewers accordingly at Stage 1 to increase the chances that at least 2 field-specialist reviewers will be able to assess the full project from start to completion.

Authors can also list up to five opposed reviewers, and need not provide any reason for opposing their involvement in the review process. It is the policy of PCI RR that no specifically-named opposed reviewer will be invited to assess a Stage 1 or Stage 2 submission. Note that recommenders are not obliged to adhere to more sweeping requests from authors, such as to exclude all reviewers from a particular university department, field or institute.

In addition to 3 field-specialist reviewers, recommenders can invite reviewers to assess any specific part of a submission, such as statistical analysis, methodology or computer code, even when the reviewer lacks expertise in the specific field or discipline of the investigation.

For the standard review workflow (as opposed to scheduled review; see Section 2.12), reviewers are requested to complete Stage 1 and Stage 2 assessments within 14 days of accepting a review request. Extensions to this period are negotiable and recommenders should exercise judgment in enabling a longer review period.

Reviewer continuity between Stage 1 and 2 is an important feature of the RR process. At peer-reviewed journals that have launched RRs, the reviewer retention rate between Stage 1 and 2 is generally high. Nevertheless, reviewer attrition is always possible and does occur. By inviting at least 3 reviewers from the start of Stage 1 in-depth review, the review process is better protected from the loss of one or two reviewers as the manuscript progresses.

Recommenders should take care to invite back the same reviewers who assessed the Stage 1 manuscript at Stage 2, keeping in mind that, during the time the authors took to complete the research (which can range from months to years), the contact details of the Stage 1 reviewers may have changed. In cases where a sufficient number of the original Stage 1 reviewers are unavailable to assess a Stage 2 submission, a recommender can invite a new Stage 2 reviewer provided the reviewer understands that the submission should be assessed in line with the Stage 2 criteria, regardless of any criticisms the reviewer may have concerning parts of the manuscript that were approved at Stage 1 (such as the study design, hypotheses, or rationale). Any such criticisms of the approved Stage 1 components can still be made by reviewers, and are appropriate to enter into the peer review record, but they must not influence the Stage 2 recommendation.

Recommenders are responsible for communicating to authors that any new criticisms of study design, hypotheses and rationale introduced during Stage 2 review must NOT prompt the authors to make major changes to the approved Stage 1 part of the manuscript. Only the most minor revision is possible to improve clarity or correct factual errors, but such criticisms could require consideration in the Discussion. Recommenders are free to consult with the Managing Board for advice on specific cases as they arise.

If you are struggling to find reviewers, try using Jane to find researchers conducting similar research and who do not have conflicts of interest with the authors of the submission you are handling. If you are still having difficulties, you can always email the current authors to ask for a list of potential reviewers with whom they do not have any conflicts of interest, and we can also help on the Managing Board (we recommend asking us for help before contacting authors). If you communicate with the authors at all, please only use email and make sure to cc contact@rr.peercommunityin.org so the Managing Board is aware of the interactions. We made a Reviewer Invitation Cheatsheet to help you keep track of the guidelines and organize your search. Feel free to download it and fill it as you handle a submission.

Reviewers are welcome to co-review registered reports with a junior academic in their group. To credit the co-reviewer, invite the junior academic as a separate reviewer. After they have accepted the invitation and the joint review has been written, they should paste the complete review at the junior academic's review space. In the senior academic's review space, they should state "Joint review: see review by X for details". If the reviews are anonymous, then state "Joint review: see review below for details" and note in the junior academic's review text that this is the joint review.

2.4 Review outcomes

Following peer review, recommenders are responsible for conveying the reviews to authors and issuing a range of judgments on submissions. There are three possible outcomes following each completed cycle of peer review:

Reject. Where expert reviews identify major flaws in a submission that are uncorrectable without designing a completely new programme of work then the submission should be rejected outright. Effective triage assessment (see Section 2.1) helps keep the rate of outright rejection following review relatively low. For example, at several major RR-adopting journals, only 10% of Stage 1 RRs that pass triage and enter in-depth review are rejected outright prior to IPA.

Invitation to revise. Where the reviews identify shortcomings that are, in principle, correctable to satisfy the review criteria, the authors should be invited to submit a revised manuscript, accompanied by a comprehensive response to reviewers.

Accept. Where a submission is judged to meet the Stage 1 or Stage 2 criteria, then it will be awarded either IPA or Stage 2 acceptance, respectively.

The Managing Board is responsible for overseeing and confirming the decisions of the recommender. The MB is ultimately responsible for all recommendations and, where necessary, can overrule a recommender's decision.

2.5 Exercising judgment

Reviews are provided as advice to the recommender rather than as binding judgments. While it can often be desirable for reviewers to reach a consensus in the assessment of Stage 1 or Stage 2 submissions, a consensus is not required to reach a recommendation, and recommenders should always be guided by their own judgment and that of the Managing Board. Moreover, in cases where only minor revisions are required, recommenders should, where possible, avoid overburdening reviewers with further review requests. While there will be reasonable exceptions to this rule, recommenders should in general assess minor revisions at desk before issuing a recommendation.

In cases where reviewers offer competing advice, it is the responsibility of the recommender to help steer the authors toward a resolution. It is therefore vital that recommenders read all reviews and the submission before rendering a recommendation, which is standard practice for recommenders on all submissions, not only those where the reviewers disagree.

2.6 Editing or discarding reviews

Reviewers and recommenders are expected to adhere to the PCI RR code of conduct, avoiding the use of abusive or discriminatory language in reviews and decision letters. Reviewers are informed that any parts of reviews that are judged to violate the code of conduct may be returned to the reviewer for editing before becoming part of the open review record (see Section 2.8). In cases where a review contains no information of value, the entire review may be discarded by the recommender. Depending on severity, recommenders may also feel it is appropriate to leave the review unedited but, if so, the recommender must clearly flag specific content in the review as unnecessary for the authors to address in revision. Recommenders should also feel free to contact reviewers and ask them to provide a higher quality review before submitting their decision.

2.7 Additional recommender assessments

In addition to weighing up the reviews and issuing decisions, in the event of a positive Stage 1 outcome, recommenders are responsible for performing a series of basic evaluations to determine which PCI RR-friendly journals will automatically offer IPA, and for writing a recommendation text.

Specifically, recommenders are asked to perform the following assessments of Stage 1 submissions:

The research discipline(s) into which the submission falls.
Whether the Stage 1 submission proposes quantitative, qualitative or mixed methods research.
What evidential standard is reached for each of the hypotheses, including assessment of power, Bayes factors, or other criterion as applicable.
What level of bias control is achieved (see Section 3.6 of Policies and Procedures). Authors are required to declare this as part of the submission process, and the recommender should check that the answer is accurate. The recommender’s designation overrules the authors’ self-declaration.
Whether the manuscript reports a meta-analysis, systematic review, umbrella review, or systematic map.
Whether the proposed research falls within typical ethical norms for the field.

The recommender should feel free to contact the authors directly (either via email while cc'ing PCI RR or at the online system by making a Revise decision without sending it back to the reviewers) to request any clarifications necessary to be able to make the above assessments, including ensuring the authors have responded to all comments in a point-by-point format and revised the manuscript accordingly.

At the point of Stage 1 IPA, and then again at Stage 2 acceptance, the recommender writes a recommendation. The recommendation is a short summary (of a few hundred words or less), similar to a News & Views piece, describing the context of the study and explaining why the article was accepted and is expected to make a valuable contribution. This recommendation and all the editorial correspondence (reviews, recommender's decision, authors’ replies) associated with the recommended article are then published by PCI RR no later than the point of Stage 2 acceptance (see Section 2.8). Please note that a recommendation text is prepared both at the point of Stage 1 IPA and additionally at Stage 2. Where authors opt for a private, embargoed submission at Stage 1, then as with the Stage 1 reviews, the Stage 1 recommendation text will remain unpublished until the point of Stage 2 acceptance.

Following Stage 1 IPA, PCI RR will formally register the protocol (see Section 3.8 of Policies and Procedures). At Stage 2, PCI RR will also assess compliance with the PCI RR TOP guidelines policy. This assessment will be undertaken by either a member of the Managing Board or a specially trained recommender.

2.8 Open review policy

PCI RR publishes all reviews of recommended manuscripts, with reviewers retaining the choice to sign their reviews or remain anonymous (unless one of the authors is a PCI RR Managing Board member, in which case all reviews must be signed. This is because Managing Board members have access to confidential information about all submissions, which poses an ethical problem if a reviewer wishes to remain anonymous from the authors). All Stage 1 and Stage 2 reviews, recommender decision letters, and the Stage 1 recommendation are published on the PCI RR platform no later than the point of Stage 2 acceptance. Authors can also opt to have the Stage 1 reviews, decision letters and recommendation published at the point of Stage 1 IPA. Reviews of rejected submissions are not published and we ask that authors and reviewers refrain from self-publishing (e.g. on the OSF) the content of the reviews, either during the review process or following rejection.

2.9 Managing appeals

Authors of rejected manuscripts may appeal the PCI RR recommendation by emailing the PCI RR Managing Board or the recommender who handled the manuscript. Appeals must be received within 30 days of PCI RR issuing a decision. The email must clearly state the case for why the decision should be changed. Appeals will only be considered if the authors a) identify factual errors made by the reviewers or recommender that had a major impact on the decision, or b) can provide a substantiated claim of unfair treatment and/or bias in the review process. Appeals for any other reason will be denied without further consideration. Appeals that meet the identified criteria will be discussed among a panel that includes the recommender and two members of the Managing Board. The panel will review the appeal and vote to uphold or reverse the original decision. Final decisions will be based on majority vote (i.e., decisions need not be unanimous). Decisions following appeal are final.

2.10 Summarised workflow

Figure 1. The submission and review workflow for Stage 1 and Stage 2 submissions.

2.11 Regular RRs and Programmatic RRs

Authors can submit using two different modes (regular vs. programmatic). A regular Stage 1 RR proposes a study or set of studies that are intended to eventually produce a single Stage 2 article. Regular RRs make up the majority of submissions.

To make RRs accessible to a wide variety of fields and working styles, PCI RR also offers authors the option to publish multiple Stage 2 articles from a single “programmatic” Stage 1 RR. The programmatic RR track can be useful for large or long-term projects where multiple studies contribute to a broader goal, and where the size or scope of the project is too great for a single published output. Authors of a programmatic Stage 1 RR should ensure that all the usual criteria for a RR are met, including detailed specification (where applicable) of theory, hypotheses, procedures, and analysis plans (see review criteria in Section 2.2). In addition, authors should ensure that the manuscript makes clear how the different elements of the programme – including different sets of research questions, hypotheses, and methods – fit together to form a cohesive package.

A Stage 1 programmatic RR must also prespecify which parts of the protocol will eventually produce separate Stage 2 articles, and as part of the Stage 1 review process, reviewers and recommenders will evaluate of the validity and substantive contribution of each component (as noted in criterion 1C; see Section 2.2). These prespecified boundaries are effectively treated as design elements; therefore, like any other design element, Stage 1 IPA will be contingent on authors adhering to the prespecified and approved article boundaries at Stage 2.

Following completion of each Stage 1 component, authors will submit a separate Stage 2 manuscript to PCI RR that includes the introduction, methods, results, and conclusions of each relevant component of the Stage 1 protocol. Because each Stage 2 submission derives from only a portion of the approved Stage 1 protocol, it is understandable that authors may need to make greater structural amendments to the introduction and methods sections of the Stage 2 manuscripts compared with a regular RR (i.e. including only the relevant content). Nevertheless, to ensure that Stage 2 criterion 2B is met (see Section 2.2), authors are reminded to make as few changes as necessary, being sure to keep the hypotheses, rationale and methodological descriptions of the relevant component in each Stage 2 manuscript as similar as possible to its corresponding component in the approved Stage 1 manuscript.

Because programmatic RRs are designed to support long-term projects, authors have the option to submit Stage 2 manuscripts corresponding to the relevant Stage 1 components as they go, receiving positive recommendations, and, if the authors wish, publishing the outputs serially in journals (including any PCI RR-friendly journals). In each case, the Stage 2 manuscript must include a section that (a) states that the current manuscript is one part of the larger protocol (listing the public URL to the protocol), (b) cites all previously published Stage 2 outputs arising from the same protocol, and (c) notes which, if any, Stage 2 components arising from the protocol are either awaiting completion or have either been rejected at Stage 2 by PCI RR or formally withdrawn by the authors.

Where authors elect for their approved programmatic RR to be registered at the point of IPA under a temporary private embargo, please note that the embargo will be released - and the entire protocol made public - at the point that the first Stage 2 manuscript of the planned series of outputs is received by PCI RR. This policy means that parts of the Stage 1 protocol will be made public that are not (yet) linked to a completed Stage 2 manuscript. If authors do not want this to happen then they should submit their programme of research as a series of separate regular RRs rather than as a single programmatic RR.

In case of a Withdrawn Registration, entire components that are planned as separate Stage 2 outputs can be withdrawn without affecting the IPA of the remaining components (see Section 3.12 of Policies and Procedures). However, just as part of a regular RR cannot be withdrawn (without withdrawing the entire RR), neither can part of a single component within a programmatic RR.

2.12 Standard review and scheduled review

In addition to submitting regular vs. programmatic proposals, PCI RR offers authors two tracks for undertaking Stage 1 peer review. Standard review (Figure 2A) is the typical track in which authors submit a complete and detailed Stage 1 manuscript for evaluation, in either the regular or programmatic formats. Authors can expect the first round of Stage 1 review via this track to take ~4-6 weeks. Scheduled review (Figure 2B) aims to significantly accelerate this process by performing key tasks in parallel. Rather than submitting a full manuscript, authors initially submit a one-page, template-based RR “snapshot” which undergoes triage by a recommender. If deemed suitable (which could require revision of the RR snapshot), the recommender then sends the RR snapshot to potential reviewers and arranges for the review process to take place during a narrow range of dates (1 week) at least 6 weeks in the future, with the submission date nominated by the authors. During the intervening time the authors then prepare the full Stage 1 manuscript. Although this process can only feasibly expedite the first round of Stage 1 review (and not the re-review of a revised Stage 1 submission), the overall time-saving is substantial because the first round of Stage 1 assessment is usually the most onerous. Authors intending to submit to the scheduled review track must do so using the PCI RR 1-page snapshot template available here. Submissions that do not use this template will not be considered. The Standard and Scheduled review tracks are available for both Regular and Programmatic RR submissions.

Figure 2. The submission timelines for (A) standard review and (B) scheduled review. Scheduled review, where feasible for authors, can significantly reduce the duration of the first round of Stage 1 review. Figure adapted from Chambers & Tzavella (2020).

2.13 Instructions for using the PCI review platform

Please see the instructions at Help > How to...?

3. Common challenges

This section offers advice on how to handle a range of common challenges experienced by recommenders at PCI RR.

3.1 Maximising the effectiveness of triage

Careful triage is vital to ensure that only submissions reaching a sufficient standard are sent for in-depth Stage 1 review. Where there are obvious flaws or oversights in meeting the Stage 1 review criteria, recommenders should reject submissions without in-depth review, and if the submissions are sufficiently promising then the recommender has the option to desk-reject while inviting a resubmission. Early in the life of RRs, where many authors are unfamiliar with the requirements of the format, it is not uncommon to triage many otherwise promising submissions with the offer to resubmit. As described in the workflow above, recommenders and Managing Board members will work together to perform effective triage.

3.2 Managing Stage 1 review

The review process for RRs can have a substantially different tone to peer review of conventional articles. Rather than focusing on reasons to reject manuscripts, reviewers at Stage 1 often make detailed suggestions for improving rationale and methodology, and crucially, at a point in time where such changes can still be made. This process often defies the conventional (gatekeeping) norm of peer review; however, with greater study input from reviewers come additional challenges as recommenders. It is not uncommon for reviewers’ suggestions at Stage 1 to be mutually exclusive, which can require careful input to help steer authors toward clarity. Recommenders (and especially those new to assessing RRs) are advised to consult closely with the Managing Board in handling these requirements.

3.3 Adhering to the review criteria

The assessment of RRs at PCI RR is rule-based, so it is vital that recommenders keep their decisions tied closely to the review criteria. At Stage 2, once the study is completed, impressions about the importance or novelty of results should never influence recommendations. It is not uncommon for reviewers to make comments on manuscripts that fall outside the review criteria -- for instance, commenting on whether they find results subjectively interesting. Recommenders should not remove such comments from the review record, but they are responsible for making clear to authors that criticisms or recommendations from reviewers that fall outside the review criteria can be ignored without influencing the final recommendation.

3.4 Prespecifying quality checks

One of the most substantial and interesting challenges in assessing Stage 1 RRs is deciding in advance of the results the quality checks that must be satisfied in order to conclude that the experiment was conducted to a sufficient standard (see Stage 1 Criterion 1E and Stage 2 Criterion 2A). Such tests must be independent of the research questions or hypotheses and might include positive control experiments, tests of intervention fidelity, reality checks, attainment of established benchmarks, or any other checks on data quality. Appropriate control conditions are important in all quantitative sciences, but we have noticed that this process is especially rigorous and intensive for Stage 1 RRs. This may be because, in some research fields, there can be a tendency during conventional peer review for the results to distract reviewers and editors from the validity of the methods, leading to a conflation of methodological quality with the outcomes of hypothesis testing. For example, the methods underpinning a study may be particularly scrutinised when the results fail to support the hypothesis or run counter to expectations.

Failure of outcome-neutral tests (Stage 2 Criterion 2A) is a justifiable cause for rejection at Stage 2. Whether, in practice, this leads to rejection is a judgment for the reviewers and recommenders based on the severity of the problem. In the most extreme case, where the outcome-neutral tests fail so severely that no part of the research question can be answered, and the failure of the outcome-neutral test itself is almost certainly due to poor methods (as opposed to, e.g. an assumed "reality check" not measuring reality) then the Stage 2 manuscript would be rejected.

In other cases, however, the impact of the failure of an outcome-neutral test may not be a fatal problem, either because there is evidence from other parts of the data confirming that the study was conducted to a high standard (i.e. a "backup" or “failsafe” test), or because the failure of the outcome-neutral test -- while absolute -- did not affect the ability of the study to answer all of the research questions. The failure of an outcome-neutral test could also be an important finding in its own right: where there is other evidence to corroborate data quality, it could imply that the test itself (e.g. a positive control or other “reality check”) is not as reliable as commonly assumed, which could be an important methodological message.

3.5 Goalpost shifting at Stage 2

At Stage 2, reviewers occasionally attempt to relitigate the rationale or methods of a study when the outcome of the Stage 1 protocol disagrees with their expectations. For example, in one paper handled by a Managing Board member at a journal, a reviewer at Stage 2 who had previously approved the Stage 1 protocol recommended that the Stage 2 manuscript be rejected because of a flaw in the methods that neither the reviewers, authors, nor editors identified during Stage 1 review. Barring rare cases where all parties (including the authors) agree that a major, serious flaw was missed, the approved methodology cannot be relitigated at Stage 2. In this particular case, the authors disagreed with the reviewer and instead mentioned and rebutted this limitation in the Discussion.

In another (more subtle) example, a reviewer at Stage 2 asked that the authors conduct a complex series of unregistered post hoc analyses because the reviewer did not find the outcomes of the preregistered analyses “convincing”. For conventional (non-RR) manuscripts, reviewers typically have great power to compel authors to conduct extra analyses and an editor would usually require authors to comply unless there was a very good reason not to. However, for RRs, this power is focused in a different way: reviewers instead have the opportunity to help shape the design and analysis at Stage 1, before the results are known. The influence of reviewers in requiring new analyses at Stage 2 is limited to overcome common biases against negative results or other outcomes that reviewers can sometimes find undesirable. Therefore, while reviewers are welcome to suggest additional post hoc analyses, and authors of Stage 2 RRs are welcome to include them, such analyses are not required unless they are necessary to satisfy one or more of the Stage 2 review criteria. For instance, a post hoc analysis may be necessary if the recommender judges that without such an analysis, the authors’ conclusions are not justified by the evidence (Criterion 2E).

3.6 Ethical concerns

Recommenders may occasionally have ethical concerns about a Stage 1 manuscript. Where ethical approval is relevant to the research field, it is generally expected that authors will have acquired all necessary ethical approvals prior to Stage 1 submission. However, such approvals will not necessarily be in place for all submissions because under certain circumstances, authors have the option to submit a Stage 1 manuscript prior to ethics approval (see Section 3.4 in Policies and Procedures). In addition, recommenders should be vigilant to the possibility that approval by a research ethics committee or institutional review board does not guarantee that the proposed research falls within established ethical norms, due to variations in ethical standards between departments, institutions and countries. For these reasons, it is important that recommenders assess the ethical status of the research as part of the overall evaluation (captured within Stage 1 Criterion 1A). Where a submission is judged by the reviewers, recommender or Managing Board to fail to meet necessary ethical standards, it should be rejected. Recommenders should consult closely with the Managing Board about any areas of concern.

3.7 Permissible changes between Stage 1 and Stage 2

Authors are informed that any deviation from the approved study procedures following Stage 1 acceptance, regardless of how minor it may seem to the authors, could lead to rejection of the manuscript at Stage 2 (criterion 2C). In cases where the preregistered protocol is altered after IPA due to unforeseen circumstances (e.g. change of equipment or unanticipated technical error), the authors must consult the recommender immediately for advice, and prior to the completion of data collection. Changes to the protocol that are essential for the research and do not significantly increase risk of introducing bias (e.g., significant changes to the study design table) into the analyses/inferences should be swiftly approved by the recommender, and minor cases can be approved in consultation with the Managing Board without being returned to the reviewers. In such cases, IPA would be preserved and the deviation reported in the Stage 2 submission. In considering such changes, recommenders should ask themselves "does this deviation increase the chances of selective reporting of outcomes?" Note that this is not necessarily deliberately on the part of the authors. The introduction of bias can be anything which might reintroduce researcher degrees of freedom or create the opportunity for the findings to be tilted in a particular direction by the deviation. So, for example, a change in exclusion criteria after seeing the data carries a high risk of bias, whereas changing a piece of equipment during data acquisition, without any knowledge of the data, has very low risk of bias.

Where authors intend to alter the study procedures more substantially following IPA, but still wish to gain a PCI RR recommendation, then the manuscript must either be readmitted to Stage 1 peer review or withdrawn and resubmitted as a new Stage 1 submission. Note that registered analyses must be undertaken, but additional unregistered analyses can also be included in the final manuscript (see Section 3.9 in Policies and Procedures).

In general, the parts of the Stage 1 manuscript that were approved at the point of IPA (including the Introduction and Methods) should remain the same at Stage 2, with the following exceptions:

Any description of the rationale or proposed methodology that was written in future tense within the Stage 1 manuscript should be changed to past tense.
The Abstract of the Stage 2 submission should be updated to include an overview of the results and conclusions.
The title of the manuscript can be changed to reflect the results and conclusion.
Any typographical or outright factual errors should be corrected.
Authors can be added or removed without needing to seek the approval of the PCI RR recommender.
Where there were any deviations from the accepted methodology, the Methods section of the Stage 2 manuscript must be altered to state the method that was implemented, accompanied by a footnote to explain the nature of, and reason for, the deviation from the planned methodology.

Note that any relevant literature that appeared following the date of IPA should be covered in the Discussion, not the Introduction. No part of the research question, rationale, or hypotheses (as applicable) can be altered at Stage 2.

All textual changes to the parts of the Stage 2 manuscript that were approved at Stage 1 must without exception be highlighted as tracked changes in the Stage 2 manuscript. Authors can meet this requirement either by highlighting the tracked changes in the published Stage 2 preprint or, where they prefer to publish a “clean” Stage 2 preprint, by linking to a duplicate version of the manuscript (e.g. in a trusted repository) that highlights the tracked changes.

3.8 Dealing with late or non-responding reviewers

The review process for RRs usually proceeds smoothly, but complications in the review process do occasionally delay submissions. The most extreme delays arise where a reviewer agrees to assess a submission but then never submits the review and becomes non-contactable. Such cases typically require the recommender to find an alternative reviewer, effectively restarting the process and delaying the decision by several weeks. As insurance against this eventuality, recommenders are advised to assign four reviewers for each Stage 1 submission -- that way, a decision could be issued if one or even two of the reviews is not forthcoming. Recommenders should also be flexible about allowing short extensions to review deadlines. Where possible, recommenders should always invite the same reviewers back at Stage 2.

3.9 Seeking statistical advice

Many Stage 1 RR submissions benefit from review of the proposed statistical methods, either by field specialists with statistical expertise or by statisticians. Sourcing such reviews can often be challenging. In general, recommenders are advised to seek assessment of statistical methods (where applicable) as part of the Stage 1 review process, particularly where the recommender themself lacks specialist expertise in the proposed techniques. The Managing Board can help provide guidance and suggestions for recruiting statistical reviewers.

3.10 Dealing with queries about data/code/materials availability

Authors will sometimes contact recommenders with questions about the requirements concerning making their data publicly available. In response, recommenders can direct authors to the PCI RR Transparency and Openness Promotion (TOP) guidelines policy. Broadly, this policy requires that authors make all study data, digital materials and code publicly available unless there are specific ethical or legal restrictions that prevent the authors from doing so. In such cases, authors must instead specify in the manuscript the process (if any) by which requesters can obtain the data, materials and code, including any specific conditions they must meet. Recommenders should also contact the Managing Board with any queries about specific cases.

4. Recommender training and certification

All new PCI RR recommenders are expected to become familiar with the policies and procedures of the platform, including the Guide to Recommenders, Section 2 onwards of the PCI RR Policies and Procedures manual, the specific guidance for reviewers in Section 3 of the Guide to Reviewers, and this overview of the history, development, and future prospects of the RR format.

Prospective recommenders must also pass the PCI RR entrance test in order to achieve formal certification as a recommender. This test is designed to assess basic knowledge of the RR format, the core policies of PCI RR, and best approaches for tackling challenging scenarios. All information that prospective recommenders need to pass this test is contained in the current guidance and the above links, and the test can be taken as many times as necessary to achieve a passing grade (scoring at least 63 out of 66 points; 95% correct).

or Register
Submit a report