Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness
BMJ 2024; 384 doi: https://doi.org/10.1136/bmj-2023-076335 (Published 06 February 2024) Cite this as: BMJ 2024;384:e076335- Chantelle Garritty, adjunct professor and manager12,
- Candyce Hamel, adjunct professor and senior epidemiologist13,
- Marialena Trivella, research associate, senior methodologist, and assistant professor456,
- Gerald Gartlehner, professor and senior health research analyst47,
- Barbara Nussbaumer-Streit, co-director4,
- Declan Devane, professor8,
- Chris Kamel, director9,
- Ursula Griebler, senior research associate4,
- Valerie J King, professor 10
- on behalf of the Cochrane Rapid Reviews Methods Group
- 1School of Epidemiology and Public Health, Faculty of Medicine, University of Ottawa, Ottawa, ON, K1G 5Z3, Canada
- 2Global Health and Guidelines Division, Public Health Agency of Canada, Ottawa, ON, Canada
- 3Canadian Association of Radiologists, Ottawa, ON, Canada
- 4Cochrane Austria, Department for Evidence-based Medicine and Evaluation, University for Continuing Education Krems, Krems, Austria
- 5Department of Cardiovascular Medicine, John Radcliffe Hospital, Oxford, UK
- 6Department of Population Health, London School of Hygiene and Tropical Medicine, London, UK
- 7Research Triangle Institute (RTI) International, Research Triangle Park, NC, USA
- 8Cochrane Ireland and Evidence Synthesis Ireland, School of Nursing and Midwifery, University of Galway, Galway, Ireland
- 9Canadian Agency for Drugs and Technologies in Health (CADTH), Ottawa, ON, Canada
- 10Center for Evidence-based Policy, Department of Family Medicine, Oregon Health and Science University (OHSU), Portland, OR, USA
- Correspondence to: C Garritty chantelle.garritty{at}uottawa.ca (or @cgarritty on Twitter)
- Accepted 2 January 2023
Summary points
This article updates the Cochrane rapid review methods guidance published in 2020 to support rapid reviews of effectiveness in the context of urgent and high priority health questions
The updated guidance incorporates new knowledge and feedback from users of the interim guidance, with input from a broader group of methodologists specialising in rapid reviews, and it is both evidence informed and user informed and widely applicable to anyone conducting a rapid review
The update clarifies key concepts underpinning rapid reviews, provides a refined list of 24 recommendations, offers supporting examples, and provides best practice considerations and practical tips for teams to increase efficiencies
The Cochrane Rapid Reviews Methods Group will continue to promote research, monitor published literature, and update recommendations to facilitate timely, evidence based decision making in healthcare
Introduction
In recent years, the Cochrane Collaboration, a global leader in producing high quality systematic reviews and methodological guidance, has taken steps to support rapid reviews. In 2020, the Cochrane Rapid Review Methods Group published interim guidance on the conduct of rapid reviews of effectiveness produced within Cochrane and beyond.1 Guidance was developed to focus on rapid reviews of health interventions to address urgent and high priority questions. The original guidance was informed by a suite of research, including a scoping review of the underlying evidence,2 a proposed definition of a rapid review,3 primary meta-epidemiological studies,45 and a survey of Cochrane community members who prioritised the appropriateness of rapid review methods across stages during the conduct of reviews.1 The original guidance offered 26 recommendations, with accompanying rationales on the steps and considerations for accelerating each part of the review process. Completion of the guidance coincided with the onset of the covid-19 pandemic, which was the catalyst for the increased number of Cochrane and non-Cochrane rapid reviews conducted and published since 2020.6 The pandemic showed the importance of expedited systematically produced evidence synthesis to address many clinical, public health, and health policy and systems related questions.
In this article, we present an update on the interim Cochrane rapid review methods guidance, integrating new knowledge on the conduct of the reviews and feedback from users of the interim guidance. We clarify key concepts underpinning rapid reviews, include a refined list of recommendations, and provide some accompanying examples, supporting information, and links to additional resources to guide methods for those interested in the methodology of rapid reviews. The decision to update the interim guidance on Cochrane rapid review methods at this juncture was driven by several critical factors. The original guidance was expedited for release during the onset of the covid-19 pandemic, ensuring timely access for Cochrane’s network of reviewers and the broader research community grappling with urgent needs for evidence synthesis. Consequently, refinement of the recommendations was temporarily halted. As stated in our interim guidance, the increase in published rapid reviews, combined with our commitment to ongoing quality improvement efforts, underscored the need to align the guidance now with the evolving landscape of rapid review methodology after the pandemic. In addition, insights gleaned from an evaluation assessing the guidance’s real world use have pinpointed areas requiring enhancement, particularly in terms of user friendliness and practicality for authors of rapid reviews with varying levels of experience.7
Methods
This update of the Cochrane rapid review methods guidance builds upon previously published interim guidance.1 We have also integrated findings from a formal evaluation conducted to assess authors’ adherence to and understanding of the interim Cochrane guidance and the guidance’s comprehensibility, usability, and usefulness. A complete description of the evaluation is available elsewhere.7 This evaluation included the text analysis of 128 rapid reviews (17 Cochrane and 111 non-Cochrane) and 20 in-depth qualitative interviews. Main findings suggested that many authors did not follow certain recommendations, such as the stepwise approach to study inclusion or peer review of search strategies. However, some recommendations, such as dual independent screening of abstracts or full texts, were exceeded. Common reasons for not adhering to the guidance included time constraints, unclear recommended approaches, or inapplicability to specific rapid reviews. Overall, the guidance was considered user friendly, but it was perceived as challenging to apply without experience of conducting systematic reviews. On this basis, we identified the need to change the wording of some recommendations and to clarify others, further keeping in mind that the guidance might be used by investigators with varied experience of systematic review and rapid review methods.7 As such, this updated guidance clarifies some defining features of rapid reviews and additional aspects to consider for Cochrane rapid reviews.
We also scanned the literature for publications related to rapid review methodology published since the initial scoping review that underpinned our interim guidance in 2020.2 To identify potentially relevant studies, we used the option for similar articles in PubMed for every article included in the initial scoping review. We also searched PubMed using a general keyword search based on our original scoping review search strategy. We limited searches from February 2019 (the previous search) to August 2022. A total of 841 citations were then screened and 87 articles assessed for relevancy in accordance with an initial scoping review of rapid review methods2 (see subsection w1 in the supplementary file for list of studies). Although none met the original eligibility criteria, five studies provided further insights to the search recommendations.89101112
Furthermore, we assembled a broader collaborative of rapid review methodologists beyond the coauthor group, who provided specific input on proposed modifications to the guidance. Led by the Cochrane Rapid Review Methods Group, this group of experts, including a patient and public partner, has produced a multipart series to further guide methods decisions in each step of the process for rapid reviews.13141516 The series expands on and explains in further detail what this updated methods guidance recommends.
Results
Cochrane methods guidance considerations
Cochrane defines a systematic review as using systematic and explicit methods to identify, select, critically appraise, extract, and analyse data from all relevant research.17 Rapid reviews also use systematic and explicit methods to appraise, extract, and analyse data. By comparison, however, specific components of the systematic review process are either restricted or omitted or the scope is narrowed for rapid reviews, to provide an evidence synthesis product more quickly. Therefore, as rapid reviews might not include all relevant studies, they may be less comprehensive. As a result, the appropriateness of undertaking a rapid review needs to be considered carefully, with a strong justification provided for using this approach instead of a systematic review, including the rationale for using restricted methods.18 General distinctions between systematic reviews and rapid reviews have been published previously.1920
In updating the interim guidance, it is also important to emphasise that Cochrane rapid reviews should be driven by the need for timely evidence for decision making purposes, including addressing urgent and emergent health issues and questions deemed high priority. Additionally, when conducting a rapid review, multiple methodological paths can be taken, and no “one size fits all” approaches can be applied. Rapid reviews are tailored and therefore can vary in scope and methodology depending on time and resources available, restricted methods used (or a combination of restrictions), and the types and levels of evidence included. Although this guidance is intended as best practice advice to determine the methodological way forward when conducting a Cochrane rapid review, not every restricted method that is recommended needs to be implemented. Teams may use stricter methods, if time and resources allow, and still call it a rapid review, as discussed below.
For this update we have clarified key defining features of rapid reviews relative to systematic reviews:
Restricted methods—To accelerate the review process, it is typical for rapid reviews to introduce methodological restrictions (shortcuts or abbreviated methods). Therefore, the reviews should be well reported, highlighting the restricted methods taken to accelerate the review, the potential biases these methods may have introduced, and other limitations of the evidence base.
Dimension of time—Despite the term “rapid” being used for these reviews, time is not the sole defining feature—although rapid reviews should be conducted in a short timeline. Timelines across the reviews vary, however, depending on several factors, including the complexity of the topic or the urgency of the decision making to meet timelines, which are often short out of necessity. Importantly, the notion that rapid reviews are simply systematic reviews done faster is misleading. A rapidly conducted systematic review would still be a systematic review if authors followed stringent systematic review methodology, such as methods proposed by Cochrane.17 Consistent with our interim guidance, we continue to endorse that Cochrane rapid reviews should take no longer than six months.
Knowledge user involvement—As decision makers typically commission rapid reviews to address specific and pressing health questions, it is common for them to be involved in the process, given the urgent nature of these inquiries. However, other important knowledge users (eg, patient and public partners, healthcare providers, and policy makers) may also be involved in shaping the rapid review. In collaboration with funders and other knowledge users, the scope of the review should be narrowed down to answer a focused question.
Cochrane rapid reviews
Definition
We recommend the following definition for a Cochrane rapid review: “A rapid review is a type of evidence synthesis that brings together and summarises information from different research studies to produce evidence for people such as the public, healthcare providers, researchers, policy makers, and funders in a systematic, resource efficient manner. This is done by speeding up the ways we plan, do, and/or share the results of conventional structured (systematic) reviews, by simplifying or omitting a variety of methods that should be clearly defined by the authors.” This definition builds upon the original definition endorsed in the interim guidance,16 and it was modified following the input of patient and public partners as part of a recent collaborative Priority Setting Partnership on rapid reviews.3
Additional aspects also need to be considered when applying this guidance:
Guidance for rapid reviews of effectiveness—Importantly, this guidance was developed within the context of Cochrane and focuses on rapid reviews concerning the effectiveness of health interventions, albeit it may be used for non-Cochrane reviews of effectiveness. This guidance has not yet been adapted beyond interventions of effectiveness to other question types relevant to rapid reviews (eg, rapid reviews of diagnostic test accuracy or screening, or rapid qualitative evidence synthesis) since specific review question types may pose unique methodological challenges.21 Examples of other review question types and methodological considerations for rapid reviews can be found at https://methods.cochrane.org/rapidreviews/rr-methods-guidance/additional-methodological-considerations.
Experience of systematic reviews—Rapid review teams should include expertise from information specialists and have access to clinical experts and individuals with expert knowledge of systematic review methods, ideally available throughout the rapid review process to guide and advise on the strengths and limitations of abbreviated methods to minimise compromising validity.
Access to electronic databases and software—Rapid review teams must have adequate resources before embarking on the review. Required resources include access to electronic databases most relevant to the topic of rapid reviews (eg, Medline, CENTRAL (Cochrane Central Register of Controlled Trials), CINAHL (Cumulative Index to Nursing and Allied Health Literature), Embase, PsycInfo), reference management software (eg, Endnote, Zotero, RefWorks), screening software (eg, Rayyan, Covidence, DistillerSR), a virtual meeting platform (eg, Google meet, Zoom, MS Teams), and possibly other messaging applications used to facilitate timely communications and project management across the rapid review team (eg, Slack). We recommend that teams use live document platforms, as these enable real time collaboration, version control, and efficient information sharing, ultimately streamlining the review process. Some of these resources are freely available, whereas others require access through a library or academic institution or a paid user licence.
Cochrane rapid review recommendations
The Cochrane Rapid Review Methods Group has issued an updated list of 24 recommendations for Cochrane rapid reviews as outlined in table 1, with rationales and examples to support the recommendations provided in subsection w2 of the supplementary file. Rapid review teams may apply all or some of the methodological restrictions proposed in the recommendations, depending on the topic, timeframe, and resources.
Topic refinement—setting the research question
Recommendation 1: Involve knowledge users to set and refine the review question, eligibility criteria, and outcomes of interest, with consultation at various stages of the review
Knowledge users are individuals or groups responsible for, or affected by, health and healthcare related decisions that rapid reviews can inform.2223 The term knowledge user includes but is not limited to healthcare providers and their professional associations, policy makers, patients, caregivers, patient groups, government agencies, and the public.24 By their very nature, rapid reviews often necessitate close and intensive collaboration between researchers and decision makers, including the organisations that commissioned them.25 However, the involvement of key knowledge users (eg, patient and public partners, healthcare providers, and policy makers) is often limited, omitted, or not reported.26 Although meaningful involvement requires time, resources, and advanced planning given the shortened timelines of rapid reviews, involving key knowledge users when possible can enhance the relevance and applicability of the review and should be encouraged.2728 The STARR (SelecTing Approaches for Rapid Reviews) tool aids authors in planning approaches to rapid reviews and obtaining structured input from users through targeted questions.29 A recent publication provides further insight on ways to involve knowledge users in the co-development of rapid reviews (eg, planning, performance, and knowledge translation of the reviews).13 Authors of Cochrane rapid reviews should be aware of Cochrane’s new framework for consumer (patient, carer, and public) engagement and involvement30 and should leverage the Cochrane Consumer Network to identify potential patients, carers, and the public as knowledge users.31
Recommendation 2: Develop a protocol that includes the review questions, description of the population, interventions, comparators, outcomes, and methods of conducting the review
As you would for a systematic review, it is important to develop a protocol for the rapid review that supports the principles of transparency and reproducibility. Protocols should include the review question or questions, using a question framework such as PICOs (population-intervention-comparator-outcomes), and detail the eligibility criteria and methods that will be used for searches, study selection, data extraction, risk of bias, and synthesis. The authors of Cochrane rapid reviews must submit a completed protocol to the Cochrane central editorial service (editorial-service@cochrane.org), which will undergo editorial and methodological checks. Cochrane has a streamlined workflow and protocol template to accommodate rapid reviews across priority topics (see https://covidreviews.cochrane.org/resources). Non-Cochrane rapid reviews may use this template as a guide. If this template is not used, protocols should be reported to the extent possible following the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P).32 If the time for peer review is limited, protocols should, at a minimum, be reviewed internally for consistency and accuracy. For transparency, authors should ensure public availability of their protocol through open access platforms (eg, PROSPERO, Open Science Framework).
Recommendation 3: Clearly define the eligibility criteria, including any restrictions or limits
To ensure rapid reviews are manageable and timely, various restrictions can be applied to eligibility criteria (eg, PICOs, timing, settings, date) (see recommendations 3.1 to 3.6). Such restrictions must be considered through discussions with the rapid review team and knowledge users.
Recommendation 3.1: Limit the number of interventions and comparators—So the review is focused and manageable, the number of interventions and comparators should be limited. Any such restrictions should not impact the decision making ability of knowledge users.
Recommendation 3.2: Limit the number of outcomes, focusing on those most important for decision making—When developing the protocol, focus on outcomes that are relevant for knowledge users and important for decision making. The recommendation is to rate outcomes by importance, with seven as a maximum33 (although fewer may be better for a rapid review, depending on available time and resources). The list of outcomes may be restricted at any point throughout the conduct of the rapid review in consultation with knowledge users. Ideally, a core set of outcome measures would be available to inform decisions on outcome selection (eg, from the Core Outcome Measures in Effectiveness Trials (COMET) Initiative34). Outcomes will depend on the needs of knowledge users and should include outcomes for both potential benefits and potential harms.
Recommendation 3.3: Consider restricting the search date of the evidence base, with clinical or methodological justification provided—Although setting a date restriction is a pragmatic decision in some cases, this needs to be carefully considered for each topic. When conducting a rapid review, authors should consider the trade-offs (ie, the potential for less accuracy with the loss of studies versus workload) by different limits of a search date.1135 To make the right decision, authors should also assess each case individually because of variation in topics. When it is deemed important to avoid date restrictions, other approaches described in this guidance may be needed to deal with the potentially high number of search results. Regardless, rapid review teams should provide an appropriate justification if instituting date restrictions (see subsection w2 in the supplementary file). Besides, close communication with information specialists and clinical experts will help to set informed date limits. Other knowledge users may also provide insight on whether date limits are appropriate for each topic.
Recommendation 3.4: Limit the setting, with clinical or methodological justification provided—Limitations on the setting may be related to geographical areas or regions (eg, studies in the UK, low income and middle income countries, rural settings) and where the study is conducted, such as in the community or in a hospital. Any restrictions in the setting should be justified and relevant (see subsection w2 in the supplementary file) and should not impact the decision making ability of knowledge users.
Recommendation 3.5: Limit the publication language to English at study selection, with other languages added when relevant—Language restrictions during the initial search process are discouraged; we advise applying these restrictions during the study selection phase. Research suggests that excluding non-English publications from systematic reviews on clinical interventions has a minimal effect on overall conclusions and can be a viable methodological shortcut for rapid reviews.4 We do not, however, recommend restricting to English only publications if previous knowledge suggests that studies relevant to the chosen rapid review topic may be published in languages other than English. Suppose, for example, rapid reviews are related to complementary and alternative medicine treatments. In that case, relevant studies in languages other than English would be expected, and studies published in these languages should be included in the rapid review. Ensure that any language limits are justified and clearly explained (see subsection w2 in the supplementary file).
Recommendation 3.6: Prioritise the inclusion of high quality study designs relevant to the review question or objective—Determine what levels of evidence to include and ensure the decision for this approach is well explained (see subsection w2 in the supplementary file). For example, if one or more well conducted systematic reviews has been done that address the question of the rapid review, including and updating these systematic reviews may be sufficient. Randomised controlled trials should be considered for effectiveness questions if no well conducted systematic reviews exist or do not address the PICOs sufficiently. In the absence of well conducted randomised controlled trials for effectiveness questions, or if time permits, or both, non-randomised studies may be considered. This implies that researchers should understand study designs and their characteristics, relationship to the review question or objective, and potential for bias. Importantly, systematic reviews and randomised controlled trials may not be available for new healthcare interventions—for example, no randomised controlled trials were available on covid-19 during the early era of the pandemic but emerged as the pandemic evolved.36
Searching
Recommendation 4: Involve an information specialist to develop the search strategy, and consider search methods, resources, and search limits
Planning a search is integral to the overall preparation of a rapid review and should form part of protocol development. At minimum an information specialist such as a librarian should be consulted to select information sources (eg, bibliographic databases, type of supplementary searches) and provide feedback on the primary search strategy.37 Information specialists can assist in selecting appropriate search methods and resources, defining search limits, designing and executing search strategies, and reporting the search methods. A preliminary or scoping search should be performed during the topic refinement stage and may help inform eligibility criteria. If the rapid review is being done by updating an existing systematic review, information specialists may use the original search strategy and adapt as necessary. Overall, the search process for a rapid review follows the same steps as for a systematic review; therefore, rapid review teams must be familiar with the general standards of systematic searching and reporting of searches.38
Recommendation 5: Select a small number (but at least two) bibliographic databases that are likely to contain relevant literature
Rapid review teams should prioritise the most relevant information sources for the topic, the type of evidence required, and access to sources. We recommend a conventional approach of selecting at least two electronic bibliographic databases, dependent on the study type and topic most likely to retrieve relevant literature based on recent evidence.123539 In addition to electronic bibliographic databases, grey literature sources and targeted supplementary search methods may be used (see recommendation 7).
For rapid reviews focused on randomised controlled trials only, use a combination of two of the following databases (if you have access): Medline, CENTRAL, and Embase.39
Additional considerations:
▪ For Cochrane rapid reviews of health interventions, where search strategies are always designed by information specialists and peer reviewed using the Peer Review of Electronic Search Strategies (PRESS) statement 37 (see recommendation 6), we recommend using CENTRAL as the primary database. This is a highly concentrated source of reports of randomised controlled trials and quasi-randomised controlled trials. In many regions, CENTRAL is free through the Cochrane Library.40 In addition, it is accessible to Cochrane members through the Cochrane Register of Studies Online (https://crso.cochrane.org/).41 Additional searches of Medline, such as through PubMed, and possibly Embase (if access is available) may be limited to the previous two months to capture the most recently published studies, as CENTRAL is currently only updated once a month.93942
▪ If CENTRAL and Embase are not available, searching Medline combined with an appropriate supplementary search (eg, a study register such as ClinicalTrials.gov, using the PubMed similar articles feature) can be considered, but this strategy might not be appropriate for all topics.914
For other rapid reviews that include non-randomised studies, database selection should be carefully considered depending on available time and resources. In many cases, Medline will be the most relevant database, but this is not always the case.1043 Searching specialised databases such as CINAHL, PsycInfo, and the Education Resources Information Center (ERIC), may be necessary for specialised review topics (eg, CINAHL for rapid reviews related to nursing care, PsycInfo for rapid reviews related to mental health, or ERIC for rapid reviews related to educational interventions); see subsection w2 in the supplementary file.
Recommendation 6: Use the PRESS checklist to peer review the primary search strategy
The primary search strategy should be peer reviewed using the PRESS checklist when possible.37 If the use of PRESS is not possible, at a minimum the search strategies should be double checked for typographical or spelling errors, missed keywords, and correct use of Boolean operators (AND, OR, NOT). Evidence suggests that the absence of peer review of the search strategy often results in many missed studies, and unless captured in accompanying supplementary searches, these studies would not appear in the published rapid review.44
Recommendation 7: Assess the need for grey literature and supplemental searching. Justify the sources to be searched
For rapid reviews, we recommend limiting grey literature and supplemental searches. For some topics, however, a search of grey literature may be more important than a search of conventional databases. If warranted, consider limiting supplemental searches to clinical trial registries and review of reference lists in included studies or similar articles searches to identify potentially relevant studies (see subsection w2 in the supplementary file file). More in-depth best practice suggestions on searching for rapid reviews are available.1445
Study selection—title and abstract and full text screening
Recommendation 8: Employ piloting exercises at abstract and full text screening levels to allow team members to test the study selection process on a selected sample of records to ensure a consistent approach to screening
Before the start of screening, a pilot exercise should be conducted using a purposive selection of records (eg, 50-100 records that reflect the complexity of the topic) assessed by the entire team of screeners to test and revise the screening forms and adapt the eligibility criteria, if necessary. This pilot exercise also allows a discussion of unclear abstracts and identification of potential difficulties in the study selection process. To ensure a consistent approach to screening is used across the entire team, all screeners should use a title and abstract screening form followed by a full text screening form. These forms should include details on the eligibility criteria with examples and can be phrased as screening questions. The screening forms should be adapted after discussions among the team, if necessary.
Recommendation 9: Conduct dual and independent screening of a proportion of records, assess reviewer agreement, and proceed with single screening if agreement is good
Generally, we recommend that two reviewers should screen at least 20% of the records, check the level of agreement, and discuss any discrepancies. Teams can proceed with single screening if agreement is high (eg, κ is ≥0.8).4647 If agreement is low, the screening team should proceed with dual reviewer screening until a better agreement has been achieved. The suggestion of 20% is based on our experience and is therefore not evidence based. The proportion may differ depending on the number of records to be screened, available resources, complexity of the review topic, and reviewers’ experience. We recommend the same approach for full text screening (see subsection w2 in the supplementary file). If resources allow, one person could check all excluded full texts to ensure no relevant study was unintentionally excluded. Further details on this process are available elsewhere.15
If a search yields a small number of records, such as a few hundred citations, consider dual, independent screening, if it is feasible. Although screening with only one reviewer for each record may be a practical solution for certain rapid reviews, we do not recommend this for Cochrane rapid reviews. Findings from two studies indicate that single screening of the titles and abstracts is not equivalent to dual screening, as more studies are missed.548 Nonetheless, forthcoming advances in automation (eg, active machine learning)4950 and crowdsourcing851 have the potential to reduce screening time when conducting rapid reviews.
Recommendation 10: Limit data extraction to only the most important data fields relevant to address the review question
No minimum set of data extraction items exists for study characteristics and outcome data. For a rapid review, however, extraction should be limited to only the most important data fields. For example, outcome data should only be extracted for those outcomes deemed as most relevant for decision making (see recommendation 3.2).
Recommendation 11: For data extraction, employ a piloting exercise to ensure all team members perform it consistently and correctly
We recommend using a form for data extraction (eg, a table or spreadsheet) and involving all reviewers in a data extraction pilot test using the same studies (at least two). This should help to identify misunderstandings early on and should reduce disagreements during the data verification step (see recommendation 12). The form should also help in deciding which study characteristics and outcome data should be extracted using concise descriptions of participant, intervention, comparator characteristics, and outcomes assessed (see recommendation 10).
Recommendation 12: Have one person extract the data, and for critical data that can affect the results or conclusions, have a second person verify the data for accuracy and completeness
We recommend that one person should extract the data for rapid reviews, with a second person verifying the key data (eg, definitions of outcomes, outcome data) for completeness and correctness to ensure integrity of the review. The second reviewer will also need to check the full texts. Any disagreements should be resolved through discussion and consensus between reviewers, involving a third reviewer if agreement cannot be reached (see subsection w2 in the supplementary file). If time and resources allow, a second person should verify the remaining data, such as characteristics of the study.
Recommendation 13: When available, extract data directly from existing systematic reviews rather than from primary studies
If good quality systematic reviews are included in the rapid review, consider extracting data directly from the systematic review. According to a case study, this approach saved time and did not alter the review results.52 However, extracting data directly from a systematic review requires good reporting of the data included in the review. As stated in recommendation 11, a second person should check the critical data extracted from the systematic review for completeness and correctness (see subsection w2 of the supplementary file).
Risk of bias assessment
We discourage omitting the risk of bias assessment entirely, as it informs the interpretation of the results.
Recommendation 14: Use validated and study design specific tools to assess the risk of bias of the included studies
To effectively manage the risk of bias assessments, it is important to use validated assessment tools specific to the study design(s) included in the rapid review. For example, for Cochrane rapid reviews, versions 1 and 2 of the Cochrane risk of bias tool for randomised trials (RoB 1 and RoB 2, respectively)5354 should be used for randomised controlled trials. Permitting both outcome level risk of bias (RoB 2) and domain level risk of bias (RoB 1) assessments in rapid reviews is suggested for practical purposes. RoB 1 assessments are generally quicker and require fewer resources than RoB 2 assessments at the outcome level. However, different rapid reviews may require different levels of detail and granularity in the risk of bias assessment that may depend on the specific characteristics of the review, such as the complexity of the interventions, type of outcomes, and available data. Allowing either approach ensures an efficient use of available resources while maintaining the quality of the review.
For non-randomised interventional studies, the Risk Of Bias In Non-Randomised Studies-of Interventions (ROBINS-I)55 should be used. AMSTAR 2 (A MeaSurement Tool to Assess systematic Reviews)56 or ROBIS57 may be used to assess risk of bias in systematic reviews.
Recommendation 15: Focus the risk of bias assessment at least on the most important outcomes
When using risk of bias tools with questions that rate the risk at an outcome level (not study level), such as RoB 2, limit the risk of bias ratings to the outcomes important for decision making (see recommendation 3.2).
Recommendation 16: Have one person perform the risk of bias assessment, with a second person verifying the judgements
The recommended approach to risk of bias assessment involves one reviewer performing the assessment, and another reviewer verifying the judgements. If only a small number of studies are eligible for inclusion, consider dual, independent risk of bias assessments for key outcomes, if feasible. In preparation, it may be helpful for all involved in risk of bias assessment to assess a small number of studies, such as two or three, concurrently and discuss the corresponding judgements so that any discrepancies may be identified and resolved (see subsection w2 in the supplementary file).
Synthesis
Teams involved in rapid reviews need to develop an appropriate analysis plan, which should be included in the protocol (see recommendation 2).
Recommendation 17: Provide a descriptive summary of the included studies
Providing a descriptive summary of the included studies at the outset of the synthesis stage helps to confirm if the studies are similar and reliable enough to synthesise and if it is possible to pool results. “Similar” studies means that they have similar PICOs, and, ideally, study designs.
Recommendation 18: Perform a synthesis of the findings
For rapid reviews that only include primary studies, reviewers need to decide how to group and tabulate data based on the review question, the type of data included, and what was planned for in the protocol, to the extent possible. Beyond a simple descriptive summary, a narrative interpretation of the evidence from multiple studies should be conducted for all rapid reviews. Reviewers should organise the synthesis around the elements of the PICO question framework, with findings grouped by key questions, comparisons, interventions, and outcomes. If a meta-analysis is possible, a descriptive summary of the body of evidence contributing to the meta-analysis is needed to interpret the collective evidence fully. Synthesis without meta-analysis (SWiM) reporting guidelines should be considered to promote transparency of narrative reporting of evidence synthesis.58
Recommendation 19: Consider a meta-analysis if appropriate and resources permit
If data are sufficient to consider a meta-analysis, the standards for a systematic review equally apply to a rapid review; meta-analysis will depend on the type of data and information provided in the individual studies. It is important to involve a statistician familiar with systematic reviews and meta-analyses. The depth and details of analysis will vary depending on the volume and type of included studies.
Recommendation 20: Consider how to synthesise evidence when including one systematic review or more
When including systematic reviews in a rapid review, synthesis may involve adding primary studies to an existing meta-analysis or narrative synthesis from the systematic review. If more than one systematic review is included, the overlap of primary studies must be identified. A formal study of overlap (eg, corrected covered area59) does not need to be undertaken for a rapid review. Instead, a cursory examination of overlap may be presented in tabular form to identify all included systematic reviews with the relevant primary studies to explore why reviews agree or disagree on important findings. A publication on synthesis for rapid reviews will be forthcoming as part of the Rapid Reviews Methods Series led by the Cochrane Rapid Review Methods Group.
Certainty of the evidence
Recommendation 21: Use the GRADE approach to assess certainty of evidence if time and resources allow
Cochrane Reviews incorporate the Grading of Recommendations Assessment, Development and Evaluation (GRADE) approach for rating the certainty of evidence60 as it is the established benchmark for use in guideline development. Several examples have been published where the GRADE approach has been used for rapid reviews.616263 We therefore recommend fully implementing GRADE for both Cochrane and non-Cochrane rapid reviews if time and resources allow. We also recommend using GRADEpro, an open access software tool for rating certainty of evidence in evidence syntheses to apply GRADE.64 Reviewers should always present results in a summary of findings table when rating certainty of evidence and use explanatory footnotes to outline reasons for uprating or downrating judgements.
Recommendation 22: Limit the certainty of evidence ratings to the main intervention and comparator and focus on critical outcomes
If time and other resources do not permit the full implementation of GRADE, we recommend that reviewers limit certainty of evidence ratings to the main intervention and comparator and focus on critical outcomes of benefits and harms. See recommendation 3.2 on the selection of critical outcomes.3360 The approach chosen should, in any case, be transparent, and any limitations acknowledged.
Recommendation 23: Have one person complete the GRADE assessment, with a second person to verify the assessment
To accelerate GRADE application, we recommend a single reviewer rating, and verification of all decisions (and footnoted rationales) by a second reviewer (see subsection w2 in the supplementary file). If effect estimates of a well conducted systematic review, meta-analysis, or network meta-analysis are incorporated to address parts of a key question of the rapid review, we advise using existing certainty of evidence grades from such systematic reviews.16 Further details on assessing the certainty of evidence for rapid reviews are available.16
Other best practice considerations
Recommendation 24: Provide a clear description of the selected rapid review approach, which includes outlining the restricted methods used. Additionally, discuss the potential limitations of these chosen methods and how they may influence the interpretation of the research findings
Describing the restricted methods used and pinpointing potential sources of bias or uncertainty in the findings resulting from methodological restrictions, enables end users to better assess the validity and reliability of the rapid review. This includes determining whether the methods were appropriate for the specific research question, studied population, and context being investigated. Additionally, this process serves to put the results in perspective, given the restrictions imposed by the methods of rapid reviews, helping to limit overgeneralisation of results or unwarranted conclusions.
Rapid reviews involve an iterative process
Sometimes, changes to the review protocol are necessary once a rapid review has started. For example, search variables may be expanded or limited depending on what the search yields, or eligibility criteria may need to be refined after the pilot screening. Therefore, the rapid review process should allow for post hoc changes to the protocol. Substantial changes should be discussed with the knowledge users involved, and any amendments should be tracked and reported in the rapid review. Moreover, authors should seek feedback from the knowledge users throughout the process to ensure the review meets their needs.
Incorporate the use of systematic review software to streamline the process
We strongly encourage using software to help produce rapid reviews to improve the efficiency of screening, tracking, and documentation, and to reduce human error. Online systematic review software enhances collaboration by allowing real time project management and multiuser participation across geographical boundaries. Importantly, the software enables members of the rapid review team to work in parallel across all stages of the review, and it provides a fully transparent process. It also facilitates the incorporation of protocol amendments and other post hoc changes that may be needed during the conduct of a rapid review. The use of software also increases efficiency through the automated collation of the screening results (inclusions and exclusions). Those undertaking rapid reviews and other types of syntheses should look for ways to harness innovation, using software and adopting automation tools that reliably assist in streamlining stages of a review’s conduct. One such example is Cochrane Crowd, which uses a machine learning platform and crowdsourcing to identify randomised controlled trials8 (for examples, see SR ToolBox at http://systematicreviewtools.com/).
Apply appropriate reporting guidelines
Importantly, given the methodological modifications inherent to rapid reviews, authors must be transparent in reporting their methods and results. See the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA-P) for reporting of rapid review protocols32 and PRIMSA-S for searches.38 Although an extension to PRISMA for rapid reviews is ongoing,65 until it is officially completed we suggest authors use the general PRISMA statement66 to the extent possible, and adapt it accordingly.
Discussion
This updated guidance on methods used in Cochrane rapid reviews of effectiveness has the potential to benefit many producers and users of this review type. Although this guidance is heavily posited within the Cochrane landscape, the recommended methods apply widely to anyone conducting a rapid review. This guidance builds upon our interim guidance and is based on an evaluation of the recommendations used in real time, the expertise of a broader group of experts in rapid reviews within and external to Cochrane, and a scan of the literature for additional publications on rapid review methods since our recommendations were first published. Therefore, the recommendations are both evidence informed and user informed.
Noteworthy changes to the updated recommendations include modifications to search sources for both Cochrane and non-Cochrane rapid reviews of randomised controlled trials and for study selection at both the title and abstract and the full text screening stages, where the process has been simplified. Subsection w3 in the supplementary file provides a comparison between the interim and updated recommendations. In addition, a new section on certainty of evidence has been added as a separate element from the synthesis of results. Using appropriate reporting guidance for the protocol, search strategies and review itself are also emphasised. Importantly, further explanations and available examples are included to help clarify considerations and recommendations for methods in rapid reviews. Some additional practical tips for rapid review teams to increase efficiencies in the review process are also provided.
Although a multipronged approach, where more than one method restriction is combined, may increase timely production of reviews, users of this guidance can still label a review as a rapid review even if some of the recommendations are not followed. The key is that the pros and cons of each methodological restriction used in a rapid review should be weighed against the scope and complexity of the review topic and other circumstances of the review, including the timeline, number of team members involved at various stages of the review and their level of expertise, along with the potential for introducing bias. Rapid review teams therefore need expertise in systematic review methods to provide balanced methodological judgements when deciding which methods restrictions to use and how this might impact the findings of a particular review.
It is important to clarify the distinction between recommendations targeting Cochrane rapid reviews and those with a more general applicability. Some recommendations are tailored to Cochrane rapid reviews because they align with Cochrane’s standards, procedures, and objectives that do not apply to all contexts of rapid reviews. On the one hand, the guidance ensures that Cochrane rapid reviews consistently meet the organisation’s rigorous quality standards. On the other hand, certain recommendations have broader applicability across all methodologies and contexts of rapid reviews. These recommendations encompass fundamental principles and best practices that are not exclusive to Cochrane but are relevant for all rapid reviews of effectiveness. The recommendations address core aspects of conducting rapid reviews, such as the importance of involving knowledge users, use of protocols, transparency, and documentation, which are essential regardless of the specific review framework. Recognising that the methodology of rapid reviews is an evolving discipline with diverse applications, the guidance aims to balance between providing Cochrane specific direction and accommodating the broader community involved in rapid reviews. This approach acknowledges the need for flexibility and adaptability, enabling the authors of rapid reviews to apply relevant guidance while considering their specific requirements and objectives.
Our approach to the methods of rapid reviews stands out from existing guidance because it was crafted explicitly within the Cochrane context, concentrating on rapid reviews of interventions. In contrast, other guides for rapid reviews have targeted health policy and systems research, public health, or rapid guideline development during public health emergencies.676869 What also distinguishes our guidance is its sharp focus on the conduct of rapid reviews rather than planning, packaging, or dissemination. Notably, our recommendations for each step of the review process are specific, setting the guidance apart from others that typically offer a more general overview of common practices in rapid reviews. It is also worth noting that Cochrane rapid reviews are explicitly designed to address urgent and high priority questions requested by decision makers, aligning with a specific mandate that not all other guides necessitate. Although these other guides are also evidence informed and we encourage their use, our guide is uniquely grounded in empirical evidence evaluating methods of rapid reviews,12 further enriched by expert input13141516 and feedback from a formal evaluation of its real world use.7
Although not all stages of a rapid review had corresponding evidence to inform the updated recommendation, as stated previously, we know that many established steps in the systematic review process are also based on evidence that is limited, outdated, or not available. Therefore, any future methodological research that informs rapid reviews will also be beneficial for systematic reviews. Future research should examine ways to adapt this guidance beyond interventions of effectiveness to other review question types, such as rapid reviews of diagnostic test accuracy or screening. Developing criteria for determining the appropriateness of undertaking rapid reviews versus systematic reviews or living systematic reviews would also be of value.
Optimising the review process, including study selection, data extraction, and synthesis of results, enables quicker production of rapid reviews. A variety of software tools, involving extraction of metadata, automation, screening software, and machine learning, have the potential to contribute to this optimisation. These tools aim to help streamline the review process, enhance transparency and reproducibility, and minimise reviewer bias. Collaboration software and live documents also facilitate real time exchanges, keeping all stakeholders in alignment throughout the process. Looking to the future, the potential of artificial intelligence (AI) will likely further advance the overall process for both rapid reviews and systematic reviews by providing intelligent assisted solutions to potentially speed up and increase efficiency of many review steps. AI may also reduce errors, improve accuracy, and improve repeatability and reproducibility of rapid reviews.
Overall, the Cochrane Rapid Reviews Methods Group offers updated, actionable recommendations to support the conduct of rapid reviews when the need for evidence is urgent. Because best practice is still limited by the lack of currently available evidence for some shortcuts used for rapid review methods, the Cochrane Rapid Review Methods Group will continue to promote research to close these gaps, monitor the published literature as additional abbreviated methods are formally evaluated, and update these recommendations as needed. Ultimately, we hope that uptake of this guidance will lead to more useful, robust, and rigorous rapid reviews, thus facilitating timely, evidence based decision making in healthcare.
Conclusions
This article offers updated guidance composed of 24 recommendations on methods for conducting rapid reviews of effectiveness produced within Cochrane and beyond to address urgent and high priority questions often requested by decision makers. The guidance aims to encourage the thoughtful use of best practices that are both user informed and evidence informed when applying abbreviated systematic review methods to rapid reviews.
Ethics statements
Ethical approval
Not required.
Acknowledgments
We thank the collaborators who contributed to this updated guidance, and Piotr Ratajczak who updated the literature scan for newly published studies. We also thank our colleagues who worked with the convenors of the Cochrane Rapid Reviews Methods Group to create a series of articles in BMJ Evidence-Based Medicine. These articles give more details about the guidance we recommend in the current article. As such, we want to recognise these colleagues for their expertise in suggesting changes to the guidance: Andrea Tricco, Maureen Smith, Danielle Pollock, Irma Klerings, Shannon Robalino, Andrew Booth, Camila Micaela Escobar-Liquitay, Isolde Sommer, Siw Waffenschmidt, Anna Noel-Storr, Livia Puljak, Leila Kahwati, Meera Viswanathan, Amir Qaseem, Eli Akl, Holger Schuenemann, Jane Noyes, Catherine Houghton, Fiona Campbell, Anthea Sutton, Hanan Khalil, Lisa Affengruber, Miriam Van der Maten, James Thomas, Chris Mavergames, and René Spijker. In addition, a special thank you to Irma Klerings and Anna Noel-Storr for their specific review of the search related recommendations. We would also like to thank members of the Cochrane Methods Executive for reviewing and providing comments.
Footnotes
Contributors: CG, CH, MT, GG, BNS, DD, CK, UG, and VJK contributed to the conceptualisation of this paper. CG wrote the first draft of the manuscript and revised the manuscript with input from CH, MT, GG, BNS, DD, CK, UG, and VJK. All authors read and approved the final version. CG is the guarantor and attests that all authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding: This work was funded in part by Cochrane and in-kind support provided by Cochrane Austria, University for Continuing Education Krems, Krems, Austria. The funder had no role in the development of this guidance, writing of the report, or decision to submit the article for publication.
Competing interests: All authors have completed the ICMJE disclosure form at http://www.icmje.org/disclosure-of-interest/ and declare: support from Cochrane for the submitted work. Certain methods studies upon which some of the recommendations are based have been carried out by authors of this guidance (GG, BNS, CH, CG, VJK, and CK). UG, BNS, and MT planned and carried out the evaluation study of the interim rapid review methods guidance.
Provenance and peer review: Not commissioned; externally peer reviewed.