Searching clinical trials registers: guide for systematic reviewersBMJ 2022; 377 doi: https://doi.org/10.1136/bmj-2021-068791 (Published 26 April 2022) Cite this as: BMJ 2022;377:e068791
- Kylie E Hunter, senior evidence analyst1,
- Angela C Webster, director1 2,
- Matthew J Page, senior research fellow3,
- Melina Willson, manager1,
- Steve McDonald, senior research fellow3,
- Slavica Berber, project officer4,
- Peta Skeers, Cochrane information specialist1,
- Ava G Tan-Koay, Cochrane information specialist1,
- Anne Parkhill, information specialist5,
- Anna Lene Seidler, senior research fellow1
- 1Evidence Integration, NHMRC Clinical Trials Centre, University of Sydney, Camperdown, NSW, Australia
- 2School of Public Health, University of Sydney, Camperdown, NSW, Australia
- 3Methods in Evidence Synthesis Unit, School of Public Health and Preventive Medicine, Monash University, Melbourne, VIC, Australia
- 4Health Technology Assessment Team, NHMRC Clinical Trials Centre, University of Sydney, Camperdown, NSW, Australia
- 5Centre for Health Communication and Participation, La Trobe University, Melbourne, VIC, Australia
- Correspondence to: K E Hunter email@example.com
- Accepted 25 February 2022
Searching of clinical trials registers is strongly recommended for comprehensive systematic reviews and is mandatory for best practice Cochrane reviews, yet guidance is scarce on how to perform these searches and how to harvest information for identified studies from registers
This article provides 11 steps and several key recommendations on where to search, how to formulate search strategies, efficient screening methods, and reporting searches
This guidance can be used by researchers to identify additional eligible studies and obtain unpublished results for inclusion in systematic reviews, thereby reducing publication bias and research waste
Retrieval of study information from trial registers is also useful to identify research gaps and inform research prioritisation, to identify studies and potential investigators for collaborative methods such as a prospective meta-analysis, and to plan updates of traditional or living systematic reviews
Systematic reviews are a cornerstone of evidence based medicine.1 Positioned at the top of the evidence hierarchy, these reviews frequently underpin healthcare guidelines, policy, and practice.2 Yet, their validity relies on identification and inclusion of all relevant and available evidence, both published and unpublished. Unpublished studies, however, are often difficult and time consuming to identify, resulting in suboptimal attempts at retrieval or even complete omission from systematic reviews.34567 This incomplete inclusion is problematic given that only about half of all biomedical studies ever publish their results,8 and those that do, tend to yield more positive results and larger effect sizes than unpublished studies9 (phenomena known as publication bias and selective outcome reporting). Substantial research waste10 is the result, as is reliance on a biased subset of the evidence, which can lead to inappropriate recommendation of treatments that have lesser to no effect or could even be harmful.11 For instance, more than 80 million patients had used the anti-inflammatory drug rofecoxib before it was withdrawn due to discovery of unpublished analyses showing that the drug increased the risk of myocardial infarction and stroke.12
Best practice for systematic reviewers involves searching for unpublished studies to synthesise the totality of available evidence and to reduce bias,13 and clinical trials registers represent a key resource for this search. Clinical trials registers are publicly available online registers of planned, ongoing, and completed clinical studies (primarily clinical trials but also some observational studies).14 The registers include structured information on study design, conduct, and administration, and, more recently, have incorporated results reporting and investigator’s data sharing plans. Registers are becoming increasingly comprehensive since prospective registration (that is, registration before enrolment of the first participant) has been mandated by several regulatory, ethical, and legislative bodies,151617 and registration of observational studies is gaining support.18 The World Health Organization’s Registry Network has 17 primary registries, which meet specific criteria for content, quality and validity, accessibility, unique identification, technical capacity, and administration.17 These registries plus ClinicalTrials.gov (https://www.clinicaltrials.gov), which is provided by the US National Library of Medicine and is the largest clinical trials register (n=391 704 records on 11 October 2021), are all recognised by the International Committee of Medical Journal Editors (known as ICMJE), and all 18 registries provide data for WHO’s International Clinical Trials Registry Platform (ICTRP; https://trialsearch.who.int), which on 31 October 2021 included more than 700 000 records of clinical trials.14 Identification of unpublished studies from trial registers offers many advantages. For instance, reviewers can obtain unpublished results by direct communication with registrants or by direct extraction from registration records. Since WHO introduced results reporting requirements in 2015,19 more registers have added results reporting functions, which has increased the opportunities for direct data extraction. In cases in which results cannot be retrieved, detection of unpublished studies is still useful to identify potential publication bias, by allowing an estimate of the amount of unavailable evidence that cannot be included in a review. In addition, identification of forthcoming evidence can inform decisions on whether additional studies on a topic are needed, when to plan an update of a traditional or living systematic review, and whether collaboration is possible via next generation systematic review methods. For instance, if several ongoing studies answering similar research questions are identified, researchers can agree to synthesise their results on completion using a prospective meta-analysis method.20
Searching trial registers can be challenging because of the varied and relatively unsophisticated search interfaces compared with large bibliographical databases such as Medline.2122 This difficulty is not surprising given that registers were initially designed to deal with transparency and publication bias,23 rather than specifically as a research resource for systematic reviews. Perhaps consequently, clear guidance on how to conduct register searches is lacking. The updated Cochrane Handbook, which is widely regarded as outlining the best practice method for conducting systematic reviews, mandates searching trial registers (via ClinicalTrials.gov and WHO ICTRP); however, a technical supplement that provides guidance on how to search these registers only contains a few sentences of advice.24 This brief text contrasts with the extensive guidance dedicated to search strategies for bibliographical databases to identify published studies.13 Therefore, to improve the validity and reliability of systematic reviews and reduce research waste and bias, consistent and clear guidance on how to search for registered studies is needed.
We present a step-by-step guide on how to search for registered studies for inclusion in systematic reviews (fig 1). A registered study is defined as one that has met registration requirements of an ICMJE and WHO recognised registry, and that has been issued with a registration number. Although registers primarily include interventional trials, this guidance could be equally applied for identifying registered observational studies. Our advice is based on: a review of the literature; information available on registry websites; an online survey and consensus workshop on 18 June 2021 among coauthors (the steering group) who have extensive international experience as information specialists, trial registry staff, systematic reviewers, biostatisticians, methodologists, clinical trial experts, guideline developers, and clinicians; and an online survey of international experts drawn from Cochrane information specialists and health technology assessment reviewers open from 9 July to 2 August 2021 (n=14 respondents). Further details are available in the supplementary material: the methods used to develop our guidance (appendix 1), the author and steering group areas of expertise (appendix 2), the results of the steering group (appendix 3), and international surveys (appendix 4). We provide guidance on the methods, rationale, and challenges for each step, to increase search literacy, and to enable more reviewers to efficiently conduct effective and reproducible searches of trial registers.
Case study—Transforming Obesity Prevention for CHILDren
We illustrate each step using the example of an ongoing systematic review and individual participant data meta-analysis of randomised controlled trials evaluating behavioural interventions for the early prevention of obesity in children (Transforming Obesity Prevention for CHILDren, TOPCHILD).25 TOPCHILD searches are updated annually, and for this illustrative case study, we focus on the most recent search conducted on 18 March 2021 (ClinicalTrials.gov) and 22 March 2021 (ICTRP). To date, we have identified 71 eligible trials, of which 15 were identified only by searching trial registers, showing the importance of trial registers as a source of information for systematic reviews.
Step 0: Defining the research question and eligibility criteria
A first step in systematic reviews is to define the research question and eligibility criteria. Searches for registered studies should not commence until this preparatory step is undertaken.
Use an appropriate framework, such as the population, intervention, comparator, outcome (PICO) framework, to define research question and eligibility criteria.
Case study—TOPCHILD answers the primary research question25: compared with usual care, no intervention, or attentional control, what are the effects of behavioural obesity prevention interventions that are focused on the parent or caregiver and commence during pregnancy or infancy on child weight status at age 24 months? The PICO system27 was applied to define the eligibility criteria in an iterative process through extensive consultation with experts in the specialty and consumers. The population is the parents or caregivers (including pregnant women) and their infants aged 0-12 months (at baseline); the intervention is the behavioural interventions targeting parents or caregivers, with the primary aim of preventing obesity in their children; the comparator is usual care, no intervention, or attentional control; the outcome is trials must collect at least one child weight related outcome post-intervention—for example, body mass index (BMI) or BMI z score, prevalence of overweight or obesity, and percentage fat content and adiposity; and the study type is randomised controlled trials (at individual level or by cluster).
Step 1: Determining where to search
With many different registry resources available for searching, researchers can find it challenging to decide where to search to maximise retrieval of relevant studies without imposing unnecessary duplication or burden.
As a minimum, search ClinicalTrials.gov and WHO ICTRP.
Explanation—ClinicalTrials.gov registry consistently scores higher than other registries in reviews of technical performance, functionality, and available features.282930 WHO ICTRP is a database or meta-register that includes data from 18 recognised registries globally. Although ICTRP includes data from ClinicalTrials.gov, Cochrane mandates searching both resources separately because unique records can be found from each and ClinicalTrials.gov has more search features and greater functionality.132431 Furthermore, ICTRP is not always accessible owing to technical reasons,30 and it might not be as up to date—for example, on 11 October 2021, the ICTRP website indicated that the last ClinicalTrials.gov data file was imported on 5 July 2021.
Since 2019, the Cochrane Central Register of Controlled Trials (CENTRAL; https://www.cochranelibrary.com/central) has included registration records sourced from ClinicalTrials.gov and ICTRP. However, searching CENTRAL alone is not supported by Cochrane guidance1324 and is insufficient to identify registered studies because of its low sensitivity.32 This low sensitivity might be because register records as they appear in CENTRAL are less comprehensive than the original register entry, and thus are at a greater risk than other systems of being missed in a search. Regardless, CENTRAL will often form part of the search methods to identify published studies.
For some research questions, consider searching the European Union Clinical Trials Information System for drug trials (known as EU-CTIS, which replaced the European Union Clinical Trials Register (EU-CTR) on 31 January 2022) or regional registries (region specific research questions).
Explanation—Given ICTRP combines data from 18 recognised registries, it is generally not necessary to search other registries individually, and any additional yield might not be justified by the extra resources spent. In some instances, however, and if resources allow, other registries within the WHO Registry Network can be searched separately. For systematic reviews focusing on drug trials, reviewers could consider searching CTIS because this resource focuses on interventional clinical trials on medicines conducted in the EU. For geographically restricted research questions, region specific registries could be searched in addition to ClinicalTrials.gov and ICTRP. For instance, if the participants of interest are Indigenous Australians, we would recommend searching the Australian New Zealand Clinical Trials Registry (known as ANZCTR), and if the topic of interest is Chinese herbal medicine, a separate search of the Chinese Clinical Trials Registry might be prudent to maximise sensitivity.
Case study—For TOPCHILD, we searched ClinicalTrials.gov and ICTRP for registered trials. Given the interventions of interest were behavioural, searching of the drug focused EU-CTR (now CTIS) was not appropriate, and because obesity is a global health issue, regional searches were deemed unnecessary.
Step 2: Identifying key search concepts and deriving search terms
Search concepts describe the broad subject areas or topics of interest. They are used to derive search terms, which are specific words, phrases, and synonyms that reflect these concepts. These terms will be used to formulate a search strategy and determine the relevance of search results.
Identify one or two key concepts from the PICO (or other appropriate framework, step 0), typically population and intervention. For each concept, list synonyms or alternative terms expressing the same concept.
Explanation—Firstly, a mind map is useful for possible search terms for the key PICO elements—typically the population and intervention. Ideas can also be gathered from search strategies of systematic reviews on similar topics (if used substantively, these should be cited),33 thesauruses, and Medical Subject Headings (MeSH) or other terms indexed for any known eligible studies. Both ClinicalTrials.gov and ICTRP also incorporate synonym searching using the Unified Medical Language System. For instance, if the term “obesity” is searched in ClinicalTrials.gov, the synonyms “obese” and “adiposity” are also automatically searched; but note that this function cannot be disabled in ClinicalTrials.gov if a search for a specific phrase only is preferred.
Case study—Before we searched the registers, we had formulated a complex search strategy for bibliographic databases (Medline, Embase, CENTRAL, CINAHL, Psycinfo) in consultation with a Cochrane information specialist. The strategy incorporated a wide range of concepts, including overweight and obesity; behavioural and lifestyle interventions; nutrition, diet, and feeding; physical activity; sedentary behaviours; sleep; health promotion and prevention; and children and families. For our register searches, we chose two main concepts: overweight/obesity and child. We decided to omit the intervention concepts because the diverse variety of eligible intervention types precluded reasonable specificity. Table 1 shows the synonyms or alternative search terms derived from our two chosen concepts. These terms were discussed among members of the research team and derived by consulting relevant Cochrane34 and non-Cochrane35 reviews, as well as from the ClinicalTrials.gov synonym function and MeSH trees.
Step 3: Formulating search strategies
This step aims to identify as many relevant records as possible to contribute to the review (that is, maximise sensitivity), while also balancing with reasonable specificity and precision so that screening is feasible.24 A search strategy is the structured combination of concepts and terms used to search a database. Although bibliographical databases offer a broad suite of tools to formulate complex search strategies, the availability and functionality of similar tools on trial registries are limited and vary widely by trial registry (table 2).
Focus search strategies on one or two key concepts identified in step 2 and aim to maximise sensitivity while balancing against reasonable specificity.
Explanation—Reviewers should start with a basic search that focuses on the single most specific concept, typically P (population or health condition) or I (intervention).2131 To enhance sensitivity, various synonyms and related terms should be combined for this concept using the Boolean operator “OR”—for example, “overweight OR obesity OR obese OR adiposity”. If the number of results retrieved is too high thus rendering screening infeasible, a second concept should be added to reduce the number of hits to only those where the two concepts overlap (fig 2).37 This overlap can be achieved using parentheses to group the terms for each concept and then combining concepts with the Boolean operator “AND”—for example, “(overweight OR obesity OR obese OR adiposity) AND (baby OR infant OR child OR toddler).” Recent updates to the ICTRP now allow use of parentheses and Boolean operators within the basic search; however, longer search strings can cause the system to time out or bring up error messages. If issues are experienced, we recommend conducting separate searches for each combination of concept terms—for example “overweight AND baby” then “overweight AND infant”. Following each search, the results would need to be downloaded, combined, and the duplicate records removed.
The advanced search interface of ICTRP should be used with caution because one study31 found that the search can reduce sensitivity (often without improvements in specificity) compared with the basic interface. The same study found that the advanced search on ClinicalTrials.gov seems to increase precision while maintaining sensitivity and therefore might be appropriate when there are large numbers of search results.
Adjust search strategies according to the specific registry resource and become familiar with search tools and rules of each.
Explanation—Search interfaces on trial registries tend to be more simplistic and less sophisticated than those on large bibliographical databases,2122 due to limited funding and resources, their relatively small size, and the quality and structure of registration records compared with bibliographical records. Variation across registries also requires that search strategies are adjusted in each database.2122 Although ClinicalTrials.gov and ICTRP have some similarities in search functionality, many differences exist that need adjustments (table 2). These limitations can make the process challenging and resource intensive, and consultation with an experienced librarian or information specialist might be useful to improve efficiency, if possible.13
Test whether the search strategy retrieves pre-identified eligible studies (if possible).
Explanation—The combination of search terms should be trialled and revised in an iterative process. If the reviewer is already aware of studies that have been pre-identified as eligible, and knows that these studies are registered, we recommend testing the initial search strategy to see whether these are retrieved. Be prepared to experiment with search interfaces to become familiar with how they work. Try adding or removing search terms if the number of records retrieved seems too low to have captured everything, or too high for screening feasibility. The appropriate number of records retrieved should be determined on a case-by-case basis, in consideration of the specificity of the concept searched and previous availability of research knowledge. For instance, searches for a rare disease or a new drug would be expected to yield fewer and more precise results than they do for a common condition such as pregnancy or drug such as paracetamol (acetaminophen). Results from register searches might also be used to test the sensitivity of search strategies used for other databases, such as Medline. Specifically, for each relevant registration record identified, a targeted search should be conducted for matching publications regardless of the recruitment status listed. If a substantial number of relevant publications are identified that were missed during the initial database search, then reviewers should revise and repeat their database search strategy to improve sensitivity.
Apply filters (eg, by study type or participant age) only in exceptional circumstances.
Explanation—Most registries offer a range of filters that can be used to refine a search (eg, by study type or participant age). Although filters can be a powerful tool to increase the precision of a search, we recommend not to use them unless circumstances are exceptional—for example, when resources are extremely limited or only a rough search is required for scoping. The reasoning for this recommendation is that optimal use of filters relies on accurate data categorisation in the registry, which is not always achieved for registration records. For instance, while testing a search string with and without application of the ClinicalTrials.gov study type filter for “Interventional Studies (Clinical Trials),” we identified three (5%) of 57 records that were randomised controlled trials but had been incorrectly categorised as observational studies and, therefore, would have been wrongly excluded by the filter (Hunter, unpublished data, 2021).
Avoid limiting searches by recruitment status because this field might not be up to date and therefore eligible studies could be missed.
Explanation—Reviewers often limit register searches by “completed” recruitment status because these are the studies for which they logically expect results data to be available.38 However, this approach should be avoided for two key reasons. Firstly, this limitation risks missing a substantial proportion of studies that are actually completed or have published results because recruitment status listed on trial registers is often out of date or inaccurate.3839 For instance, as of 23 December 2021, 46 406 (12%) of 399 046 records on ClinicalTrials.gov were labelled as having an “unknown” recruitment status, meaning that their last known status for “recruiting,” “not yet recruiting,” or “active, not recruiting” had not been updated or verified within the past two years despite having passed the completion date. This finding also underscores the need to conduct targeted searches for publications linked to studies identified by register searches, regardless of recruitment status. Secondly, filtering precludes the ability to identify ongoing trials and therefore negates many of the advantages of register searches described in this paper, such as the ability to assess the potential impact of publication bias on review findings.
Case study—We chose “overweight/obesity” as the primary concept of interest and avoided application of any filters or limits (eg, by start or end dates or recruitment status). When the output yielded too many irrelevant results, we added the concept “child” to enhance precision.
WHO ICTRP search strategy—Table 3 shows the TOPCHILD search strategy formulated for the ICTRP. We used the basic search interface to search for only the key concepts “overweight/obesity” and “child.” At the time of searching (22 March 2021), parentheses were not available for nested searching, only very short search strings worked, and synonym and truncation functions had issues. For instance, combining the search term “obesity” with variations of the term “infant” retrieved discrepant results (132 for “infants AND obesity”, 96 for “infant AND obesity”, 123 for “infant* AND obesity”), and the truncated term “obes*” retrieved fewer results than “obese” and “obesity”, indicating truncation was not functioning.28 Because of these issues, we chose to conduct multiple separate searches combining varied terms for the two chosen concepts, then merge and remove duplicate records (this process is described further in step 4).
The ICTRP search function has since been updated (20 July 2021), and we re-ran our search strategy on 15 September 2021 to evaluate key changes (table 3). Although we noted some apparent issues with synonyms, truncation, and duplication, sensitivity increased markedly from 0.18 to 0.78 (appendix 2), and we commend ICTRP’s efforts to improve the functionality of this important resource despite the limited funding and resources. These experiences from our case study highlight that reviewers should be cognisant of functionalities that might not always perform as intended, and therefore might require a flexible and adaptive approach.
ClinicalTrials.gov search strategy—Box 1 shows the ClinicalTrials.gov search strategy for TOPCHILD. We found the basic search interface to be sufficient for our requirements and were able to efficiently combine all terms for both concepts in a single search string. In the “Condition or disease” field, we linked several terms for the key concept “obesity” with the Boolean operator OR and this was combined with varied terms for the concept “child” in the “Other terms” field. The “Other terms” search encompasses many data fields, including study location and researcher affiliation. Thus, a few irrelevant studies were retrieved simply because they were conducted at an institution with “child” (or a synonym) in the name, eg, “Seattle Children’s Hospital”. Searching of fields that were further refined was not possible, a process that would be valuable to overcome this issue. Although ClinicalTrials.gov offers a filter for the word “child”, we chose not to use it to prevent erroneously excluding relevant records. On testing, we discovered that nine eligible trials enrolling mothers and infants (as dyads) only listed ages for the mother (>18 years) in the “Ages eligible for study” section, and thus, these trials would have been missed if the child filter was applied.
TOPCHILD (Transforming Obesity Prevention for CHILDren) search string for ClinicalTrials.gov advanced interface
Condition or disease: overweight OR obesity OR obese OR adiposity OR BMI OR weight gain
Other terms: baby OR infant OR child OR paediatric OR pediatric OR toddler OR offspring
2756 records were retrieved.
Step 4: Conducting the search, removing duplicate records, and preparing records for screening
Detailed records of all registry searches are important to keep for transparency, reproducibility, and efficiency.28 This record keeping will also assist with completing the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) flow diagram40 (step 7) and reporting the search (step 10).
Keep detailed records of all register searches, including the exact date the search was conducted, names of registers searched, interfaces used (basic or advanced), full search strings, and number of records retrieved from each.
Explanation—Reviewers should document the date that each search is conducted and by whom; trial registers searched; interface used (basic or advanced); full search strings, including any limits or filters applied; and number of records retrieved from each. This documentation can be done in Microsoft Word or Excel or, as suggested in one study,21 by screen capture or use of a note-taking software such as Evernote or OneNote. No options are currently available to save search strategies and history within ICTRP or ClinicalTrials.gov. However, ClinicalTrials.gov offers RSS (Really Simple Syndication) feeds and the option to save selected studies for easy retrieval, and if the search results page is bookmarked, updates will be automatic every time the page is opened.
Download search records into your preferred software and remove duplicates.
Explanation—Various methods are available for exporting and deduplicating search records in preparation for screening, and the optimal methods depend on the register searched and the reviewer’s preferred reference management or screening software (eg, Excel, Endnote, Refworks, Abstrackr, Mendeley, Covidence, DistillerSR, EPPI-Reviewer, Zotero, and Rayyan).13 As each search string is run, the records retrieved should be downloaded by methods appropriate to the chosen software. For instance, to download results to Excel, use the CSV or TSV options on ICTRP and ClinicalTrials.gov, whereas for Endnote, download results in plain text format from ClinicalTrials.gov and in XML format from ICTRP, then import them using the appropriate Endnote filter (https://endnote.com/downloads/filters/). Note that the non-bibliographical nature of registration records makes mapping of data to Endnote fields difficult, and therefore much of the information only imports to the “Notes” field or is omitted. The Cochrane Collaboration’s preferred software, Covidence,41 allows imports in EndNote XML format, and it is also compatible with Zotero, Refworks, Mendeley, or any tool that supports RIS, CSV, or PubMed XML formats.
Once all records are imported into the preferred software, duplicate records need to be identified and removed (a process often referred to as deduplication). We define duplicate records as records with the exact same registration number and title. Note that some researchers choose to register a single study on two different registers, resulting in two unique registration numbers. Although these records relate to the same study, they are not considered to be duplicates because they can contain different information, particularly if one is more up to date than the other. In this instance, both records should be kept but grouped as a single study.
Although the literature offers plenty of advice on deduplication for bibliographical databases,4243 many of these methods are not applicable for registration records because they rely on sorting by data fields that are not collected by registries—for example, author, journal, volume, or pages. Instead, we recommend use of unique trial registration numbers for deduplication in Excel, either by highlighting and manually deleting duplicates using the “Conditional formatting” function, or by use of the automatic “Remove duplicates” function if the number of records is large. Registration records sourced directly from a registry contain more detailed and up-to-date information than do those obtained from ICTRP, and therefore should be retained in case of duplicates.
Case study—For TOPCHILD, we recorded search dates, who conducted the searches, registries and interfaces searched, full search strings, and the number of records retrieved for each in a simple table in Microsoft Word. All records retrieved from ICTRP and ClinicalTrials.gov were downloaded into Excel using the TSV format and combined to one spreadsheet. ICTRP and ClinicalTrials.gov records are formatted differently so we needed to manually align key columns containing registration ID, title, and study type. Next, we deduplicated records using the registration ID column and “Remove duplicates” function in Excel.
Step 5: Title screening (optional)
The purpose of this step is to remove any obviously irrelevant records so that fewer records need to be screened in full, thus improving efficiency. In contrast with published studies, which generally have a structured abstract available for screening, the information available from downloaded registration records can range from only a title and web link to a full record of all available data fields. Depending on the amount of information available, the number of records, and personal preferences, some reviewers might choose to skip this step and screen all records in full.
If preliminary title screening is conducted, only exclude obviously irrelevant records.
Explanation—Screeners should review all study titles against eligibility criteria using their chosen software. Reviewers should be over-inclusive at this stage13 and only exclude studies that are obviously irrelevant. In our experience, titles are sometimes not overly representative of the study content. Any uncertainties should be resolved through discussion.
Case study—Two reviewers independently screened all registration record titles in Excel using separate copies of the deduplicated spreadsheet created in step 4. We added a column adjacent to the titles and populated this with either “maybe” (proceed to full text screening) or “no” (exclude) for each record. Discrepancies were resolved by consensus or consultation with a third reviewer. We then sorted and copied all records marked “maybe” to a new spreadsheet. Examples of records excluded by title were those stating obviously irrelevant participants (eg, “Adolescents With Hepatosteatosis” and “Obese Adolescent Girls”), obviously irrelevant interventions (eg, “metformin” and “Setmelanotide”), or obviously irrelevant health conditions (eg, “Dengue fever”, “Endometrial Cancer”, and “vision impairment”).
Step 6: Full record screening
This step determines the final eligible studies to be included in the review and contribute to results.
Screen full registration records at the source registry website.
Explanation—Full registration records should be viewed at the source registry website by use of the link or registration ID downloaded with search results. This ensures access to the most detailed and recent information, which can be lost when downloading, importing, or uploading records to various software, especially since records are not in bibliographical format. For instance, registration records imported into Covidence from ClinicalTrials.gov often include only the title and record link, and records on ICTRP contain fewer data than does the source registry.
Screen all records in full at least once and consider an independent second reviewer if resources allow.
Explanation—Best practice for study selection in systematic reviews is independent double screening to ensure that no eligible studies are missed.4445 However, for rapid reviews or when resources are limited, one experienced reviewer is sufficient to screen registration records. A key factor to consider when deciding the number of screeners is the potential consequences of missing an eligible study. For instance, if a prospective meta-analysis is being conducted, identification of eligible studies is essential before results are known, and therefore, missing a potentially eligible study at the screening phase might be more consequential than for retrospective reviews and limit opportunities for harmonisation.
Screen records systematically with a hierarchical list of eligibility criteria, starting from the simplest (eg, study design, then population) and use the structured data fields on registers to expedite this process.
Explanation—Similar to screening full text publications, we recommend creating a simple hierarchy of reasons for exclusion, with study design first, then variables relating to participants, interventions, and outcomes. Downloaded registry records are generally displayed in a structured format in which each column represents a data field (eg, study design and eligibility criteria), and this structure should be leveraged to expedite screening. For example, registration records can be sorted by study design (interventional v observational) in Excel and this information can be quickly verified by viewing the full text record.
Once studies are determined to be eligible, reviewers should link corresponding registration records and publications, when applicable. As such, records are not counted as two separate studies, which could introduce bias.13 Additionally, reviewers can check all identified data sources to extract the most comprehensive information. This process can lead to increased data availability, particularly for adverse events because evidence shows that these are more completely reported in ClinicalTrials.gov registration records than in linked publications.46 Registration numbers are useful for linking records and publications, and other criteria to consider are detailed in the Cochrane Handbook.13 Importantly, reviewers should consider and document how they will incorporate multiple data sources into a review (eg, registry data and published data), particularly if these sources contain conflicting information.47
Case study—Two reviewers independently screened all records using a multi-step deductive process. Firstly, we sorted the Excel spreadsheet created in step 5 by the “Study type” column (interventional v observational) and checked entries labelled “observational” against the full record at the source registry’s website. If the “observational” categorisation was verified, we entered “no” (excluded) in an adjacent column because only randomised controlled trials are eligible for TOPCHILD. Interventional trials were labelled “yes” and copied to a new spreadsheet, where we created columns for key eligibility criteria to be checked hierarchically against the source registration record and marked yes or no: randomised controlled trials, intervention start was <1 year ago, lifestyle intervention, intervention continues after pregnancy, prevention focused, and infant weight related outcome. Once any category was marked with “no”, the record was moved to the “Excluded” category and a review of the remaining criteria was unnecessary. We also added a “Reason for exclude” column with the options: ineligible study design (not a randomised controlled trials), ineligible population (child aged >12 months), ineligible intervention (no lifestyle component, antenatal intervention only, not prevention focused), or no infant weight related outcome. Additionally, we added a “Notes” column for any comments (appendix 3).
Step 7: Completing a PRISMA flow diagram
This step transparently summarises the flow of information through the searching and screening stages of a review.
Complete a PRISMA flow diagram, which includes records retrieved from trial register searches.
Explanation—The PRISMA 2020 flow diagram40 enables reviewers to report the number of records retrieved by searches (databases, registries, or other sources) and the number of studies screened, included, and excluded (with reasons).
Case study—Figure 3 is the PRISMA flow diagram for TOPCHILD. In summary, we identified 15 extra eligible trials by searching trial registers, in addition to 56 trials identified by database searches or other sources. These 15 trials included 8764 participants that met TOPCHILD eligibility criteria. Of these 15 trials, 13 were registered on ClinicalTrials.gov, one on the Chinese Clinical Trials Registry, and one on the Netherlands Trial Registry. Two trials were published but not identified in searches of bibliographical databases.
Step 8: Finalising eligible studies
Sometimes information is insufficient in a registration record to conclusively determine eligibility. For example, details of sequence generation might be lacking or participant eligibility criteria can be ambiguous.
If there are uncertainties about study eligibility, contact study registrants for clarification, if feasible.
Explanation—To facilitate communication with study investigators, registries display contact details that can be used to clarify eligibility queries. A particular issue with some prospectively registered studies is determining whether the study proceeded (eg, funding might have been withdrawn or never obtained), particularly if records are out of date and study investigators cannot be contacted. This situation can arise because some researchers think that registration of their study before submitting a grant application increases their chance of funding success. With low funding rates in many competitive schemes, this early registration can lead to many so-called zombie records of studies that did not start. In such cases, factors to consider are existence of ethical approval or related publications, whether the study is listed on an institutional or researcher webpage, and whether recruitment dates or other updates to the record were provided.
Case study—We emailed investigators from 19 trials for clarification of TOPCHILD eligibility. Most queries related to intervention timing or content, whether a child weight outcome was assessed after intervention and study design. We were informed that three trials never began due to withdrawal of funding.
Step 9: Obtaining data then synthesising as applicable
Trial registries are a useful resource for obtaining summary results.4849 Once eligible studies are determined, and a systematic review protocol has been published or is publicly available, attempts should be made to obtain data for inclusion in the review.
Attempt to obtain unpublished results data for eligible studies by checking registers and repositories and contacting study registrants.
Explanation—The process to obtain data depends on the type of review. For standard systematic reviews (retrospective reviews synthesising aggregate data), summary results might be available on registers, within other systematic reviews, or elsewhere, but often registrants need to be contacted for their data. For next generation systematic review approaches, such as individual participant data meta-analysis50 and prospective meta-analysis,20 study investigators should be invited to join a collaboration and share their raw data (individual participant data meta-analysis) or work together to harmonise outcomes to facilitate evidence synthesis (prospective meta-analysis or nested prospective meta-analysis), or both.20 Obtaining individual participant data from study investigators can be challenging, despite strong support in principle for the concept of data sharing.51 Barriers include concerns about participant consent, confidentiality, and data misuse; mechanisms to tackle these have been published.51
Explore the potential effect of publication bias, selective outcome reporting, and data availability bias when results are missing.
Explanation—Often, aggregate data or individual participant data cannot be obtained for all eligible studies and for all outcomes. If non-reported results differ systematically from those that are reported, biases can be introduced, and it is important to explore the potential effect of this difference. For instance, the identification of additional unpublished studies or data via registries will give reviewers an idea of the extent of unpublished evidence that they could be missing from their review and thus allow for an assessment of risk of publication bias. Selective outcome reporting can also be detected by comparing outcomes documented in study registration records with those that are subsequently published.48 For individual participant data meta-analyses, data availability bias should also be assessed because non-provision of data can be representative of poor study quality52 or unfavourable results.50 Extensive guidance on assessing risk of bias due to missing results is available in the Cochrane Handbook53 and will be available in the forthcoming ROB-ME (Risk Of Bias due to Missing Evidence) tool.54
Case study—The protocol for the TOPCHILD individual participant data meta-analysis is publicly available25 and prespecifies methods to investigate potential bias arising from non-reporting of results. Representatives from 47 trials have joined the TOPCHILD Collaboration to date, and we have commenced data collection by direct communication with trial representatives and review of information available on registration records.
Step 10: Reporting the search
Clear and comprehensive reporting of search details is essential for transparency and reproducibility.
Report register searches in accordance with the PRISMA 2020 statement and PRISMA-Search.
Explanation—Items 6 and 7 of PRISMA 202040 are relevant to reporting register searches. Item 6 (“Information sources”) requires specification of all registers searched and the date each register was searched. Item 7 (“Search strategy”) requires presenting the full line-by-line search strategies used for each register, including any limits or filters applied (eg, date restrictions), as well as specifying whether the search strategy was validated, peer reviewed, or adapted or re-used from a previous review. Further guidance and examples for reporting register searches are available in PRISMA-Search,33 which is a search specific extension to the PRISMA statement. PRISMA-Search also requires a description of the record deduplication process, including any software and processes used, and specifying the methods used for updating searches—for example, email alerts and re-running searches with date restrictions.
Case study—We will report TOPCHILD searches according to PRISMA 2020 and PRISMA-Search in upcoming publications. To enable reporting of each item, we have been recording relevant details, such as names of registers searched, search strategies and dates, and record management via a PRISMA flow chart.
Step 11: Updating searches
This step aims to identify any new eligible studies and to ensure reviews remain up to date.
Update searches at an appropriate frequency, depending on available resources, the research question (slow v fast-moving field) and type of review (eg, annually for standard reviews and monthly for living reviews).
Explanation—Typically, updating a register search involves repeating the initial search strategy, but restricting by registration date to avoid duplication of effort. Restricting by study start or completion dates should be avoided because a large proportion of trials are registered retrospectively (19% of trials on WHO ICTRP in 2020).55 Both ClinicalTrials.gov and ICTRP have a function to limit searches by registration date in their advanced interface, labelled “First posted” in ClinicalTrials.gov and “Date of registration” in ICTRP. Alternatively, reviewers can sort retrieved records by date of registration, and select studies that have been registered since the last search. Regardless of the method used, reviewers should be conservative with date restrictions (that is, allow some overlap in search dates) because uploads could be delayed—for example, ICTRP uploads data from some registries weekly and from others every four weeks (last upload dates are listed on the ICTRP search page). The PRISMA flow diagram for updates40 should be completed to summarise updated and cumulative searches.
Researchers can also decide to refine their search strategy for an update (beyond simply restricting dates) if they have identified shortcomings with their previous strategy—for example, eligible studies were missed. This stage is also a good opportunity to check for updates to registration records of studies already identified as eligible, such as to retrieve any recently linked publications or check if recruitment status has changed.
Case study—We updated TOPCHILD searches on 18 March 2021 (ClinicalTrials.gov) and 22 March 2021 (ICTRP), with minor refinements to resolve shortcomings in our initial March 2020 searches. We limited registration date from 1 January 2020 to the date of search (18 or 22 March 2021), allowing a few months overlap in search periods for delays in uploading records. For ClinicalTrials.gov, this date limit was applied by use of the advanced search field “First posted”. Date limits were not functioning on ICTRP, so we filtered records manually in Excel using the “Date of registration” field.
Although searching trial registers is often recommended for systematic reviews, generally little is understood about how best to conduct these searches, and their utility. Consequently, registry searches can be performed suboptimally (if at all) and can be considered an inconvenient afterthought to standard bibliographical searches. We address this gap by providing practical step-by-step guidance on how to search for trial registration records for inclusion in systematic reviews, and by highlighting the importance of these searches to mitigate bias and generate robust results based on the totality of evidence.
Although the focus of this paper is on searching for registration records, searching of other sources of unpublished studies should also be explored. The Cochrane Handbook offers helpful advice on this process, such as searching grey literature databases for reports, conference abstracts, and theses; contacting expert stakeholders; and creating a study website for outreach.13
Opportunities for searching trial registers will change in line with innovative technological developments, such as text mining, artificial intelligence, machine learning, and big data. Trial registries can diversify to increase their value and utility, particularly in relation to results reporting, data sharing, and collaboration. Since WHO introduced results reporting requirements in 2015,19 and other legislative, ethical, and regulatory levers followed, clinical trial registries have become an increasingly important hub for obtaining unpublished data, by providing direct access to summary results and providing links to preprints and publications. As of 11 October 2021, 51 372 studies registered on ClinicalTrials.gov had posted summary results.56 In future, registries could be a gateway or link users to individual participant data sharing repositories.51 In combination with results reporting, the need to publish all results of every study could be negated and the dissemination of evidence in clinical research could be transformed.
The use of trial registries can be greatly broadened in the context of emerging next generation systematic review methods. For instance, registries could have a pivotal role in enabling data sharing, which is typically a difficult and time-consuming process for those conducting individual participant data meta-analysis.50 Since 2019, registries have collected information on data sharing plans for each study, which can inform on data availability for individual participant data meta-analysis. Searching registers also allows investigators to determine if any similar studies are planned or ongoing. This information enables researchers to avoid duplication (if emerging evidence is plentiful), or, for example, to align their research with other ongoing investigations by collecting the same core outcomes at the same timepoints. These efforts can involve the formation of a prospective meta-analysis. For prospective meta-analyses, eligible studies need to be identified for inclusion before their results are known.20
In future, rather than relying solely on researchers to search for potential studies to collaborate with, registries could automatically link registrants planning similar studies at time of registration. This linkage would help to facilitate collaboration, enable prospective meta-analysis, maximise the value of research data, and avoid unnecessary duplication.20 The potential benefits of such an approach became clear during the covid-19 pandemic, which resulted in an abundance of related trials rapidly launching.57 Many of these trials have insufficient sample size alone to detect effects on important outcomes, such as mortality. Collaboration with similar trials is an efficient way to increase sample size and obtain sufficient statistical power to answer important research questions. One such example is an influential prospective meta-analysis,58 which reported that corticosteroid treatment for covid-19 is associated with reduced 28 day all cause mortality compared with usual care or placebo.
To effectively achieve these goals, funding is required for technological innovation of trial registries, including increased automation and improved functionality of searching. In addition, further mechanisms should be introduced to promote and enforce prospective study registration and reporting of results, to maximise the retrieval of information via registry searches. Such innovations would position trial registries as the first place to search to access new evidence, ahead of searches for published studies, which can have a long lag time after study completion.
We hope that this step-by-step guidance on searching trial registers will facilitate identification of registered studies for inclusion in systematic reviews and encourage future innovative use of trial registries. These improvements would help to streamline evidence synthesis, reduce bias, and enhance the reliability and validity of systematic reviews, ultimately leading to better health outcomes.
We are grateful to Lisa Askie (methods scientist, WHO) and Lotty Hooft (director of Cochrane Netherlands, member of ICTRP Advisory Group) for their input at the conceptualisation phase of this study.
Contributors: KEH and ALS conceived the study. KEH designed the surveys and led the consensus workshop; all co-authors (ACW, MJP, MW, SM, SB, PS, AGT-K, AP, and ALS) are members of the steering group, completed the author survey, and actively participated in the workshop. KEH wrote the first draft of the manuscript, and all authors contributed to and revised the manuscript. KEH is the guarantor. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.
Funding: None. KEH receives research funding support via two scholarships administered by the University of Sydney (Postgraduate Research Supplementary Scholarship in Methods Development (SC3504), and Research Training Program Stipend (SC3227)).
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: KEH receives research funding support via two scholarships administered by the University of Sydney (Postgraduate Research Supplementary Scholarship in Methods Development), and Research Training Program Stipend). ALS is chair of the TOPCHILD Collaboration, of which KEH is a chief investigator, and ACW is a member of the Advisory Group. MW has been employed as manager of the Australian New Zealand Clinical Trials Registry (ANZCTR) from October 2020 to present, which involves overseeing the operational and some research activities of the ANZCTR. ALS, AGT-K, and PS work for the ANZCTR. KEH worked at the ANZCTR from 2009-20 in a project officer/research role; she remains peripherally involved in ANZCTR related research. ALS is convenor, and KEH and ACW are associate convenors, of the Cochrane Prospective Meta-analysis Methods Group. PS and AGT-K work as information specialists for the Cochrane Breast Cancer Group; this role involves the design and execution of search strategies for systematic reviews, as well as author support throughout the review process. MJP is lead author and SM is co-author of the PRISMA 2020 statement; MJP is also a co-author of the PRISMA-Search extension; they have no commercial interest in the use of these reporting guidelines.
Patient and public involvement: This guidance was tested by two first time registry users as well as by several experienced systematic reviewers.
Provenance and peer review: Not commissioned; externally peer reviewed.