Intended for healthcare professionals

Research Methods & Reporting

The PRECIS-2 tool: designing trials that are fit for purpose

BMJ 2015; 350 doi: https://doi.org/10.1136/bmj.h2147 (Published 08 May 2015) Cite this as: BMJ 2015;350:h2147
  1. Kirsty Loudon, research assistant1,
  2. Shaun Treweek, professor of health services research1,
  3. Frank Sullivan, director of University of Toronto Practice-Based Research Network (UTOPIAN), Gordon F Cheesbrough chair in family and community medicine2,
  4. Peter Donnan, professor of epidemiology and biostatistics, director of Dundee Epidemiology & Biostatistics Unit (DEBU), co-director of Tayside Clinical Trials Unit (TCTU)3,
  5. Kevin E Thorpe, assistant professor; head of biostatistics4,
  6. Merrick Zwarenstein, director5
  1. 1Health Services Research Unit, University of Aberdeen,
  2. 2North York General Hospital, Toronto, ON M2K 1E1, Canada
  3. 3Division of Population Health Sciences, University of Dundee, Dundee DD2 4BF, UK
  4. 4Dalla Lana School of Public Health, University of Toronto; Applied Health Research Centre (AHRC), Li Ka Shing Knowledge Institute of St Michael’s, Toronto, Canada
  5. 5Centre for Studies in Family Medicine, Schulich School of Medicine & Dentistry, Western University, Western Centre for Public Health and Family Medicine, London, ON N6A 3K7, Canada
  1. Correspondence to: K Loudon kirsty.loudon{at}abdn.ac.uk
  • Accepted 12 February 2015

PRECIS is a tool to help trialists make design decisions consistent with the intended purpose of their trial. This paper gives guidance on how to use an improved, validated version, PRECIS-2, which has been developed with the help of over 80 international trialists, clinicians, and policymakers. Keeping the original simple wheel format, PRECIS-2 has nine domains—eligibility criteria, recruitment, setting, organisation, flexibility (delivery), flexibility (adherence), follow-up, primary outcome, and primary analysis—scored from 1 (very explanatory) to 5 (very pragmatic) to facilitate domain discussion and consensus. It is hoped PRECIS-2 will be valuable in supporting the explicit matching of design decisions to how the trial results are intended to be used

Summary points

  • PRECIS (2009) was a tool with 10 domains to design clinical trials on a continuum of explanatory attitude (ideal situation) to more pragmatic attitude (usual care)

  • Cited over 300 times by end of 2014, but weaknesses have been highlighted: no rating scale, problems with some domains, needing better guidance, and not validated

  • This paper presents PRECIS-2—a validated, improved version of the tool—together with guidance for how to use it

  • PRECIS-2 has nine domains including three new ones (recruitment, setting, and organisation), each scored on a 5-point Likert continuum (from 1=very explanatory “ideal conditions” to 5=very pragmatic “usual care conditions”) so that trialists, clinicians, and policymakers can more easily consider whether design decisions match their intended purpose

Randomised trials are hard work. Like much that is hard, this toil is only worth it because of the prospect of a substantial reward. For many important stakeholders (patients or others who may benefit from an intervention, funders of healthcare and of research, practitioners who may deliver clinical care and often the researchers themselves) the anticipated reward for a trial is that the results can be used to directly support decisions on delivering an intervention that will improve health outcomes for patients and the public. Key to this reward is that the trial can indeed support end user decisions in the ways intended by the trial design team. That this is not always, or even mostly, the case was highlighted nearly 50 years ago, by Schwartz and Lellouch in their paper on pragmatic and explanatory attitudes to randomised controlled trials.1

Simply put, these authors describe two purposes for randomised trials. A pragmatic randomised trial is undertaken in the “real world” and with usual care and is intended to help support a decision on whether to deliver an intervention. An explanatory randomised trial is undertaken in an idealised setting, to give the initiative under evaluation its best chance to demonstrate a beneficial effect.2 3 4 These two approaches represent different attitudes to decision making on the usefulness of interventions. Moreover, although we will refer to trials as having an explanatory or a pragmatic intention throughout this paper, we recognise that there is no simple threshold and that there are few purely explanatory or pragmatic trials; rather than dichotomy there is a continuum.

There is widespread concern that explanatory randomised trials can be poor predictors of the real world effectiveness of the interventions they test.4 5 6 7 Schwartz and Lellouch suggested that trialists often wish to inform a real world decision, but, almost by default, tend to design trials that are more explanatory than they should be given the designers’ intention. The original PRECIS (PRagmatic Explanatory Continuum Indicator Summary) tool was developed in 2005–08 by 25 international trialists and methodologists8 to help trialists work through their design decisions so that they might avoid designing a trial that did not match their own intentions for the trial. There is demand for such tools; PRECIS has been cited over 300 times since 2009, it is one of eight “useful papers” for trialists listed on the UK National Institute for Health Research Randomised Trials Methods website (www.nets.nihr.ac.uk/resources/trials-coordination), and many investigators have used PRECIS prospectively and retrospectively to consider trial design.9 While acknowledging the usefulness of PRECIS, these latter authors have identified weaknesses, including unclear face validity and inter-rater reliability, the lack of a scoring system, redundancy in some PRECIS domains, and the need for more guidance on how to use the tool.

This paper describes a new version of PRECIS, PRECIS-2, which aims to address these (and other) weaknesses while keeping the strengths of the original tool, especially its simple format. The paper also provides guidance on how to use the tool. As with the original 2009 PRECIS tool, we intend PRECIS-2 to be used at the design stage of a trial in order to help trialists make the purpose of their trial explicit and to ensure that their design choices are concordant with their intended purpose. We believe that it may have a role in other situations such as critical appraisal and systematic review as well as funding, ethics, and publication decisions on randomised trials.

Aim

To present the PRECIS-2 tool and provide guidance on how to use it.

Methods

This work is part of a larger project concerned with ensuring that trials are designed so that their results are relevant to, and used by, patients, clinicians, and policy decision makers.9 This paper presents the PRECIS-2 tool and gives guidance on how to use it but does not present in full the work that led to it, which will be published separately (see Loudon et al9 or contact the authors for more information).

Briefly, we ran a two-round Delphi communication with contact authors who had cited PRECIS but who had neither been involved in the original development of PRECIS nor evaluated the utility of PRECIS. We discussed how to improve PRECIS, focusing on issues raised by PRECIS users9 and brainstorming with trialists in Dundee. The Delphi results were the basis for a brainstorming meeting in Toronto involving some of the original developers of PRECIS plus some of those who had undertaken methodological work using PRECIS, together with clinicians and policymakers. We then user-tested candidate PRECIS-2 models, with 19 international trialists, on a one-to-one basis (in person or via Skype). PRECIS-2 was modified in response to user testing.

The version of PRECIS-2 that finally emerged was used in validity and reliability testing by 19 raters. These 19 raters had responded to an invitation we sent to 33 individuals, identified from four sources: the Delphi, user testing, the Toronto brainstorming meeting, and through our personal networks of experienced trialists. From these groups, 19 accepted the invitation (six from the Delphi, two from user testing, three from the Toronto meeting, and eight from our networks). The raters were asked to score a varied sample of 15 trial protocols. These protocols were purposefully selected to range from highly pragmatic to highly explanatory, including both drug and non-drug trials. This testing gave support for the proposed version of PRECIS 2, which is the version discussed in this paper.

Focus of PRECIS-2

The PRECIS-2 tool focuses on trial design choices which determine the applicability of a trial. Applicability (the ability for a trial result to be applied or used in a particular situation) is the outcome of these choices, which affect the ease with which the trial results can be applied to and by the usual community of users of the intervention in the settings in which the trial designers envisioned it being used. This may, of course, vary for different readers, and thus applicability has both “local” and more distant meanings. The aim of a highly pragmatic trial would be to maximise applicability of the intervention to usual care across a range of local and distant settings. The aim of a highly explanatory trial would be to maximise the intervention’s chance of demonstrating an effect through the expected mechanism, with little attention paid to the issue of whether this outcome would be achieved under real world conditions, neither locally nor in more distant settings. Applicability is a continuum, and various trial design choices can make a trial more, or less, applicable; in other words more or less pragmatic.

In general, pragmatic trials are focused on care in the most common settings and are less commonly focused on highly specialised care settings. However, we believe that, even in specialised settings, a trial can be more or less applicable depending on the degree to which it provides care with resources and clinicians standard in that setting, to patients who are typical of that setting. Note that pragmatism constitutes only a close matching between the care delivered in the setting in which the trial was conducted and the care delivered in the setting to which its results are applied. This close matching provides directly applicable information and thus helps decision makers choose whether to implement the intervention tested in the trial.

What is new in PRECIS-2 compared with PRECIS

  • A scale for all domains which can be scored from 1 (very explanatory) to 5 (very pragmatic)

  • Eligibility criteria has been placed on the 12 o’clock spoke

  • Clockwise use of PRECIS-2 with logical ordering of adjacent domains

  • The participant eligibility criterion (which might formerly have been thought of as including issues of setting and issues of multicentre trials) has been separated into “Eligibility” and “Setting”

  • New domains of “Recruitment” and “Organization”,

  • Domain name changes:

    • - From “Practitioner adherence” to “Flexibility: delivery”

    • - From “Participant compliance” to “Flexibility: adherence”

    • - From “Follow up intensity” to “Follow up”

    • - From “Outcome” to “Primary outcome”

  • Removal of the comparison intervention domains of “Practitioner expertise” and “Flexibility of the comparison intervention”

  • Domain name labels on the PRECIS-2 wheel now come with short explanations

  • Improved guidance on how to use PRECIS-2

How to use PRECIS-2

The tool has been developed to be used by a multidisciplinary team throughout the process of designing their trial. This team will usually include expertise in the particular intervention and health service setting in which the intervention will be used, should it be shown to be effective, as well as statistical expertise.

There are four steps to using PRECIS-2, which may be iterative if there remains a gap between the intended purpose of the trial and the initial design choices, which will be revealed as the developing design is viewed through the prism of the PRECIS-2 tool.

Step 1: What design approach are you taking?

The first step is to be clear about your intention or attitude to the trial you are designing. Are you

  • Aiming to take an explanatory approach to answer the question, “Can this intervention work under ideal conditions?”

  • Aiming to take a pragmatic approach to answer the question, “Does this intervention work under usual conditions?”

Both approaches to trial design have their place, but trialists should be clear which they would prefer to emphasise for the trial they are designing. As mentioned previously, trialists have often emphasised the first approach by default rather than as a deliberate decision.1

Step 2: Consider your trial design choices for each of the PRECIS-2 domains

This step is explained in more detail for each domain later on. Imagining first the most pragmatic and the most explanatory design choice you could make for each domain may help you in deciding how you wish to implement that domain choice, and thus how pragmatic or explanatory you want your actual decision on that domain to be.

Before discussing the domains further, we should clarify what is meant by “participant.” Participants include patients or other individual recipients of an intervention, or providers of the intervention, or both. This may include individual participants or one or more levels of clusters. For example, in a trial of a continuing education intervention, participants may be health professionals and trained instructors, and the trial may be randomised into clusters at the level of the instructor. In some cluster trials, if randomisation occurs at two levels (such as organisation and individual participants), scoring of the domains may need to be considered separately.

Step 3: Score 1 to 5 for the choices made in Step 2 and mark on the PRECIS-2 wheel

If there is uncertainty over how explanatory or pragmatic your proposed trial design element is for a particular domain, then we suggest the score for this domain should be left blank; this will highlight the uncertainty and encourage discussion. Domains which have greater scoring variation by raters probably require further discussion and reiteration of steps 1 to 4 to clarify the design of this domain so there is greater agreement.

Having considered your design choices in Step 2, position your choice for each domain on the PRECIS-2 wheel to record how pragmatic or explanatory these choices are for each domain. Scoring each domain can be done using a 5-point Likert scale:

  • 1. Very explanatory

  • 2. Rather explanatory

  • 3. Equally pragmatic and explanatory

  • 4. Rather pragmatic

  • 5. Very pragmatic.

A toolkit with guidance and practical help is available to download from www.precis-2.org. The appendix on this website provides four examples of comparative effectiveness trials with explanations of our scoring decisions for the PRECIS-2 domains: a cluster randomised trial in rural India to improve maternal and newborn health,10 a pharmacological primary care trial in UK to control asthma,11 an international music therapy trial to treat preschool children with autism,12 and an international surgical trial for coronary angiography.13 None of these trials, as far as we are aware, was designed using the PRECIS tool, and they do not refer to the Schwartz and Lellouch paper,1 so this is a post facto assessment, but nevertheless gives the reader some idea of how each domain may vary from pragmatic to explanatory.

Step 4: Review the PRECIS-2 wheel

Review your set of nine domain design choices (Step 2) on the PRECIS-2 wheel (figure) to see whether they will produce a trial that will support the overall aim you identified for your trial in Step 1. Go back to Step 2 and modify your design choices if required.

Figure1

The PRagmatic-Explanatory Continuum Indicator Summary 2 (PRECIS-2) wheel.

The nine domains of PRECIS-2

PRECIS-2 has nine domains, each of which is intended to help trialists think about the consequences of that design decision for applicability of the results of their trial. The PRECIS-2 tool is focused exclusively on the issue of applicability, and readers should use other tools to assess the internal validity or any other aspect of their design choices.

Each domain encourages trialists to think about their trial and the recipients in the usual care situation in which their results might be applied if the intervention proves beneficial. If trialists are aiming for high applicability (that is, a pragmatic approach to design decisions), then we would expect the match between trial and usual care to be very good. If trialists are aiming to give the intervention its best shot at showing a benefit (that is, an explanatory approach), then the match between trial and usual care may not be close. The point of PRECIS-2 is not to always to produce a predominantly pragmatic design (close match between trial and usual care) but to make trialists think about the aim of their trial and decide whether the design is appropriate given the aim of their trial.

In pragmatic trials the comparator is usual care. In explanatory trials it may not be. In PRECIS-2 the domains are based on the assumption that the trial is two armed, one of which is usual care with no changes. If usual care is not the comparator, or there are multiple intervention arms that are very different from each other, then the arms will need to be scored separately; there may be differences in Organisation and Flexibility (delivery) and Flexibility (adherence).

The nine PRECIS-2 domains are:

  • 1. Eligibility—To what extent are the participants in the trial similar to those who would receive this intervention if it was part of usual care?

  • 2. Recruitment—How much extra effort is made to recruit participants over and above what would be used in the usual care setting to engage with patients?

  • 3. Setting—How different are the settings of the trial from the usual care setting?

  • 4. Organisation—How different are the resources, provider expertise, and the organisation of care delivery in the intervention arm of the trial from those available in usual care?

  • 5. Flexibility (delivery)—How different is the flexibility in how the intervention is delivered and the flexibility anticipated in usual care?

  • 6. Flexibility (adherence)—How different is the flexibility in how participants are monitored and encouraged to adhere to the intervention from the flexibility anticipated in usual care?

  • 7. Follow-up—How different is the intensity of measurement and follow-up of participants in the trial from the typical follow-up in usual care?

  • 8. Primary outcome—To what extent is the trial’s primary outcome directly relevant to participants?

  • 9. Primary analysis—To what extent are all data included in the analysis of the primary outcome?

The domains in detail

1. Eligibility criteria—Who is selected to participate in the trial?

A highly pragmatic approach to eligibility criteria would be to include in the trial anyone with the condition of interest who is likely to be a candidate for the intervention if it was being provided in usual care for this condition. So, if in usual care the intervention would likely be used for children, elderly people, people with a range of comorbidities, and people with a probable diagnosis rather than, say, a laboratory confirmed diagnosis, then all of these groups should be in the trial too. If people aged over 80 or with multiple morbidities would not receive the intervention in usual care because we already know that this intervention does more harm than good in elderly or multi-morbid patients, then a pragmatic design choice would be to exclude them from the trial. A trial with eligibility design choices like this would score at or close to 5 on the PRECIS-2 scale.

As the level of similarity between the people in the trial and those in usual care decreases, then so would the PRECIS-2 score. Many things could reduce the score, but some examples include

  • Excluding people not known or shown to be highly adherent to the intervention.

  • Excluding people using tests or measures that are not used or available in usual care.

  • Excluding people not expected to be highly responsive to the intervention.

  • Excluding people who would be likely to receive the intervention in usual care but inclusion in the trial is considered too difficult because of challenges unrelated to the delivery of the intervention. Children, people over the age of 65, and pregnant women are good examples of people commonly excluded because of anticipated difficulties of gaining ethical approval rather than whether they would be likely to receive an intervention in usual care.

  • Excluding people whose adherence and follow-up may pose difficulties, whether because of social disadvantage, personal circumstances, or illness (such as homelessness, substance misuse, mental health problems, or poor motivation).

  • Excluding people dependent on help for activities of daily living, unless they would usually not be eligible for intervention.

Trialists may make their trials less like usual care by having tight inclusion criteria and broad exclusion criteria. For trialists taking an explanatory approach this is often appropriate, but this is unlikely to be the case for trialists aiming to design a trial that is directly applicable in a particular usual care setting, let alone one intended to have wide generalisability across multiple care settings. Under certain circumstances, a trial design may be pragmatic (and the results highly applicable to the usual care of the participants under study) even with narrow eligibility criteria if the particular narrow criteria are similar to those routinely used for those participants in their usual care setting. Some rare disease trials may fall into this category. Trials in myotonic dystrophy type 1, for example, may require a genetic confirmation of the diagnosis, but since most patients with myotonic dystrophy type 1 receive their diagnosis through genetic confirmation this inclusion criterion simply reflects what is usual practice. Finally, exclusion criteria for known safety issues would not generally reduce the PRECIS-2 score since such individuals would not be expected to get the intervention in usual care.

Eligibility criteria, example 1: Early treatment with prednisolone or acyclovir in Bell’s palsy*

  • Inclusion criteria—Patients with confirmed diagnosis: ≥16 years of age with unilateral facial nerve weakness of no identifiable cause who presented to primary care or an emergency department and could be referred to a collaborating otorhinolaryngologist <72 hours after the onset of symptoms.

  • Exclusion criteria—Pregnancy, breast feeding, uncontrolled diabetes, peptic ulcer disease, suppurative otitis media, herpes zoster, multiple sclerosis, systematic infection, sarcoidosis and other rare conditions, and an inability to provide informed consent.

  • Extra test—Randomised controlled trial of Bell’s palsy treatment required senior otorhinolaryngologist in hospitals to confirm a patient’s eligibility to participate. Bell’s palsy is usually diagnosed by a general practitioner in primary care.

  • Suggested PRECIS score—2, rather explanatory.

  • *Sullivan FM, Swan IR, Donnan PT, et al. A randomised controlled trial of the use of aciclovir and/or prednisolone for the early treatment of Bell’s palsy: the BELLS study. Health Technol Assess 2009;13:iii-iv, ix-xi, 1-130.

Eligibility criteria, example 2: Practical approach to lung health in South Africa*

  • Inclusion criteria—All patients aged ≥15 with suspected asthma were included in the study, based on a crude clinical diagnosis of asthma, as the standard respiratory testing machinery was not available in this rural setting in South Africa.

  • Suggested PRECIS score—5, very pragmatic.

  • *Fairall LR, Zwarenstein M, Bateman ED, et al. Effect of educational outreach to nurses on tuberculosis case detection and primary care of respiratory illness: pragmatic cluster randomised controlled trial. BMJ 2005;331:750-4.

2. Recruitment—How are participants recruited into the trial?

This domain has been included to encourage trialists to consider how and where they will recruit people to their trial. (For clarity, we have described only trials involving patients, but, as stated above, participants could be groups of practitioners or health system where the goal is to improve some aspect of care at the intervention unit of clinics, wards, hospitals, or geographic area.)

The most extreme pragmatic approach to recruitment would be to do this in usual care so that only the people who attend a clinic with the condition of interest are recruited after they present on their own behalf without any overt recruitment effort. A highly pragmatic approach to trial design would also recruit from more than one clinic as an explicit way of increasing applicability of the trial results. Recruiting through usual appointments at a diverse range of clinics is likely to score at or close to 5 on the PRECIS-2 scale.

If the above is not feasible, at least as the sole recruitment route, trialists will need to add other recruitment strategies. If extra resources will be required to recruit people to the trial, trialists need to consider if this affects applicability, especially if participants are recruited from settings other than the one in which the results are likely to be applied. For example, recruiting in secondary care for an intervention that will mainly be used in primary care will ensure that the recruited patients are different in severity and treatment than the patients with this same condition in primary care; and so this will make the trial more explanatory and reduce the PRECIS 2 score. Additional recruitment approaches that could potentially reduce the PRECIS-2 score include

  • Searching medical record systems for eligible participants and then mailing invitation letters. Depending on local resources, such as electronic medical records and comprehensive local disease registers, this approach may be easily achievable in usual care and so would not reduce the PRECIS-2 score.

  • Media advertising campaigns such as television and radio advertisements, newspaper advertisements, flyers, websites, press releases, letters to the editor, interviews with the chief investigator on television or radio. The issue to consider here is whether these campaigns are being used simply to speed up recruitment, or because they are the only way to reach a dispersed population. In the latter case, if such campaigns would not be available in usual care then this trial recruitment route is more explanatory.

  • Incentives such as cash payment, vouchers, travel costs, or childcare. Some of these incentives may be better considered as part of the intervention (for example, a lifestyle change intervention aimed at new mothers may only be effective because childcare was also provided and women could therefore attend sessions).

Recruitment, example: Leukotriene antagonists for asthma treatment*

  • Initially extra resources were used to recruit patients at 53 primary care practices. Patients were recruited via a postal questionnaire to identify symptoms and trial eligibility, not just to invite to participate. This would push the recruitment path of this domain towards the explanatory end. By using this method of recruitment, which requires administration not normally present in primary care, it is possible that responders may be healthier than those at the clinic being invited to the trial and also more highly motivated and compliant as they have come through a different route than those invited during a clinic attendance.

  • In this trial, recruitment was inadequate using a postal questionnaire, so participants were then recruited through clinic attendances changing the recruitment towards a more pragmatic trial design, creating results which are more applicable to users of the results in a primary care setting.

  • Suggested PRECIS score—2, rather explanatory; but, as trial continued, a PRECIS score of 3 (equally pragmatic and explanatory) since trial now more a mix of recruitment methods, some of which are feasible in usual care.

  • *Price D, Musgrave SD, Shepstone L, et al. Leukotriene antagonists as first-line or add-on asthma-controller therapy. N Engl J Med 2011;364:1695-707.

3. Setting—Where is the trial being done?

This domain has been included to encourage trialists to explicitly consider the match between the setting of their trial and the setting where their results are likely to be applied. While there are no accepted guidelines for how to assess the impact of setting on applicability,5 there are several characteristics of the setting that will almost always need to be considered, including geography, healthcare system, country, and the socioeconomic and ethnic mix of the population.4 14 15 All of these might affect the applicability of results from a particular trial.

The most extremely pragmatic approach to setting would be to do the trial in an identical setting to which you intend the results to be applied. Such a trial is likely to score at or close to 5 on PRECIS-2.

Even settings that seem rather restrictive could still be a highly pragmatic design choice if this setting is the usual care setting where patients are treated for the particular health condition. For example, if a trial was carried out in the most specialist intensive care units in the country and the intention of the trial was to support decision making in these highly specialised units, then the design choice with regard to setting is still pragmatic. Conversely, if the trial results were intended to be useful for all intensive care units, but the trial included only highly specialised units, then the trial is now taking a more explanatory approach to the choice of setting that goes against the intention of the trial.

Determining how pragmatic or explanatory a setting is can be problematic when there is a proposed change in the setting for care delivery. For instance, if a trial is carried out in secondary care and the intention is that, if the change in setting is successful, all future care will be delivered in this setting, then this new setting will be become the usual care setting. In this case the setting domain in the trial would be explanatory compared with current usual care, but trialists can note that the difference is an explicit feature of the intervention.

Approaches to setting that are likely to reduce the PRECIS-2 score include

  • Selecting participating centres from among only specialist or academic centres when the trial is intended to be applicable to all types of practice or clinic treating the condition of interest.

  • Running the trial in a single centre. It is rare that a trial is done with the intention that its results should apply to only that single centre. Including several centres makes it easier to claim that the trial setting matches that to which the results will be applied.

Setting, example 1: Manual physical therapy versus corticosteroid injection to treat shoulder impingement*

  • Single centre and specialised centre (Madigan Army Medical Center, USA), unlikely to be the usual setting for most individuals receiving physiotherapy for shoulder impingement.

  • Suggested PRECIS score—2, rather explanatory, dependent on how different raters think the treatment centre is similar from usual setting in the country they live in.

  • *Rhon DI, Boyles RE, Cleland JA, et al. A manual physical therapy approach versus subacromial corticosteroid injection for treatment of shoulder impingement syndrome: a protocol for a randomised clinical trial. BMJ Open 2011;1:e000137.

Setting, example 2: Ibuprofen, paracetamol, and steam for patients with respiratory tract infections in primary care*

  • Identical setting to usual care setting: primary care, where patients usually go for advice and treatment of the common cold in the UK.

  • Multi-centre: 25 practices in UK.

  • Suggested PRECIS score—5, very pragmatic

  • *Little P, Moore M, Kelly J, et al. Ibuprofen, paracetamol, and steam for patients with respiratory tract infections in primary care: pragmatic randomised factorial trial. BMJ 2013;347:f6041.

Trialists working on trials that cross jurisdictions (such as international trials) need to use their judgment with regard to the impact these different settings have on the overall trial applicability. If the trial is delivered in the same setting as usual care in each jurisdiction then the trial is likely to score as pragmatic on PRECIS-2, even if the settings are different in each jurisdiction. In this case, the trial randomisation would be stratified by jurisdiction.

4. Organisation—What expertise and resources are needed to deliver the intervention?

This domain has been included to encourage trialists to consider the match between how care is organised and delivered in the trial and how the intervention would be made available to patients in usual care. As there have been acknowledged difficulties in implementing results post-trial (even in settings where the trial was conducted16), we wanted trialists to consider organising the delivery of their trial intervention with an awareness of how easy it would be to implement the intervention post-trial .

A highly pragmatic design would aim to slot the intervention into the usual organisation of care for the condition of interest, making use of no more than the existing healthcare staff and resources in that setting. A design like this is likely to score at or close to 5 on PRECIS-2.

Sometimes a change to how care is organised is itself the intervention being evaluated (see, for example, Zwarenstein et al’s trial of directly observed treatment (DOTS) versus self-supervised treatment for tuberculosis17), and the difference between the trial and usual care is clear. In this case the organisation domain would be explanatory compared with current usual care, but trialists can note that the difference is an explicit feature of the intervention.

The PRECIS-2 organisation score will be reduced to the degree trialists make changes in how care is delivered compared with the usual recipients and care delivery. For example, the trial team may, in the intervention arm, provide additional training to staff or provide additional staff so that the intervention can be delivered. Patients who would normally be seen in primary care may be seen by specialists in secondary care instead, or they may receive additional expensive diagnostic procedures unavailable in usual care. The success or otherwise of the trial then depends on resources unavailable in usual care, reducing applicability. If these additional resources are essential to the effectiveness of the intervention they should be considered to be part of the intervention and be specified in the part of the protocol describing the intervention. In that case these resources should not be counted when this element of the trial is judged.

Organisational approaches that are likely to reduce the PRECIS-2 score include the following (but only when they are not incorporated and described as necessary elements of the intervention):

  • Increasing the number of healthcare or other professionals available to deliver the intervention over and above the levels available in usual care.

  • Providing significant levels of additional training to increase the expertise of healthcare professionals.

  • Requiring healthcare professionals to have some minimum level of experience, defined by length of time, in working with the intervention that is greater than would be the case in usual care.

  • Requiring healthcare professionals to have a specialty certification that would not be considered essential to deliver the intervention in usual care.

  • Increasing the resources (such as facilities, diagnostic equipment, consumables) available to deliver the intervention, measure outcomes, or do follow-up over and above what would be available in usual care.

Organisation, example 1: Establishment of Acute Respiratory Distress Syndrome (ARDS) Network in 1994*

  • Multicentre clinical trials of ARDS treatments, but there was difficulty translating results from a trial involving low tidal volume (Vt) into usual clinical practice

  • Ten academic centres with 75 intensive care units

  • Extra staff, and very labour intensive

  • Used additional equipment beyond usual care, none of which was planned for at the trial design stage

  • Suggested PRECIS score—1, very explanatory

  • *Ventilation with lower tidal volumes as compared with traditional tidal volumes for acute lung injury and the acute respiratory distress syndrome. The Acute Respiratory Distress Syndrome Network. N Engl J Med 2000;342:1301-8.

Organisation, example 2: Early lens extraction and intraocular lens implantation to treat glaucoma*

  • Identical organisation to usual care, usual clinic

  • Same number of staff

  • Usual experience—fully qualified ophthalmologists who have completed general and specialist training (in ophthalmology and glaucoma, respectively) and able to perform lens extraction procedures

  • No additional training

  • No additional resources

  • Suggested PRECIS score—5, very pragmatic

  • *Azuara-Blanco A, Burr JM, Cochran C, et al. The effectiveness of early lens extraction with intraocular lens implantation for the treatment of primary angle-closure glaucoma (EAGLE): study protocol for a randomized controlled trial. Trials 2011;12:133.

5. Flexibility (delivery)—How should the intervention be delivered?

In PRECIS-2, trialists wishing to conduct more pragmatic trials are encouraged to think about how the intervention will be implemented post-trial in the setting where the results are expected to be applied. The most pragmatic design approach to delivery flexibility would leave the details of how to implement the intervention up to providers, in other words, what happens in usual care. Thus, the methodology of how to deliver an intervention is not rigidly prescriptive in the protocol. For example, the details of how to perform a surgical procedure could be left entirely to the surgeon, or how to deliver an educational programme is left to the discretion of the educator. Additionally, a pragmatic approach would not dictate which other interventions were permitted, or how to deliver them, which again is the situation in usual practice. This sort of delivery flexibility would score at or close to 5 on PRECIS-2. Flexibility (delivery) applies to all intervention arms and the comparator (control) arm if usual care is not the comparator, and if necessary each of these arms will need to be scored separately. If usual care is the intended comparator but any elements relating to delivery are changed it ceases to be usual care and needs to be scored.

As delivery flexibility is reduced, the trial moves towards a more explanatory approach in this domain. Approaches to delivery flexibility that are likely to reduce the PRECIS-2 score include

  • A highly specified, protocol driven intervention. For example, specific direction is given for how to the administer the intervention (such as dose, dosing schedule, surgical procedures, educational material, and delivery)

  • Having measures in place to monitor the compliance of those delivering the intervention (such as doctors) with the protocol and measures (up to and including exclusion) to address poor compliance.

  • The timing of intervention delivery is tightly defined and designed to maximise the intervention effect.

  • Providers undertake additional interventions that would not occur in usual care

  • Restrictions are placed on the number and types of co-interventions, particularly if excluded co-interventions would dilute any intervention effect

  • There is specific direction for applying permitted co-interventions

  • There are specific directions for managing complications or side effects of the intervention.

Flexibility (delivery), example 1: Cognitive behavioural therapy (CBT) for depression*

  • Measures in place to monitor and improve compliance: In this trial therapy was delivered by selected experts who received regular training and supervision (but were independently assessed as representative of NHS staff). There was also independent assessment to confirm minimum standard of delivery of CBT but no measures in place to improve compliance as this was self regulated by therapists.

  • No specific protocol for timing or co-interventions

  • Suggested PRECIS score—5, very pragmatic

  • *Wiles N, Thomas L, Abel A, et al. Cognitive behavioural therapy as an adjunct to pharmacotherapy for primary care based patients with treatment resistant depression: results of the CoBalT randomised controlled trial. Lancet 2013;381:375-84.

Flexibility (delivery), example 2: Elective caesarean section syntocinon infusion trial*

  • Protocol driven—Much detail given, with protocol violations recorded in self reported case form. Investigators accept this may occur due to clinical needs (such as anaesthesia).

  • Co-interventions—Specific direction

  • Complications—Specific directions for managing complications or side effects

  • Improving adherence—No measures in place

  • Suggested PRECIS score—2, rather explanatory

  • *Murphy DJ, Carey M, Montgomery AA, et al. Study protocol. ECSSIT—Elective Caesarean Section Syntocinon Infusion Trial. A multi-centre randomised controlled trial of oxytocin (Syntocinon) 5 IU bolus and placebo infusion versus oxytocin 5 IU bolus and 40 IU infusion for the control of blood loss at elective caesarean section. BMC Pregnancy Childbirth 2009;9:36.

6. Flexibility (adherence)—What measures are in place to ensure participants adhere to the intervention?

It is important that trialists focused on applicability consider how probable it is that there will be uptake of the intervention being tested and how recipients will engage with it in the setting in which it will be later applied. A highly pragmatic design approach would allow for full flexibility in how end user recipients engage with the intervention. In usual care, health professionals encourage patients to take medication or follow therapy as best they can, and such encouragement would not count against a pragmatic design; if it also happens in usual care, allowing it in the trial is a pragmatic design decision. A trial with no special measures to enforce engagement or compliance would score at or close to 5 on PRECIS-2. On the other hand, a trial protocol that lays out methods to monitor and ensure patient compliance would score at or close to 1 on PRECIS-2. Flexibility (adherence) applies to all intervention arms and the comparator (control) arm if usual care if not the comparator, and if necessary each of these arms will need to be scored separately. If usual care is the intended comparator but any elements relating to adherence are changed it ceases to be usual care and needs to be scored.

It is possible that improving recipients’ ability to take medication or follow therapy is the intervention under evaluation, a situation similar to that already described for the Setting and Organisation domains. In this case, the Flexibility (adherence) domain would be explanatory compared with current usual care, but trialists can note that the difference is an explicit feature of the intervention.

As flexibility for recipients is reduced, the trial moves towards a more explanatory approach on this domain. Approaches to recipient flexibility that are likely to reduce the PRECIS-2 score include

  • Having a trial pre-screening stage where patients are evaluated for adherence with the intervention (such as dose, dosing schedule, attendance at therapy sessions). Patients judged to not be adherent are excluded.

  • Withdrawing patients from the trial if their adherence with the intervention drops below some specified level. For example, a trial might withdraw patients if they failed to attend two consecutive sessions of cognitive behavioural therapy or failed to take more than 90% of their medication.

  • Having measures in place to monitor patient adherence with the protocol and measures (up to and including withdrawal) to address poor adherence. Measures to increase patient adherence should this fall below some specified level could, for example, include scheduling an extra discussion with a research nurse about why following the trial instructions is important.

Flexibility (adherence), example: Music therapy to support communication in autistic children*

  • The sessions were all individual based on interaction with child and allowed for range of responses to the intervention

  • Suggested PRECIS score—5, very pragmatic

  • *Geretsegger M, Holck U, Gold C. Randomised controlled trial of improvisational music therapy’s effectiveness for children with autism spectrum disorders (TIME-A): study protocol. BMC Pediatr 2012;12:2.

In some trials—such as surgical trials where patients are being operated on or intensive care unit trials where patients are being given intravenous drugs—this domain is not applicable as there is no adherence issue after consent has been given, so we suggest that the PRECIS-2 score should be left blank.

7. Follow-up—How closely are participants followed up?

This domain encourages trialists to think about follow-up with regard to the setting where the results will be applied. The most pragmatic position with regard to follow-up would be to have no more follow-up of recipients than would be the case in usual care. Indeed, the most extreme position is to have no follow-up contact at all with recipients and to obtain outcome data by other means (such as electronic medical records or other usual data to measure mortality or hospital admissions). Trials that have no more follow-up than is normal in usual care and have minimal additional data collection (or use data obtained from administrative or clinical record systems without direct contact with the participant) would be likely to score at or close to 5 on PRECIS-2.

As follow-up becomes more intense, the trial becomes more explanatory and the PRECIS-2 score will decrease. Follow-up approaches that are likely to reduce the PRECIS-2 score include

  • Follow-up visits that are more frequent than would typically occur under usual care.

  • Unscheduled follow-up visits are triggered by a primary outcome event.

  • Unscheduled follow-up visits are triggered by an intervening event that is likely to lead to the primary outcome event.

  • Patients are contacted if they fail to keep trial appointments. This would not reduce the PRECIS-2 score if this was also done in usual care.

  • More extensive data are collected, particularly intervention related data, than would be typical outside the trial

  • Visits are longer than usual care and involve additional or different staff.

Often the required trial outcomes may be obtained only through contact with the recipients. Even in the “no follow-up” approach, assessment of outcomes may be achieved with a single follow-up at the end of the study. The end of study would need to be defined so that there is sufficient time for the desired study outcomes (see Primary outcome domain below) to be observed. When the follow-up is done in this way, it is unlikely to have an impact on engagement or response to the treatment, and so such a follow-up approach would be viewed as pragmatic and rated at or close to 5.

It is often the case that explanatory trials are interested in the effect of an intervention only during a brief intervention period or shortly after. On the other hand, trials taking a pragmatic approach to design may follow recipients well beyond the intervention period. This longer period of follow-up may require more contacts than would be the case in usual care, but this need not be inconsistent with a pragmatic approach if it does not result in care management that differs from the usual conditions. It is, however, important to consider the burden of follow-up for participants.

Follow-up, example 1: Perioperative β blockade for patients undergoing infrarenal vascular surgery*

  • Clinical follow-up until patient left hospital (discharge or death) or until 30 days after surgery, whichever was the longer, so more than usual care.

  • Monitoring intensity involved more extensive data collection than usual:

    • Pre-operation—three-lead electrocardiogram (ECG) Holter monitor (Flashcard with 2×48 hour recording) set up on each patient and maintained for 72 hours.

    • Troponin values at 1, 3, and 7 days after surgery (more usual for only 1 and 3 days after surgery)

    • ECG after randomisation and at 7 and 30 days after surgery.

  • Unscheduled follow-up visits triggered by primary outcome: a cardiovascular event (such as angina, myocardial infarction, stroke)

  • Suggested PRECIS score—1, very explanatory

  • *Brady AR, Gibbs JSR, Greenhalgh RM, et al. Perioperative beta-blockade (POBBLE) for patients undergoing infrarenal vascular surgery: results of a randomized double-blind controlled trial. J Vasc Surg 2005;41:602-9.

Follow-up, example 2: Self management course for chronic musculoskeletal pain*

  • Participants followed up two times over a year (at 6 and 12 months)—not intense. This is quite close to usual care for pain related disability.

  • Follow-up for the primary outcome requires self completion of a postal validated questionnaire chronic pain grade, which contains three questions to determine the primary outcome “pain related disability” (and secondary outcome “pain intensity”).

  • No additional follow up visits—All additional data collection for secondary outcomes use six additional short postal questionnaires sent at 6 and 12 months: EQ-5D (health utility), Pain Self-Efficacy Questionnaire, Hospital Anxiety and Depression Scale (mood), Chronic Pain Acceptance Questionnaire (coping), Health Education Impact Questionnaire (social integration subscale), Census global health question (general health).

  • Additional data from GP electronic records

  • Suggested PRECIS score—4, rather pragmatic

  • *Carnes D, Taylor SJ, Homer K, et al. Effectiveness and cost-effectiveness of a novel, group self-management course for adults with chronic musculoskeletal pain: study protocol for a multicentre, randomised controlled trial (COPERS). BMJ Open 2013;3:e002492.

8. Primary outcome—How relevant is it to participants?

The choice of primary outcome is a crucial trial design decision, and a pragmatic approach would be to select an outcome that is of obvious importance from the patient’s perspective. Post-trial, an outcome selected using a pragmatic approach would also be relevant to commissioners of care, the people who decide whether to implement the intervention on the basis of its results. For example, an intervention that aims to reduce falls in elderly people living independently in the community should have as its primary outcome the number of falls in the elderly living independently in the community. This outcome has meaning to patients, their relatives and friends, healthcare professionals, and policymakers. Measures of, say, bone density, muscle strength, or functional ability are distant from the key question of whether the intervention prevents elderly people falling in their own homes. Trials that choose outcomes of obvious importance to patients, and measure them in a way that is the same or similar to the way they are measured in usual care would be likely to score at or close to 5 on PRECIS-2.

As the primary outcome becomes less recognisably important to patients, or is assessed on criteria seldom used in usual care, the trial becomes more explanatory and the PRECIS-2 score will decrease. Approaches to the primary outcomes that are likely to reduce the PRECIS-2 score include

  • Choosing a surrogate outcome (such as blood test) that the intervention is expected to have a direct effect on, or the use of safety climate surveys rather than medical error events or near miss events.

  • Using a composite primary outcome in which some of the elements are less important to patients or participants than others (such as incidence of death, stroke, myocardial infarction, or renal dysfunction at 30 days).

  • Having central adjudication of the outcome or using an assessment that needs special training or tests not normally used in usual care.

  • Choosing an outcome that is important but mainly to providers, typically because it is a physiological outcome considered useful in treatment planning and monitoring.

  • Measuring an outcome that is important but at a time that is earlier than would be normal in usual care. For example, stroke could be a primary outcome in trials making explanatory or pragmatic design choices. However, time horizons may vary from short term after a one-time intervention (more explanatory) to long term (more pragmatic).

Primary outcome, example: Early treatment with prednisolone or acyclovir in Bell’s palsy*

  • Primary outcome—Recovery of facial function as rated on the House-Brackmann scale.

  • Test not routinely used in primary care and requires training. It is, however, an easy clinical test widely used in secondary care for grading recovery from facial nerve paralysis caused by damage to lower motor neurons.

  • Central adjudication—Photographs taken of patients were assessed and graded independently by a panel of three experts (not general practitioners, who usually assess).

  • Suggested PRECIS score—1, very explanatory

  • *Sullivan FM, Swan IR, Donnan PT, et al. A randomised controlled trial of the use of aciclovir and/or prednisolone for the early treatment of Bell’s palsy: the BELLS study. Health Technol Assess 2009;13:iii-iv, ix-xi, 1-130.

9. Primary analysis—To what extent are all data included?

Most trials are a superiority design so the most pragmatic approach with regard to the analysis would be to make no special allowance in the analysis for non-adherence, practice variability, etc. In other words, the pragmatic approach to the analysis would typically be an intention-to-treat analysis using all available data. This sort of analysis would score at or close to 5 on PRECIS-2. (Non-inferiority and equivalence trials are usually contrary to intention-to-treat analysis.)

The intention-to-treat analysis has recently also become the norm for superiority trials taking a more explanatory design approach, especially when regulatory approval for an intervention is being sought. The most explanatory approach would be to use the “as treated analysis” as the primary analysis, in which only those patients who actually received (or did not receive) the intervention would be analysed in the intervention and control groups respectively, irrespective of their initial randomised group allocation.

Other causes of missing data may be important but may not affect how pragmatic or explanatory a trial is. Systematic exclusion of data from participants because, say, they were poorly adherent would make a trial more explanatory. Using all data but doing nothing to try and fill gaps caused by missing data would not in itself make trial more pragmatic or explanatory; missing data, especially if there is a lot of it, makes any conclusions more uncertain regardless of the design approach taken. As Vickers and Altman state, “analysis of missing data teaches us the importance of avoiding missing data in the first place: an informed guess, even using a technique as sophisticated as multiple imputation, is still a guess.”18

Approaches to the primary analysis that are likely to reduce the PRECIS-2 score include

  • Excluding non-compliant recipients (per protocol analysis)

  • Analyse recipients to treatment received instead of treatment randomised (as treated analysis)

  • Excluding data from non-adherent providers

  • Excluding data from trial sites or providers who recruit below expected volume.

For some trials taking an explanatory approach (dose finding trials are an example), it may be appropriate to have primary analysis restricted in the ways mentioned; otherwise such restricted analyses of the primary outcome could be pre-planned as secondary analyses of the primary outcome.

Primary analysis, example 1—Self management course for chronic musculoskeletal pain*

  • Intention-to-treat principle was applied with multiple imputation of data

  • Suggested PRECIS score—5, very pragmatic

  • *Carnes D, Taylor SJ, Homer K, et al. Effectiveness and cost-effectiveness of a novel, group self-management course for adults with chronic musculoskeletal pain: study protocol for a multicentre, randomised controlled trial (COPERS). BMJ Open 2013;3:e002492.

Primary analysis, example 2—Effects of rosuvastatin versus atorvastatin on LDL and HDL cholesterol in patients with type IIa or IIb hypercholesterolemia*

  • Dietary lead in to screen and exclude non-compliers, then post-randomisation excluded non-compliers who did not take medication, so “per protocol analysis.” The trial did, however, include those who violated protocol, deviated from protocol, or withdrew (mainly due to adverse events)

  • Suggested PRECIS score—2, rather explanatory

  • *Davidson M, Ma P, Stein EA, et al. Comparison of effects on low-density lipoprotein cholesterol and high-density lipoprotein cholesterol with rosuvastatin versus atorvastatin in patients with type IIa or IIb hypercholesterolemia. Am J Cardiol 2002;89:268-75.

Conclusion

PRECIS-2 has been developed through extensive international consultation with trialists, and we believe it will help trialists (new and experienced) ensure that they match their design decisions to the needs of those they intend to use the results of the trial. It makes trial teams explicitly aware of the range of opinions within the team and facilitates discussion and eventual consensus. The tool aims to assist in obtaining consistency in decision making, meaning decisions made on each domain of PRECIS-2 should broadly be in keeping with each other. It does not remove the need for judgment because there is no single “correct” answer to be discovered. The advantage of PRECIS-2 is that it makes these judgments explicit and therefore able to be discussed by the trial team.

PRECIS-2 could be a tool in a multifaceted package to help trialists design trials that meet the users’ needs. For instance, in the US, the NIH Clinical and Translational Science Awards (CTSA) Consortium does not conduct trials but facilitates their conduct and implementation, and PRECIS-2 could used as part of those activities.19 Trials units and other centres of trial expertise could work through the PRECIS-2 wheel with investigators to ensure that trial design decisions are consistent with the investigators’ intended purpose. The tool could also be a way of promoting trials that efficiently assist in the continued development of evidence based care.20 It would thus continue to promote and re-energise Archie Cochrane’s vision of efficient effective healthcare by addressing the issue of applicability and extrapolating trial results into real life.21

There is a substantial degree of waste in medical research (see http://researchwaste.net). Some of this waste is due to trials providing results that are irrelevant to the healthcare decisions of those for whom the trial was intended to help.4 5 6 We believe PRECIS-2 can help to reduce this waste by raising awareness of the need to explicitly consider the match between design decisions (and the consequences of these) and the usefulness of the future results to the intended audience. As Dave Sackett has pointed out, the applicability of the results of a trial can never be assured,22 because of the complexity of patients, health professionals, clinical settings, cultures, and healthcare systems. But using PRECIS-2 to consider applicability at the design stage will make it more likely that the results of a trial are more useful to their intended users than they otherwise would have been.

As in the original PRECIS tool, the authors welcome feedback on PRECIS-2 to continue the quest to design trials fit for purpose.

Notes

Cite this as: BMJ 2015;350:h2147

Footnotes

  • We acknowledge the contributions made by the participants at the THETA meeting in Toronto in March 2013, the Delphi participants, the user testing participants, the raters who helped validate PRECIS-2, and the web designer: Lusine Abrahamyan, Fernando Althabe, Doug Altman, Luca Boni, Eric Brass, Daniel Bratton, Marion Campbell, June Carroll, Heather Cassie, Vilemine Carayanni, Kalipso Chalkidou, An-Wen Chan, Sandra Eldridge, Louis D Fiore, Gordon Forbes, Bridget Gaglio, Christian Gold, Amanda Graham, Bruce Guthrie, Joel Gagnier, David L Hahn, Peter Hall, Hans Hobbelen, Sally Hopewell, Gisele Huf, Eric Johnson, David Kent, Rodger Kessler, Jerry Krishnan, Thomas Lamont, Roberta Littleford, Ingrid Liodden, Noemi Lois, Graeme MacLennan, Vishnu Madhok, Janet Martin, Alison McDonald, David Moher, Franz Portzsolt, John Powers, Marianna Purgato, Valeria Rac, Jean Raymond, Clare Relton, Kim Reynolds, Dan Riddle, Russell Glasgow, Jodi Segal, Peter Selby, Daniel Steinfort, Blair Smith, Grace Thompson, Leandro Tramma, Peter Tugwell, Sabine N Van der Veer, Vivian Welch, Gianni Virgili, Fiona Williams, Hywel C Williams, Diane Witham, Miles Witham, Claudia Witt.

  • Contributors: KL, ST, MZ, FS, and PD designed the study. KL undertook the work while supervised by ST, FS, PD, and MZ. KL, ST, and PD analysed the data. KL, ST, and MZ drafted the original paper, FS and PD drafted and revised the paper, and KET revised the draft paper after contributing at Theta Brainstorming meeting, Toronto. All authors contributed to, and approved, the final version of this paper. KL is guarantor.

  • Competing interests: All authors have completed the ICMJ uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: this work was supported by the Chief Scientist Office (CSO) of Scotland grant CZH/4/773, the UK Medical Research Council and the University of Dundee work through the provision of a PhD stipend for KL and from the Health Services Research Unit at the University of Aberdeen, which is core funded by the CSO of the Scottish Government Health Directories.

References

View Abstract