- David H Peters, professor1,
- Taghreed Adam, scientist2,
- Olakunle Alonge, assistant scientist1,
- Irene Akua Agyepong, specialist public health3,
- Nhan Tran, manager4
- 1Johns Hopkins University Bloomberg School of Public Health, Department of International Health, 615 N Wolfe St, Baltimore, MD 21205, USA
- 2Alliance for Health Policy and Systems Research, World Health Organization, CH-1211 Geneva 27, Switzerland
- 3University of Ghana School of Public Health/Ghana Health Service, Accra, Ghana
- 4Alliance for Health Policy and Systems Research, Implementation Research Platform, World Health Organization, CH-1211 Geneva 27, Switzerland
- Correspondence to: D H Peters
- Accepted 8 October 2013
The field of implementation research is growing, but it is not well understood despite the need for better research to inform decisions about health policies, programmes, and practices. This article focuses on the context and factors affecting implementation, the key audiences for the research, implementation outcome variables that describe various aspects of how implementation occurs, and the study of implementation strategies that support the delivery of health services, programmes, and policies. We provide a framework for using the research question as the basis for selecting among the wide range of qualitative, quantitative, and mixed methods that can be applied in implementation research, along with brief descriptions of methods specifically suitable for implementation research. Expanding the use of well designed implementation research should contribute to more effective public health and clinical policies and programmes.
Defining implementation research
Implementation research attempts to solve a wide range of implementation problems; it has its origins in several disciplines and research traditions (supplementary table A). Although progress has been made in conceptualising implementation research over the past decade,1 considerable confusion persists about its terminology and scope.2 3 4 The word “implement” comes from the Latin “implere,” meaning to fulfil or to carry into effect.5 This provides a basis for a broad definition of implementation research that can be used across research traditions and has meaning for practitioners, policy makers, and the interested public: “Implementation research is the scientific inquiry into questions concerning implementation—the act of carrying an intention into effect, which in health research can be policies, programmes, or individual practices (collectively called interventions).”
Implementation research can consider any aspect of implementation, including the factors affecting implementation, the processes of implementation, and the results of implementation, including how to introduce potential solutions into a health system or how to promote their large scale use and sustainability. The intent is to understand what, why, and how interventions work in “real world” settings and to test approaches to improve them.
Principles of implementation research
Implementation research seeks to understand and work within real world conditions, rather than trying to control for these conditions or to remove their influence as causal effects. This implies working with populations that will be affected by an intervention, rather than selecting beneficiaries who may not represent the target population of an intervention (such as studying healthy volunteers or excluding patients who have comorbidities).
Context plays a central role in implementation research. Context can include the social, cultural, economic, political, legal, and physical environment, as well as the institutional setting, comprising various stakeholders and their interactions, and the demographic and epidemiological conditions. The structure of the health systems (for example, the roles played by governments, non-governmental organisations, other private providers, and citizens) is particularly important for implementation research on health.
Implementation research is especially concerned with the users of the research and not purely the production of knowledge. These users may include managers and teams using quality improvement strategies, executive decision makers seeking advice for specific decisions, policy makers who need to be informed about particular programmes, practitioners who need to be convinced to use interventions that are based on evidence, people who are influenced to change their behaviour to have a healthier life, or communities who are conducting the research and taking action through the research to improve their conditions (supplementary table A). One important implication is that often these actors should be intimately involved in the identification, design, and conduct phases of research and not just be targets for dissemination of study results.
Implementation outcome variables
Implementation outcome variables describe the intentional actions to deliver services.6 These implementation outcome variables—acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, coverage, and sustainability—can all serve as indicators of the success of implementation (table 1⇓). Implementation research uses these variables to assess how well implementation has occurred or to provide insights about how this contributes to one’s health status or other important health outcomes.
|Implementation outcome||Working definition*||Related terms†|
|Acceptability||The perception among stakeholders (for example, consumers, providers, managers, policy makers) that an intervention is agreeable||Factors related to acceptability (for example, comfort, relative advantage, credibility)|
|Adoption||The intention, initial decision, or action to try to employ a new intervention||Uptake, utilisation, intention to try|
|Appropriateness||The perceived fit or relevance of the intervention in a particular setting or for a particular target audience (for example, provider or consumer) or problem||Relevance, perceived fit, compatibility, perceived usefulness or suitability|
|Feasibility||The extent to which an intervention can be carried out in a particular setting or organisation||Practicality, actual fit, utility, trialability|
|Fidelity||The degree to which an intervention was implemented as it was designed in an original protocol, plan, or policy||Adherence, delivery as intended, integrity, quality of programme delivery, intensity or dosage of delivery|
|Implementation cost||The incremental cost of the implementation strategy (for example, how the services are delivered in a particular setting). The total cost of implementation would also include the cost of the intervention itself||Marginal cost, total cost‡|
|Coverage||The degree to which the population that is eligible to benefit from an intervention actually receives it.||Reach, access, service spread or effective coverage (focusing on those who need an intervention and its delivery at sufficient quality, thus combining coverage and fidelity), penetration (focusing on the degree to which an intervention is integrated in a service setting)|
|Sustainability||The extent to which an intervention is maintained or institutionalised in a given setting||Maintenance, continuation, durability, institutionalisation, routinisation, integration, incorporation|
Adapted from references 6 and 33.
*Original definitions referred to individual “innovations or evidence-based practices.” This table uses the term “intervention” so that the definitions are more broadly applicable to programmes and policies. The original authors used the term “penetration” rather than “coverage.”
†Other terms are more commonly found in implementation literature on large scale programmes and policies.8,34-36
‡Cost data also provide numerators for measures of efficiency and specifically measures of cost-utility, cost-benefit, or cost effectiveness.
Curran and colleagues defined an “implementation intervention” as a method to “enhance the adoption of a ‘clinical’ intervention,” such as the use of job aids, provider education, or audit procedures.7 The concept can be broadened to any type of strategy that is designed to support a clinical or population and public health intervention (for example, outreach clinics and supervision checklists are implementation strategies used to improve the coverage and quality of immunisation).
A review of ways to improve health service delivery in low and middle income countries identified a wide range of successful implementation strategies (supplementary table B).8 Even in the most resource constrained environments, measuring change, informing stakeholders, and using information to guide decision making were found to be critical to successful implementation.
Implementation influencing variables
Other factors that influence implementation may need to be considered in implementation research. Sabatier summarised a set of such factors that influence policy implementation (clarity of objectives, causal theory, implementing personnel, support of interest groups, and managerial authority and resources).9
The large array of contextual factors that influence implementation, interact with each other, and change over time highlights the fact that implementation often occurs as part of complex adaptive systems.10 Some implementation strategies are particularly suitable for working in complex systems. These include strategies to provide feedback to key stakeholders and to encourage learning and adaptation by implementing agencies and beneficiary groups. Such strategies have implications for research, as the study methods need to be sufficiently flexible to account for changes or adaptations in what is actually being implemented.8 11 Research designs that depend on having a single and fixed intervention, such as a typical randomised controlled trial, would not be an appropriate design to study phenomena that change, especially when they change in unpredictable and variable ways.
Another implication of studying complex systems is that the research may need to use multiple methods and different sources of information to understand an implementation problem. Because implementation activities and effects are not usually static or linear processes, research designs often need to be able to observe and analyse these sometimes iterative and changing elements at several points in time and to consider unintended consequences.
Implementation research questions
As in other types of health systems research, the research question is the king in implementation research. Implementation research takes a pragmatic approach, placing the research question (or implementation problem) as the starting point to inquiry; this then dictates the research methods and assumptions to be used. Implementation research questions can cover a wide variety of topics and are frequently organised around theories of change or the type of research objective (examples are in supplementary table C).12 13
Implementation research can overlap with other types of research used in medicine and public health, and the distinctions are not always clear cut. A range of implementation research exists, based on the centrality of implementation in the research question, the degree to which the research takes place in a real world setting with routine populations, and the role of implementation strategies and implementation variables in the research (figure⇓).
A more detailed description of the research question can help researchers and practitioners to determine the type of research methods that should be used. In table 2⇓, we break down the research question first by its objective: to explore, describe, influence, explain, or predict. This is followed by a typical implementation research question based on each objective. Finally, we describe a set of research methods for each type of research question.
|Objective||Description||Implementation question||Research methods and data collection approaches|
|Explore||Explore an idea or phenomenon to make hypotheses or generalisations from specific examples||What are the possible factors and agents responsible for good implementation of a health intervention? For enhancing or expanding a health intervention?||Qualitative methods: grounded theory, ethnography, phenomenology, case studies and narrative approaches; key informant interviews, focus groups, historical reviews|
|Quantitative: network analysis, cross sectional surveys|
|Mixed methods: combining qualitative and quantitative methods|
|Describe||Identify and describe the phenomenon and its correlates or possible causes||What describes the context in which implementation occurs? What describes the main factors influencing implementation in a given context?||Quantitative: cross sectional (descriptive) surveys, network analysis|
|Qualitative methods: ethnography, phenomenology, case studies and narrative approaches; key informant interviews, focus groups, historical reviews|
|Mixed methods: both qualitative and quantitative inquiry with convergence of data and analyses|
|Influence||Test whether an intervention produces an expected outcome|
|With adequacy||With sufficient confidence that the intervention and outcomes are occurring||Is coverage of a health intervention changing among beneficiaries of the intervention?||Before-after or time series in intervention recipients only; participatory action research|
|With plausibility||With greater confidence that the outcome is due to the intervention||Is a health outcome plausibly due to the implemented intervention rather than other causes?||Concurrent, non-randomised cluster trials: health intervention implemented in some areas and not in others; before-after or cross sectional study in programme recipients and non-recipients; typical quality improvement studies|
|With probability||With a high (calculated) probability that the outcome is due to the intervention||Is a health outcome due to implementation of the intervention?||Partially controlled trials: pragmatic and cluster randomised trials; health intervention implemented in some areas and not in others; effectiveness-implementation hybrids|
|Explain||Develop or expand a theory to explain the relation between concepts, the reasons for the occurrence of events, and how they occurred||How and why does implementation of the intervention lead to effects on health behaviour, services, or status in all its variations?||Mixed methods: both qualitative and quantitative inquiry with convergence of data and analyses|
|Quantitative: repeated measures of context, actors, depth and breadth of implementation across subunits; network identification; can use designs for confirmatory inferences; effectiveness-implementation hybrids|
|Qualitative methods: case studies, phenomenological and ethnographic approaches with key informant interviews, focus groups, historical reviews|
|Participatory action research|
|Predict||Use prior knowledge or theories to forecast future events||What is the likely course of future implementation?||Quantitative: agent based modelling; simulation and forecasting modelling; data extrapolation and sensitivity analysis (trend analysis, econometric modelling)|
|Qualitative: scenario building exercises; Delphi techniques from opinion leaders|
Adapted from references 8, 14, and 33.
Much of evidence based medicine is concerned with the objective of influence, or whether an intervention produces an expected outcome, which can be broken down further by the level of certainty in the conclusions drawn from the study. The nature of the inquiry (for example, the amount of risk and considerations of ethics, costs, and timeliness), and the interests of different audiences, should determine the level of uncertainty.8 14 Research questions concerning programmatic decisions about the process of an implementation strategy may justify a lower level of certainty for the manager and policy maker, using research methods that would support an adequacy or plausibility inference.14 Where a high risk of harm exists and sufficient time and resources are available, a probability study design might be more appropriate, in which the result in an area where the intervention is implemented is compared with areas without implementation with a low probability of error (for example, P< 0.05). These differences in the level of confidence affect the study design in terms of sample size and the need for concurrent or randomised comparison groups.8 14
Implementation specific research methods
A wide range of qualitative and quantitative research methods can be used in implementation research (table 2⇑). The box gives a set of basic questions to guide the design or reporting of implementation research that can be used across methods. More in-depth criteria have also been proposed to assess the external validity or generalisability of findings.15 Some research methods have been developed specifically to deal with implementation research questions or are particularly suitable to implementation research, as identified below.
Key questions to assess research designs or reports on implementation research33
Does the research clearly aim to answer a question concerning implementation?
Does the research clearly identify the primary audiences for the research and how they would use the research?
Is there a clear description of what is being implemented (for example, details of the practice, programme, or policy)?
Does the research involve an implementation strategy? If so, is it described and examined in its fullness?
Is the research conducted in a “real world” setting? If so, is the context and sample population described in sufficient detail?
Does the research appropriately consider implementation outcome variables?
Does the research appropriately consider context and other factors that influence implementation?
Does the research appropriately consider changes over time and the level of complexity of the system, including unintended consequences?
Pragmatic trials, or practical trials, are randomised controlled trials in which the main research question focuses on effectiveness of an intervention in a normal practice setting with the full range of study participants.16 This may include pragmatic trials on new healthcare delivery strategies, such as integrated chronic care clinics or nurse run community clinics. This contrasts with typical randomised controlled trials that look at the efficacy of an intervention in an “ideal” or controlled setting and with highly selected patients and standardised clinical outcomes, usually of a short term nature.
Effectiveness-implementation hybrid trials
Effectiveness-implementation hybrid designs are intended to assess the effectiveness of both an intervention and an implementation strategy.7 These studies include components of an effectiveness design (for example, randomised allocation to intervention and comparison arms) but add the testing of an implementation strategy, which may also be randomised. This might include testing the effectiveness of a package of delivery and postnatal care in under-served areas, as well testing several strategies for providing the care. Whereas pragmatic trials try to fix the intervention under study, effectiveness-implementation hybrids also intervene and/or observe the implementation process as it actually occurs. This can be done by assessing implementation outcome variables.
Quality improvement studies
Quality improvement studies typically involve a set of structured and cyclical processes, often called the plan-do-study-act cycle, and apply scientific methods on a continuous basis to formulate a plan, implement the plan, and analyse and interpret the results, followed by an iteration of what to do next.17 18 The focus might be on a clinical process, such as how to reduce hospital acquired infections in the intensive care unit, or management processes such as how to reduce waiting times in the emergency room. Guidelines exist on how to design and report such research—the Standards for Quality Improvement Reporting Excellence (SQUIRE).17
Speroff and O’Connor describe a range of plan-do-study-act research designs, noting that they have in common the assessment of responses measured repeatedly and regularly over time, either in a single case or with comparison groups.18 Balanced scorecards integrate performance measures across a range of domains and feed into regular decision making.19 20 Standardised guidance for using good quality health information systems and health facility surveys has been developed and often provides the sources of information for these quasi-experimental designs.21 22 23
Participatory action research
Participatory action research refers to a range of research methods that emphasise participation and action (that is, implementation), using methods that involve iterative processes of reflection and action, “carried out with and by local people rather than on them.”24 In participatory action research, a distinguishing feature is that the power and control over the process rests with the participants themselves. Although most participatory action methods involve qualitative methods, quantitative and mixed methods techniques are increasingly being used, such as for participatory rural appraisal or participatory statistics.25 26
Mixed methods research uses both qualitative and quantitative methods of data collection and analysis in the same study. Although not designed specifically for implementation research, mixed methods are particularly suitable because they provide a practical way to understand multiple perspectives, different types of causal pathways, and multiple types of outcomes—all common features of implementation research problems.
Many different schemes exist for describing different types of mixed methods research, on the basis of the emphasis of the study, the sampling schemes for the different components, the timing and sequencing of the qualitative and quantitative methods, and the level of mixing between the qualitative and quantitative methods.27 28 Broad guidance on the design and conduct of mixed methods designs is available.29 30 31 A scheme for good reporting of mixed methods studies involves describing the justification for using a mixed methods approach to the research question; describing the design in terms of the purpose, priority, and sequence of methods; describing each method in terms of sampling, data collection, and analysis; describing where the integration has occurred, how it has occurred, and who has participated in it; describing any limitation of one method associated with the presence of the other method; and describing any insights gained from mixing or integrating methods.32
Implementation research aims to cover a wide set of research questions, implementation outcome variables, factors affecting implementation, and implementation strategies. This paper has identified a range of qualitative, quantitative, and mixed methods that can be used according to the specific research question, as well as several research designs that are particularly suited to implementation research. Further details of these concepts can be found in a new guide developed by the Alliance for Health Policy and Systems Research.33
Implementation research has its origins in many disciplines and is usefully defined as scientific inquiry into questions concerning implementation—the act of fulfilling or carrying out an intention
In health research, these intentions can be policies, programmes, or individual practices (collectively called interventions)
Implementation research seeks to understand and work in “real world” or usual practice settings, paying particular attention to the audience that will use the research, the context in which implementation occurs, and the factors that influence implementation
A wide variety of qualitative, quantitative, and mixed methods techniques can be used in implementation research, which are best selected on the basis of the research objective and specific questions related to what, why, and how interventions work
Implementation research may examine strategies that are specifically designed to improve the carrying out of health interventions or assess variables that are defined as implementation outcomes
Implementation outcomes include acceptability, adoption, appropriateness, feasibility, fidelity, implementation cost, coverage, and sustainability
Cite this as: BMJ 2013;347:f6753
Contributors: All authors contributed to the conception and design, analysis and interpretation, drafting the article, or revising it critically for important intellectual content, and all gave final approval of the version to be published. NT had the original idea for the article, which was discussed by the authors (except OA) as well as George Pariyo, Jim Sherry, and Dena Javadi at a meeting at the World Health Organization (WHO). DHP and OA did the literature reviews, and DHP wrote the original outline and the draft manuscript, tables, and boxes. OA prepared the original figure. All authors reviewed the draft article and made substantial revisions to the manuscript. DHP is the guarantor.
Funding: Funding was provided by the governments of Norway and Sweden and the UK Department for International Development (DFID) in support of the WHO Implementation Research Platform, which financed a meeting of authors and salary support for NT. DHP is supported by the Future Health Systems research programme consortium, funded by DFID for the benefit of developing countries (grant number H050474). The funders played no role in the design, conduct, or reporting of the research.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf and declare: support for the submitted work as described above; NT and TA are employees of the Alliance for Health Policy and Systems Research at WHO, which is supporting their salaries to work on implementation research; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.
Provenance and peer review: Invited by journal; commissioned by WHO; externally peer reviewed.