Qualitative Research: Case study evaluationBMJ 1995; 311 doi: http://dx.doi.org/10.1136/bmj.311.7002.444 (Published 12 August 1995) Cite this as: BMJ 1995;311:444
- aCorrespondence to: Dr Keen.
Case study evaluations, using one or more qualitative methods, have been used to investigate important practical and policy questions in health care. This paper describes the features of a well designed case study and gives examples showing how qualitative methods are used in evaluations of health services and health policy.
The medical approach to understanding disease has traditionally drawn heavily on qualitative data, and in particular on case studies to illustrate important or interesting phenomena. The tradition continues today, not least in regular case reports in this and other medical journals. Moreover, much of the everyday work of doctors and other health professionals still involves decisions that are qualitative rather than quantitative in nature.
This paper discusses the use of qualitative research methods, not in clinical care but in case study evaluations of health service interventions. It is useful for doctors to understand the principles guiding the design and conduct of these evaluations, because they are frequently used by both researchers and inspectorial agencies (such as the Audit Commission in the United Kingdom and the Office of Technology Assessment in the United States) to investigate the work of doctors and other health professionals.
We briefly discuss the circumstances in which case study research can usefully be undertaken in health service settings and the ways in which qualitative methods are used within case studies. Examples show how qualitative methods are applied, both in purely qualitative studies and alongside quantitative methods.
Case study evaluations
Doctors often find themselves asking important practical questions, such as should we be involved in the management of hospitals and, if so, how? how will new government policies affect the lives of our patients? and how can we cope with changes in practice in our local setting? There are, broadly, two ways in which such questions can usefully be addressed. One is to analyse the proposed policies themselves, by investigating whether they are internally consistent and by using theoretical frameworks to predict their effects on the ground. National policies, including the implementation of the NHS internal market1 and the new community care arrangements2 have been examined in this way by using economic theory to analyse their likely consequences.
The other approach, and the focus of this article, is to study implementation empirically. Empirical evaluative studies are concerned with placing a value on an intervention or policy change, and they typically involve forming judgments, firstly about the appropriateness of an intervention for those concerned (and often by implication also for the NHS as a whole) and, secondly about whether the outputs and outcomes of interventions are justified by their inputs and processes.
Case study evaluations are valuable where broad, complex questions have to be addressed in complex circumstances. No one method is sufficient to capture all salient aspects of an intervention, and case studies typically use multiple methods.
The methods used in case studies may be qualitative or quantitative, depending on the circumstances. Case studies using qualitative methods are most valuable when the question being posed requires an investigation of a real life intervention in detail, where the focus is on how and why the intervention succeeds or fails, where the general context will influence the outcome and where researchers asking the questions will have no control over events. As a result, the number of relevant variables will be far greater than can be controlled for, so that experimental approaches are simply not appropriate.
Other conditions that enhance the value of the case study approach concern the nature of the intervention being investigated. Often an intervention is ill defined, at least at the outset, and so cannot easily be distinguished from the general environment. Even where it is well defined, an intervention may not be discrete but consist of a complex mix of changes that occur over different timescales. This is a pervasive problem in health services in many countries, which are experiencing many parallel and interrelated changes. The doctor weighing up whether or how to become involved in hospital management would have to assess the various impacts on the managerial role of clinical audit, resource management, consultant job plans, and a raft of government legislation. Secondly, any intervention will typically depend for its success on the involvement of several different interested groups. Each group may have a legitimate, but different, interpretation of events; capturing these different views is often best achieved by using interviews or other qualitative methods within a case study design. Thirdly, it is not clear at the outset whether an intervention will be fully implemented by the end of a study period--accounts of major computer system failures show this.3 Yet study of these failures may provide invaluable clues for future success.
Taken together, these conditions exclude experimental approaches to evaluation. The case study is an alternative approach--in effect, a different way of thinking about complex situations which takes the conditions into account, but is nevertheless rigorous and facilitates informed judgments about success or failure.
The design of case studies
As noted earlier, case studies using qualitative methods are used by bodies that inspect and regulate public services. Examples include the work of the National Audit Office and the Audit Commission4 in the United Kingdom and the Office of Technology Assessment in the United States.5 Sometimes these studies are retrospective, particularly in investigations of failed implementations of policies. Increasingly, though, these bodies use prospective studies designed to investigate the extent to which centrally determined standards or initiatives have been implemented. For example, the National Audit Office recently examined hospital catering in England, focusing on the existence of, and monitoring of, standards as required by the citizen's charter and on the application of central policy and guidance in the areas of nutritional standards and cost control.6
Prospective studies have also been used by academic researchers, for example, to evaluate the introduction of general management7 in Britain after the Griffiths report,8 in the studies of specific changes following the 1989 NHS review9 which were commissioned by the King's Fund,10 and in the introduction of total quality management in hospitals in the United States.11 In these cases the investigators were interested in understanding what happened in a complex environment where they had no control over events. Their research questions emerged from widespread concerns about the implications of new policies or management theories, and were investigated with the most appropriate methods at their disposal.
THE NATURE OF RESEARCH QUESTIONS
Once a broad research question has been identified, there are two approaches to the design of case study research, with appropriateness depending on the circumstances. In the first approach, precise questions are posed at the outset of the research and data collection and analysis are directed towards answering them. These studies are typically constructed to allow comparisons to be drawn.12 The comparison may be between different approaches to implementation, or a comparison between sites where an intervention is taking place and ones where normal practice prevails.
An example is the recent study by Glennerster et al of the implementation of general practitioner fundholding.13 Starting with a broad question about the value of general practitioner fundholding, the researchers narrowed down to precise questions about the extent to which the fundholding scheme promoted efficiency and preserved equity. They used one qualitative method, semistructured interviews, with the general practitioners and practice managers and also with people responsible for implementing the policy at national and regional level. The interviews were complemented by the collection of quantitative data such as financial information from the practices (box 1).
Box 1 Outline of case study of GPfundholding13
Mix of qualitative and quantitative methods
Fundholding and non-fundholding practices
Programme of interviews with key staff at practices
Interviews with people responsible for imple-menting national policy
Study found that the general practitioner fund-holding scheme was achieving the aims set for it bygovernment and that adverse selection (“creamskimming”) of patients was less likely than some commentators had feared
The second approach is more open and in effect starts by asking broad questions such as what is happening here? and, what are the important features and relationships that explain the impact of this intervention? These questions are then refined and become more specific in the course of fieldwork and a parallel process of data analysis. This type of design, in which the eventual research questions emerge during the research, is termed ethnography and has been advocated for use in the study of the impact of government policies in the health system.14 15 In some ways it is similar to the way in which consultations are conducted in that it involves initial exploration, progressing over time towards a diagnosis inferred from the available data.
The evaluation of resource management in the NHS,16 which investigated the progress of six pilot hospitals in implementing new management arrangements, focused particularly on identifying ways in which doctors and general managers could jointly control the allocation and commitment of resources (box 2). At the outset the nature of resource management was unclear--sites were charged with finding ways of involving doctors in management, but how this would be achieved and, if achieved, how successful it would be in improving patient care were open questions. The researchers selected major specialties within each site and conducted interviews with relevant staff, observed meetings, and analysed documentation. Over time, the data were used to develop a framework which captured the essential features of resource management at the time and which was used to evaluate each site's progress in implementing it.
Box 2 Evaluation of resourcemanagement16
Six hospitals, a mix of teaching and non-teaching
Focus on major specialties: general surgery and general medicine
Mix of qualitative and quantitative methods
Methods and data sources independent of each other
Qualitative methods included interviews, non-participant observation of meetings, analysis of documentation
Evaluation found that there were important changes in management processes, but little evidence of improvement in patient care
SELECTION OF SITES
The process of selecting sites for study is central to the case study approach. Researchers have developed a number of selection strategies, the objectives of which, as in any good research study, are to ensure that misinterpretation of results is as far as possible avoided. Criteria include the selection of cases that are typical of the phenomenon being investigated, those in which a specific theory can be tested, or those that will confirm or refute a hypothesis.
Researchers will benefit from expert advice from those with knowledge of the subject being investigated, and they can usefully build into the initial research design the possibility of testing findings at further sites. Replication of results across sites helps to ensure that findings are not due to characteristics of particular sites; hence it increases external validity.17
SELECTION OF METHODS
The next step is to select research methods, the process being driven by criteria of validity and reliability.18 A distinctive but not unique feature of case study research is the use of multiple methods and sources of evidence to establish construct validity. The use of particular methods is discussed in other papers in this series; the validity and reliability of individual methods is discussed in more detail by Mays and Pope.19
Case studies often use triangulation20 to ensure the validity of findings. In triangulation all data items are corroborated from at least one other source and normally by another method of data collection. The fundholding study referred to earlier13 used interviews in combination with several different quantitative sources of data to establish an overall picture. The evaluation of resource management, in contrast, used a wider range of qualitative and quantitative methods.16
Any one of these methods by itself might have produced results of weak validity, but the different methods were used to obtain data from different sources. When they all suggested the emergence of an important development, therefore, they acted to strengthen the researchers' belief in the validity of their observations.
Another technique is to construct chains of evidence; these are conceptual arguments that link phenomena to one another in the following manner: “if this occurs then some other thing would be expected to occur; and if not, then it would not be expected.” For example, if quantitative evidence suggested that there had been an increase or decrease in admission rates in several specialties within a resource management site and if an interview programme revealed that the involvement of doctors in management (if developed as part of the resource management initiative) had led to a higher level of coordination of admissions policies, then this is evidence that resource management may facilitate the introduction of such policies. This type of argument is not always appropriate, but it can be valuable where it is important to investigate causation in complex environments.
The collection of data should be directed towards the development of an analytical framework that will facilitate interpretation of findings. Again, there are several ways in which this might be done. In the study of fundholding13 the data were organised to “test” hypotheses which were derived from pre-existing economic theories. In the case of resource management there was no obvious pre-existing theory that could be used; the development of a framework during the study was crucial to help organise and evaluate the data collected. The framework was not imposed on the data but derived from it in an iterative process over the course of the evaluation; each was used to refine the other over time (box 3).15
Framework: five interrelated elements of resource management16
The target should be a reduction in the consumption itself
Commitment to resource management by the relevant personnel at each level in the organisation
Devolution of authority for the management ofresources
Collaboration within and between disciplines insecuring the objectives of resource management
Management infrastructure, particularly in termsof organisational structure and provision of information
A clear focus for the local resource management strategy
The investigator is finally left with the difficult task of making a judgment about the findings of a study. The purpose of the steps in designing and building the case study research is to maximise confidence in the findings, but interpretation inevitably involves value judgments. The findings may well include divergences of opinion among those involved about the value of the intervention, and the results will often point towards different conclusions.
The extent to which research findings can be assembled into a single coherent account of events varies widely. In some circumstances widely differing opinions are themselves very important and should be reflected in any report. Where an evaluation is designed to inform policy making, however, some attempt has to be made at an overall judgment of success or failure; this was the case in the evaluation of resource management, where it was important to indicate to policy makers and the NHS whether it was worth while.
The complexity of the issues that health professionals have to deal with and the increasing recognition by policy makers, academics, and practitioners of the value of case studies in evaluating health service interventions suggest that the use of such studies is likely to increase in the future. Qualitative methods can be used within case study designs to address many practical and policy questions that impinge on the lives of professionals, particularly where those questions are concerned with how or why events take a particular course.