Qualitative Research: Rigour and qualitative researchBMJ 1995; 311 doi: http://dx.doi.org/10.1136/bmj.311.6997.109 (Published 08 July 1995) Cite this as: BMJ 1995;311:109
- Nicholas Mays, director of health services researcha,
- Catherine Pope, lecturer in social and behavioural medicineb
- aKing's Fund Institute, London W2 4HT
- bDepartment of Epidemiology and Public Health, University of Leicester, Leicester LE1 6TP
- Correspondence to: Mr Mays.
Various strategies are available within qualitative research to protect against bias and enhance the reliability of findings. This paper gives examples of the principal approaches and summarises them into a methodological checklist to help readers of reports of qualitative projects to assess the quality of the research.
Criticisms of qualitative research
In the health field--with its strong tradition of biomedical research using conventional, quantitative, and often experimental methods--qualitative research is often criticised for lacking scientific rigour. To label an approach “unscientific” is peculiarly damning in an era when scientific knowledge is generally regarded as the highest form of knowing. The most commonly heard criticisms are, firstly, that qualitative research is merely an assembly of anecdote and personal impressions, strongly subject to researcher bias; secondly, it is argued that qualiative research lacks reproducibility--the research is so personal to the researcher that there is no guarantee that a different researcher would not come to radically different conclusions; and, finally, qualitative research is criticised for lacking generalisability. It is said that qualitative methods tend to generate large amounts of detailed information about a small number of settings.
Is qualitative research different?
The pervasive assumption underlying all these criticisms is that quantitative and qualitative approaches are fundamentally different in their ability to ensure the validity and reliability of their findings. This distinction, however, is more one of degree than of type. The problem of the relation of a piece of research to some presumed underlying “truth” applies to the conduct of any form of social research. “One of the greatest methodological fallacies of the last half century in social research is the belief that science is a particular set of techniques; it is, rather, a state of mind, or attitude, and the organisational conditions which allow that attitude to be expressed.”1 In quantitative data analysis it is possible to generate statistical representations of phenomena which may or may not be fully justified since, just as in qualitative work, they will depend on the judgment and skill of the researcher and the appropriateness to the question answered of the data collected. All research is selective--there is no way that the researcher can in any sense capture the literal truth of events. All research depends on collecting particular sorts of evidence through the prism of particular methods, each of which has its strengths and weaknesses. For example, in a sample survey it is difficult for the researcher to ensure that the questions, categories, and language used in the questionnaire are shared uniformly by respondents and that the replies returned have the same meanings for all respondents. Similarly, research that relies exclusively on observation by a single researcher is limited by definition to the perceptions and introspection of the investigator and by the possibility that the presence of the observer may, in some way that is hard to characterise, have influenced the behaviour and speech that was witnessed. Britten and Fisher summarise the position neatly by pointing out that “there is some truth in the quip that quantitative methods are reliable but not valid and that qualitative methods are valid but not reliable.”2
Strategies to ensure rigour in qualitative research
As in quantitative research, the basic strategy to ensure rigour in qualitative research is systematic and self conscious research design, data collection, interpretation, and communication. Beyond this, there are two goals that qualitative researchers should seek to achieve: to create an account of method and data which can stand independently so that another trained researcher could analyse the same data in the same way and come to essentially the same conclusions; and to produce a plausible and coherent explanation of the phenomenon under scrutiny. Unfortunately, many qualitative researchers have neglected to give adequate descriptions in their research reports of their assumptions and methods, particularly with regard to data analysis. This has contributed to some of the criticisms of bias from quantitative researchers.
Yet the integrity of qualitative projects can be protected throughout the research process. The remainder of this paper discusses how qualitative researchers attend to issues of validity, reliability, and generalisability.
Much social science is concerned with classifying different “types” of behaviour and distinguishing the “typical” from the “atypical.” In quantitative research this concern with similarity and difference leads to the use of statistical sampling so as to maximise external validity or generalisability. Although statistical sampling methods such as random sampling are relatively uncommon in qualitative investigations, there is no reason in principle why they cannot be used to provide the raw material for a comparative analysis, particularly when the researcher has no compelling a priori reason for a purposive approach. For example, a random sample of practices could be studied in an investigation of how and why teamwork in primary health care is more and less successful in different practices. However, since qualitative data collection is generally more time consuming and expensive than, for example, a quantitative survey, it is not usually practicable to use a probability sample. Furthermore, statistical representativeness is not a prime requirement when the objective is to understand social processes.
An alternative approach, often found in qualitative research and often misunderstood in medical circles, is to use systematic, non-probabilistic sampling. The purpose is not to establish a random or representative sample drawn from a population but rather to identify specific groups of people who either possess characteristics or live in circumstances relevant to the social phenomenon being studied. Informants are identified because they will enable exploration of a particular aspect of behaviour relevant to the research. This approach to sampling allows the researcher deliberately to include a wide range of types of informants and also to select key informants with access to important sources of knowledge.
“Theoretical” sampling is a specific type of non-probability sampling in which the objective of developing theory or explanation guides the process of sampling and data collection.3 Thus, the analyst makes an initial selection of informants; collects, codes, and analyses the data; and produces a preliminary theoretical explanation before deciding which further data to collect and from whom. Once these data are analysed, refinements are made to the theory, which may in turn guide further sampling and data collection. The relation between sampling and explanation is iterative and theoretically led.
To return to the example of the study of primary care team working, some of the theoretically relevant characteristics of general practices affecting variations in team working might be the range of professions represented in the team, the frequency of opportunities for communication among team members, the local organisation of services, and whether the practice is in an urban, city, or rural area. These factors could be identified from other similar research and within existing social science theories of effective and ineffective team working and would then be used explicitly as sampling categories. Though not statistically representative of general practices, such a sample is theoretically informed and relevant to the research questions. It also minimises the possible bias arising from selecting a sample on the basis of convenience.
ENSURING THE RELIABILITY OF AN ANALYSIS
In many forms of qualitative research the raw data are collected in a relatively unstructured form such as tape recordings or transcripts of conversations. The main ways in which qualitative researchers ensure the retest reliability of their analyses is in maintaining meticulous records of interviews and observations and by documenting the process of analysis in detail. While it is possible to analyse such data singlehandedly and use ways of classifying and categorising the data which emerge from the analysis and remain implicit, more explicit group approaches, which perhaps have more in common with the quantitative social sciences, are increasingly used. The interpretative procedures are often decided on before the analysis. Thus, for example, computer software is available to facilitate the analysis of the content of interview transcripts.4 A coding frame can be developed to characterise each utterance (for example, in relation to the age, sex, and role of the speaker; the topic; and so on), and transcripts can then be coded by more than one researcher.5 One of the advantages of audiotaping or videotaping is the opportunity the tapes offer for subsequent analysis by independent observers.
The reliability of the analysis of qualitative data can be enhanced by organising an independent assessment of transcripts by additional skilled qualitative researchers and comparing agreement between the raters. For example, in a study of clinical encounters between cardiologists and their patients which looked at the differential value each derived from the information provided by echocardiography, transcripts of the clinic interviews were analysed for content and structure by the principal researcher and by an independent panel, and the level of agreement was assessed.6
Alongside issues of reliability, qualitative researchers give attention to the validity of their findings. “Triangulation” refers to an approach to data collection in which evidence is deliberately sought from a wide range of different, independent sources and often by different means (for instance, comparing oral testimony with written records). This approach was used to good effect in a qualitative study of the effects of the introduction of general management into the NHS. The accounts of doctors, managers, and patient advocates were explored in order to identify patterns of convergence between data sources to see whether power relations had shifted appreciably in favour of professional managers and against the medical profession.7
Validation strategies sometimes used in qualitative research are to feed the findings back to the participants to see if they regard the findings as a reasonable account of their experience8 and to use interviews or focus groups with the same people so that their reactions to the evolving analysis become part of the emerging research data.9 If used in isolation these techniques assume that fidelity to the participants' commonsense perceptions is the touchstone of validity. In practice, this sort of validation has to be set alongside other evidence of the plausibility of the research account since different groups are likely to have different perspectives on what is happening.10
A related analytical and presentational issue is concerned with the thoroughness with which the researcher examines “negative” or “deviant” cases--those in which the researcher's explanatory scheme appears weak or is contradicted by the evidence. The researcher should give a fair account of these occasions and try to explain why the data vary.11 In the same way, if the findings of a single case study diverge from those predicted by a previously stated theory, they can be useful in revising the existing theory in order to increase its reliability and validity.
VALIDITY AND EXPLANATION
It is apparent in qualitative research, particularly in observational studies (see the next paper in this series for more on observational methods), that the researcher can be regarded as a research instrument.12 Allowing for the inescapable fact that purely objective observation is not possible in social science, how can the reader judge the credibility of the observer's account? One solution is to ask a set of questions: how well does this analysis explain why people behave in the way they do; how comprehensible would this explanation be to a thoughtful participant in the setting; and how well does the explanation it advances cohere with what we already know?
This is a challenging enough test, but the ideal test of a qualitative analysis, particularly one based on observation, is that the account it generates should allow another person to learn the “rules” and language sufficiently well to be able to function in the research setting. In other words, the report should carry sufficient conviction to enable someone else to have the same experience as the original observer and appreciate the truth of the account.13 Few readers have the time or inclination to go to such lengths, but this provides an ideal against which the quality of a piece of qualitative work can be judged.
The development of “grounded theory”3 offers another response to this problem of objectivity. Under the strictures of grounded theory, the findings must be rendered through a systematic account of a setting that would be clearly recognisable to the people in the setting (by, for example, recording their words, ideas, and actions) while at the same time being more structured and self consciously explanatory than anything that the participants themselves would produce.
Attending to the context
Some pieces of qualitative research consist of a case study carried out in considerable detail in order to produce a naturalistic account of everyday life. For example, a researcher wishing to observe care in an acute hospital around the clock may not be able to study more than one hospital. Again the issue of generalisability, or what can be learnt from a single case, arises. Here, it is essential to take care to describe the context and particulars of the case study and to flag up for the reader the similarities and differences between the case study and other settings of the same type. A related way of making the best use of case studies is to show how the case study contributes to and fits with a body of social theory and other empirical work.12 The final paper in this series discusses qualitative case studies in more detail.
COLLECTING DATA DIRECTLY
Another defence against the charge that qualitative research is merely impressionistic is that of separating the evidence from secondhand sources and hearsay from the evidence derived from direct observation of behaviour in situ. It is important to ensure that the observer has had adequate time to become thoroughly familiar with the milieu under scrutiny and that the participants have had the time to become accustomed to having the researcher around. It is also worth asking whether the observer has witnessed a wide enough range of activities in the study site to be able to draw conclusions about typical and atypical forms of behaviour--for example, were observations undertaken at different times? The extent to which the observer has succeeded in establishing an intimate understanding of the research setting is often shown in the way in which the subsequent account shows sensitivity to the specifics of language and its meanings in the setting.
MINIMISING RESEARCHER BIAS IN THE PRESENTATION OF RESULTS
Although it is not normally appropriate to write up qualitative research in the conventional format of the scientific paper, with a rigid distinction between the results and discussion sections of the account, it is important that the presentation of the research allows the reader as far as possible to distinguish the data, the analytic framework used, and the interpretation.1 In quantitative research these distinctions are conventionally and neatly presented in the methods section, numerical tables, and the accompanying commentary. Qualitative research depends in much larger part on producing a convincing account.14 In trying to do this it is all too easy to construct a narrative that relies on the reader's trust in the integrity and fairness of the researcher. The equivalent in quantitative research is to present tables of data setting out the statistical relations between operational definitions of variables without giving any idea of how the phenomena they represent present themselves in naturally occurring settings.1 The need to quantify can lead to imposing arbitrary categories on complex phenomena, just as data extraction in qualitative research can be used selectively to tell a story that is rhetorically convincing but scientifically incomplete.
The problem with presenting qualitative analyses objectively is the sheer volume of data customarily available and the relatively greater difficulty faced by the researcher in summarising qualitative data. It has been suggested that a full transcript of the raw data should be made available to the reader on microfilm or computer disk,11 although this would be cumbersome. Another partial solution is to present extensive sequences from the original data (say, of conversations), followed by a detailed commentary.
Another option is to combine a qualitative analysis with some quantitative summary of the results. The quantification is used merely to condense the results to make them easily intelligible; the approach to the analysis remains qualitative since naturally occurring events identified on theoretical grounds are being counted. The table shows how Silverman compared the format of the doctor's initial questions to parents in a paediatric cardiology clinic when the child was not handicapped with a smaller number of cases when the child had Down's syndrome. A minimum of interpretation was needed to contrast the two sorts of interview.15 16
Assessing a piece of qualitative research
This short paper has shown some of the ways in which researchers working in the qualitative tradition have endeavoured to ensure the rigour of their work. It is hoped that this summary will help the prospective reader of reports of qualitative research to identify some of the key questions to ask when trying to assess its quality. A range of helpful checklists has been published to assist readers of quantitative research assess the design17 and statistical18 and economic19 aspects of individual published papers and review articles.20 Likewise, the contents of this paper have been condensed into a checklist for readers of qualitative studies, covering design, data collection, analysis, and reporting (box). We hope that the checklist will give readers of studies in health and health care research that use qualitative methods the confidence to subject them to critical scrutiny.
Questions to ask of a qualitative study
Overall, did the researcher make explicit in the account the theoretical framework and methods used at every stage of the research?
Was the context clearly described?
Was the sampling strategy clearly described and justified?
Was the sampling strategy theoretically com-prehensive to ensure the generalisability of the conceptual analyses (diverse range of individuals and settings, for example)?
How was the fieldwork undertaken? Was it described in detail?
Could the evidence (fieldwork notes, inter-view transcripts, recordings, documentary analysis, etc) be inspected independently by others; if relevant, could the process of transcription be independently inspected?
Were the procedures for data analysis clearly described and theoretically justified? Did they relate to the original research questions? How were themes and concepts identified from the data?
Was the analysis repeated by more than one researcher to ensure reliability?
Did the investigator make use of quantitative evidence to test qualitative conclusions where appropriate?
Did the investigator give evidence of seeking out observations that might have contradicted or modified the analysis?
Was sufficient of the original evidence pre-sented systematically in the written account to satisfy the sceptical reader of the relation between the interpretation and the evidence (for example, were quotations numbered and sources given)?
Hammersley M. Reading ethnographic research. London: Longman, 1990.