Three techniques for integrating data in mixed methods studiesBMJ 2010; 341 doi: https://doi.org/10.1136/bmj.c4587 (Published 17 September 2010) Cite this as: BMJ 2010;341:c4587
- 1Medical Care Research Unit, School of Health and Related Research, University of Sheffield, Sheffield S1 4DA, UK
- 2University of Leicester, Leicester, UK
- Correspondence to: A O’Cathain
- Accepted 8 June 2010
Health researchers are increasingly using designs that combine qualitative and quantitative methods, and this is often called mixed methods research.1 Integration—the interaction or conversation between the qualitative and quantitative components of a study—is an important aspect of mixed methods research, and, indeed, is essential to some definitions.2 Recent empirical studies of mixed methods research in health show, however, a lack of integration between components,3 4 which limits the amount of knowledge that these types of studies generate. Without integration, the knowledge yield is equivalent to that from a qualitative study and a quantitative study undertaken independently, rather than achieving a “whole greater than the sum of the parts.”5
Barriers to integration have been identified in both health and social research.6 7 One barrier is the absence of formal education in mixed methods research. Fortunately, literature is rapidly expanding to fill this educational gap, including descriptions of how to integrate data and findings from qualitative and quantitative methods.8 9 In this article we outline three techniques that may help health researchers to integrate data or findings in their mixed methods studies and show how these might enhance knowledge generated from this approach.
Researchers will often use qualitative and quantitative methods to examine different aspects of an overall research question. For example, they might use a randomised controlled trial to assess the effectiveness of a healthcare intervention and semistructured interviews with patients and health professionals to consider the way in which the intervention was used in the real world. Alternatively, they might use a survey of service users to measure satisfaction with a service and focus groups to explore views of care in more depth. Data are collected and analysed separately for each component to produce two sets of findings. Researchers will then attempt to combine these findings, sometimes calling this process triangulation. The term triangulation can be confusing because it has two meanings.10 It can be used to describe corroboration between two sets of findings or to describe a process of studying a problem using different methods to gain a more complete picture. The latter meaning is commonly used in mixed methods research and is the meaning used here.
The process of triangulating findings from different methods takes place at the interpretation stage of a study when both data sets have been analysed separately (figure⇓). Several techniques have been described for triangulating findings. They require researchers to list the findings from each component of a study on the same page and consider where findings from each method agree (convergence), offer complementary information on the same issue (complementarity), or appear to contradict each other (discrepancy or dissonance).11 12 13 Explicitly looking for disagreements between findings from different methods is an important part of this process. Disagreement is not a sign that something is wrong with a study. Exploration of any apparent “inter-method discrepancy” may lead to a better understanding of the research question,14 and a range of approaches have been used within health services research to explore inter-method discrepancy.15
The most detailed description of how to carry out triangulation is the triangulation protocol,11 which although developed for multiple qualitative methods, is relevant to mixed methods studies. This technique involves producing a “convergence coding matrix” to display findings emerging from each component of a study on the same page. This is followed by consideration of where there is agreement, partial agreement, silence, or dissonance between findings from different components. This technique for triangulation is the only one to include silence—where a theme or finding arises from one data set and not another. Silence might be expected because of the strengths of different methods to examine different aspects of a phenomenon, but surprise silences might also arise that help to increase understanding or lead to further investigations.
The triangulation protocol moves researchers from thinking about the findings related to each method, to what Farmer and colleagues call meta-themes that cut across the findings from different methods.11 They show a worked example of triangulation protocol, but we could find no other published example. However, similar principles were used in an iterative mixed methods study to understand patient and carer satisfaction with a new primary angioplasty service.16 Researchers conducted semistructured interviews with 16 users and carers to explore their experiences and views of the new service. These were used to develop a questionnaire for a survey of 595 patients (and 418 of their carers) receiving either the new service or usual care. Finally, 17 of the patients who expressed dissatisfaction with aftercare and rehabilitation were followed up to explore this further in semistructured interviews. A shift of thinking to meta-themes led the researchers away from reporting the findings from the interviews, survey, and follow-up interviews sequentially to consider the meta-themes of speed and efficiency, convenience of care, and discharge and after care. The survey identified that a higher percentage of carers of patients using the new service rated the convenience of visiting the hospital as poor than those using usual care. The interviews supported this concern about the new service, but also identified that the weight carers gave to this concern was low in the context of their family member’s life being saved.
Morgan describes this move as the “third effort” because it occurs after analysis of the qualitative and the quantitative components.17 It requires time and energy that must be planned into the study timetable. It is also useful to consider who will carry out the integration process. Farmer and colleagues require two researchers to work together during triangulation, which can be particularly important in mixed methods studies if different researchers take responsibility for the qualitative and quantitative components.11
Following a thread
Moran-Ellis and colleagues describe a different technique for integrating the findings from the qualitative and quantitative components of a study, called following a thread.18 They state that this takes place at the analysis stage of the research process (figure⇑). It begins with an initial analysis of each component to identify key themes and questions requiring further exploration. Then the researchers select a question or theme from one component and follow it across the other components—they call this the thread. The authors do not specify steps in this technique but offer a visual model for working between datasets. An approach similar to this has been undertaken in health services research, although the researchers did not label it as such, probably because the technique has not been used frequently in the literature (box)
An example of following a thread19
Adamson and colleagues explored the effect of patient views on the appropriate use of services and help seeking using a survey of people registered at a general practice and semistructured interviews. The qualitative (22 interviews) and quantitative components (survey with 911 respondents) took place concurrently.
The researchers describe what they call an iterative or cyclical approach to analysis. Firstly, the preliminary findings from the interviews generated a hypothesis for testing in the survey data. A key theme from the interviews concerned the self rationing of services as a responsible way of using scarce health care. This theme was then explored in the survey data by testing the hypothesis that people’s views of the appropriate use of services would explain their help seeking behaviour. However, there was no support for this hypothesis in the quantitative analysis because the half of survey respondents who felt that health services were used inappropriately were as likely to report help seeking for a series of symptoms presented in standardised vignettes as were respondents who thought that services were not used inappropriately. The researchers then followed the thread back to the interview data to help interpret this finding.
After further analysis of the interview data the researchers understood that people considered the help seeking of other people to be inappropriate, rather than their own. They also noted that feeling anxious about symptoms was considered to be a good justification for seeking care. The researchers followed this thread back into the survey data and tested whether anxiety levels about the symptoms in the standardised vignettes predicted help seeking behaviour. This second hypothesis was supported by the survey data. Following a thread led the researchers to conclude that patients who seek health care for seemingly minor problems have exceeded their thresholds for the trade-off between not using services inappropriately and any anxiety caused by their symptoms.
Mixed methods matrix
A unique aspect of some mixed methods studies is the availability of both qualitative and quantitative data on the same cases. Data from the qualitative and quantitative components can be integrated at the analysis stage of a mixed methods study (figure⇑). For example, in-depth interviews might be carried out with a sample of survey respondents, creating a subset of cases for which there is both a completed questionnaire and a transcript. Cases may be individuals, groups, organisations, or geographical areas.9 All the data collected on a single case can be studied together, focusing attention on cases, rather than variables or themes, within a study. The data can be examined in detail for each case—for example, comparing people’s responses to a questionnaire with their interview transcript. Alternatively, data on each case can be summarised and displayed in a matrix8 9 20 along the lines of Miles and Huberman’s meta-matrix.21 Within a mixed methods matrix, the rows represent the cases for which there is both qualitative and quantitative data, and the columns display different data collected on each case. This allows researchers to pay attention to surprises and paradoxes between types of data on a single case and then look for patterns across all cases20 in a qualitative cross case analysis.21
We used a mixed methods matrix to study the relation between types of team working and the extent of integration in mixed methods studies in health services research (table⇓).22 Quantitative data were extracted from the proposals, reports, and peer reviewed publications of 75 mixed methods studies, and these were analysed to describe the proportion of studies with integrated outputs such as mixed methods journal articles. Two key variables in the quantitative component were whether the study was assessed as attempting to integrate qualitative or quantitative data or findings and the type of publications produced. We conducted qualitative interviews with 20 researchers who had worked on some of these studies to explore how mixed methods research was practised, including how the team worked together.
The shared cases between the qualitative and quantitative components were 21 mixed methods studies (because one interviewee had worked on two studies in the quantitative component). A matrix was formed with each of the 21 studies as a row. The first column of the matrix contained the study identification, the second column indicated whether integration had occurred in that project, and the third column the score for integration of publications emerging from the study. The rows were then ordered to show the most integrated cases first. This ordering of rows helped us to see patterns across rows.
The next columns were themes from the qualitative interview with a researcher from that project. For example, the first theme was about the expertise in qualitative research within the team and whether the interviewee reported this as adequate for the study. The matrix was then used in the context of the qualitative analysis to explore the issues that affected integration. In particular, it helped to identify negative cases (when someone in the analysis doesn’t fit with the conclusions the analysis is coming to) within the qualitative analysis to facilitate understanding. Interviewees reported the need for experienced qualitative researchers on mixed methods studies to ensure that the qualitative component was published, yet two cases showed that this was neither necessary nor sufficient. This pushed us to explore other factors in a research team that helped generate outputs, and integrated outputs, from a mixed methods study.
Themes from a qualitative study can be summarised to the point where they are coded into quantitative data. In the matrix (table⇑), the interviewee’s perception of the adequacy of qualitative expertise on the team could have been coded as adequate=1 or not=2. This is called “quantitising” of qualitative data23; coded data can then be analysed with data from the quantitative component. This technique has been used to great effect in healthcare research to identify the discrepancy between health improvement assessed using quantitative measures and with in-depth interviews in a randomised controlled trial.24
We have presented three techniques for integration in mixed methods research in the hope that they will inspire researchers to explore what can be learnt from bringing together data from the qualitative and quantitative components of their studies. Using these techniques may give the process of integration credibility rather than leaving researchers feeling that they have “made things up.” It may also encourage researchers to describe their approaches to integration, allowing them to be transparent and helping them to develop, critique, and improve on these techniques. Most importantly, we believe it may help researchers to generate further understanding from their research.
We have presented integration as unproblematic, but it is not. It may be easier for single researchers to use these techniques than a large research team. Large teams will need to pay attention to team dynamics, considering who will take responsibility for integration and who will be taking part in the process. In addition, we have taken a technical stance here rather than paying attention to different philosophical beliefs that may shape approaches to integration. We consider that these techniques would work in the context of a pragmatic or subtle realist stance adopted by some mixed methods researchers.25 Finally, it is important to remember that these techniques are aids to integration and are helpful only when applied with expertise.
Health researchers are increasingly using designs which combine qualitative and quantitative methods
However, there is often lack of integration between methods
Three techniques are described that can help researchers to integrate data from different components of a study: triangulation protocol, following a thread, and the mixed methods matrix
Use of these methods will allow researchers to learn more from the information they have collected
Cite this as: BMJ 2010;341:c4587
Funding: Medical Research Council grant reference G106/1116
Competing interests: All authors have completed the unified competing interest form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare financial support for the submitted work from the Medical Research Council; no financial relationships with commercial entities that might have an interest in the submitted work; no spouses, partners, or children with relationships with commercial entities that might have an interest in the submitted work; and no non-financial interests that may be relevant to the submitted work.
Contributors: AOC wrote the paper. JN and EM contributed to drafts and all authors agreed the final version. AOC is guarantor.
Provenance and peer review: Not commissioned; externally peer reviewed.