Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study
BMJ 2009; 339 doi: https://doi.org/10.1136/bmj.b3496 (Published 10 September 2009) Cite this as: BMJ 2009;339:b3496All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
Lewin and colleagues show the lack of attention paid to integration
of the qualitative research undertaken alongside randomised controlled
trials of organisational interventions, making the point that researchers
may be failing to exploit the potential of mixed methods research.1 This
has also been found when qualitative methods are used alongside trials of
other types of interventions and other quantitative methods.2 We undertook
a study of 75 mixed methods projects in health services research in
England and found that researchers may not integrate data or findings from
different components of their study, may not publish qualitative
components, or may publish each component in separate journals, sometimes
without reference to each other.2 This breaking up of a study into its
methodological pieces makes it hard for health professionals, policy
makers and patients to identify what can be learnt from a single research
project. Separate publication is often necessary due to the word limit of
journals but some researchers have found a way of publishing integration
within this constraint. A good model is the publication of one component
of a study first and then the publication of the second component can draw
on findings from the first publication within its introduction, results or
discussion to aid analysis or interpretation.3 We also interviewed 20
researchers and found that the way researchers use, integrate and report
methods is affected by their reasons for undertaking a mixed methods study
in the first place, how the researchers interact as a team, and structural
issues. Some researchers undertake mixed methods studies for strategic
reasons rather than the intrinsic value of combining methods e.g. they add
a qualitative component because they perceive that funding bodies will
like this, or to facilitate publication if the trial findings are null.4
Some research teams work as separate qualitative and quantitative teams in
a multidisciplinary fashion which leads to a lack of integration between
components; others meet frequently to share findings and interpretations
in an interdisciplinary way which can lead to publication of knowledge
accessed through integration of components.5 How researchers report their
studies can be determined by templates available within their community.6
If researchers see stand alone publications of different components of
studies then this perpetuates a lack of integration within studies. This,
along with other structural issues such as the lack of formal education
around ways of integrating data and findings, contributes to researchers
sometimes failing to deliver the promise of combining methods.7 Lewin and
colleagues also express concern about the quality of qualitative
components, and this too is relevant to wider mixed methods studies.8,9
This is by no means as depressing as it sounds. When we undertook our
study of mixed methods research, we found inspiring examples of
integration and publication of mixed methods studies. There is a need to
communicate these exemplars to the research community because we all learn
by seeing things done well.
1. Lewin S, Glenton C, Oxman DA. Use of qualitative methods alongside
randomised controlled trials of complex healthcare interventions:
methodological study. BMJ 2009; 339:b3496.
2. O'Cathain A, Murphy E, Nicholl J. Integration and publications as
indicators of 'yield' from mixed methods studies Journal of Mixed Methods
Research 2007;1(2):147-163.
3. Donovan JL, Peters TJ, Noble S, Powell P, Gillat D, Oliver SR, et
al. Who can best recruit to randomised trials? Randomised trial comparing
surgeons and nurses recruiting patients to a trial of treatments for
localized prostate cancer (the ProtecT study). Journal of Clinical
Epidemiology 2003;56:605-609.
4. O'Cathain A, Murphy E, Nicholl J. Why, and how, mixed methods
research is undertaken in health services research: a mixed methods study.
BMC Health Services Research 2007;7:85.
5. O'Cathain A, Murphy E, Nicholl J. Multidisciplinary,
interdisciplinary or dysfunctional? Team working in mixed methods
research. Qualitative Health Research 2008;18(11):1574-1585.
6. O'Cathain A. Reporting results. In: Andrew S, Halcomb E, editors.
Mixed methods research for nursing and the health sciences Blackwell
Publishing, 2009, pp 135-158.
7. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M.
Developing and evaluating complex interventions: the new Medical Research
Council guidance. BMJ 2008;337:a1655.
8. O'Cathain A, Murphy E, Nicholl J. The quality of mixed methods
studies in health services research. Journal of Health Services Research
and Policy 2008;13(2):92-98.
9.Pope C, Mays N. Critical reflections on the rise of qualitative
research. BMJ 2009;339:b3425.
Competing interests:
None declared
Competing interests: No competing interests
Why are qualitative approaches not used appropriately in clinical trials?
Lewin and colleagues raise timely concerns about why qualitative
approaches are not used appropriately in clinical trials, when there is
growing recognition of their important role in the design and evaluation
of interventions. Similarly, they suggest that when qualitative studies
are used they tend to be of poor quality and their integration within
trials rather limited, adding little, if any significance to trial
outcomes.
Why has this situation arisen? Study authors cited poor access to
relevant expertise as a reason and a further clue lies hidden in table 1
of Lewin et.al’s paper. 1 Hypothesis generation and developing theory are
key tenets of qualitative research (reasons 2, 10 and 11 in Table 1). Yet
in only 1 of the 23 trials were these reasons cited for including
qualitative research in the trial. This appears to support research
suggesting that qualitative research can be undertaken for strategic
reasons rather than for its intrinsic value. 2
Our experience as qualitative researchers working in mixed methods
studies suggests that part of the problem may be related to how well
trained the researchers are. Attending one or two courses in qualitative
methods is unlikely to ensure qualitative rigor in sampling and the
selection of a theoretical base upon which to pin a conceptual analysis
which generates the interpretative conclusions found in high quality
qualitative research. This concealed problem becomes chronic, as the
poorly trained go on to supervise other new qualitative researchers, thus
perpetuating the case of the poorly sighted leading the blind. The
chronicity becomes acute when blame is attributed to the
‘unscientificness’ of the qualitative product, and what it has to offer to
the evidence-base, rather than the inadequate training of the individuals
using it.
Why do researchers, poorly trained in qualitative research, feel the
need to use qualitative methodology in the way they might not be so
tempted to do with other methods, such as statistics and economics? Are
commissioning bodies, and the current trend towards multidisciplinary
research, partly to blame and if so, what is it about the qualitative
approach that makes it seem such a ‘free for all’?
In addition to more research into the barriers of integrating
qualitative research into trials, we believe that there is an urgent need
for more rigorous qualitative methods training to ensure quality assurance
and, as O’Cathain suggests, utilise the full yield of qualitative
approaches in trials. 1 & 2
References
1. Lewin S, Glenton C, Oxman DA. Use of qualitative methods alongside
randomised controlled trials of complex healthcare interventions:
methodological study. BMJ 2009; 339:b3496.
2. O'Cathain A, Murphy E, Nicholl J. Why, and how, mixed methods research
is undertaken in health services research: a mixed methods study. BMC
Health Services Research 2007;7:85.
Competing interests:
None declared
Competing interests: No competing interests