Assessing quality in qualitative researchBMJ 2000; 320 doi: https://doi.org/10.1136/bmj.320.7226.50 (Published 01 January 2000) Cite this as: BMJ 2000;320:50
All rapid responses
This letter is an epilogue to the article by Mays and Pope (2000), “Assessing quality in qualitative research”. The authors adapt a subtle realist position to evaluate quality in research. They offer seven different ways of ensuring quality as well as a set of questions to ask of a qualitative research study.
Being a researcher from the subtle realist camp, this article is an interesting and informative read. I have been following the work of several authors who offered distinct guidelines for enhancing quality in different qualitative research methods (for example, Gibbert et al. 2008; Lincoln and Guba, 1985). This letter draws from these prior works and is intended to be a rejoinder to Mays and Pope – basically proposing three more aspects which could be important in assessing the quality of qualitative studies. Based on prior literature, I suggest that a high quality qualitative study should ideally exhibit the three interconnected aspects of methodological originality, theoretical/practical/heuristic significance, as well as timeliness.
A methodologically original study
Qualitative research is dynamic. Various discussions of methodological breakthroughs which could facilitate this dynamism have been going on during the past decade. Innovation in methods is highly prioritized by research funding bodies and journals (Jewitt et al., 2016) as one of the prerequisites of scholarly impact (Berg and Perry, 2006). Being innovative or original brings recognition and esteem to scientists (Guetzkow et al., 2004; Judge et al., 2007). Considering the importance of originality in scientific progress, the application of relevant innovative methods in answering research questions could be considered as one of the criteria for quality. The dimensions of methodological originality could pertain to a study’s research design, way of synthesis, or newness in the discipline.
A significant study
Tracy (2010) describes a significant study as one which extends knowledge, improves existing practices, and/or empowers others. Based on this, I would propose that qualitative researchers should make sure that their study is theoretically, practically, and/or heuristically significant. A theoretically significant study would extend, build, and critique existing theory and offer new understanding of social phenomenon. Practical significance pertains to the study’s ability to provide hands-on solutions to societal issues. A heuristically significant study would develop curiosity in people (Abbott, 2004) and suggesting new arenas for future research. Indeed, it is unrealistic to expect one study to address all dimensions of significance. But being significant in one way or other could be expected of a high quality study.
A timely study
Today qualitative methods are increasingly being incorporated into researching the social, political, or managerial dimensions of contemporary phenomena. This raises a call for studies which are timely. Since data content changes so swiftly, relevant data should be generated, acquired, and utilized in a timely way (McGivray, 2010) to ensure the meaningfulness and usefulness of the study (Cai and Zhu, 2015). Timeliness of the study thus constitutes the third aspect of a quality study.
If implemented systematically, qualitative research has the potential to make contributions to research and society that go beyond mainstream conceptual development. Undeniably, the list of aspects I mention here are not exhaustive. This letter is just an attempt to enable and encourage a more holistic approach to conceptualizing quality in qualitative research, by adding on to the existing exemplar work by the authors.
Abbott, A. (2004). Methods of Discovery: Heuristics for the Social Sciences. New York: W.W. Norton.
Bergh, D. D., & Perry, J. (2006). Some predictors of SMJ article impact. Strategic Management Journal, 27(1), 81-100.
Cai, L., & Zhu, Y. (2015). The challenges of data quality and data quality assessment in the big data era. Data Science Journal, 14 (2), 1-10, DOI: http://dx.doi.org/10.5334/dsj-2015-002.
Gibbert, M., Ruigrok, W., & Wicki, B. (2008). What passes as a rigorous case study? Strategic Management Journal, 29(13), 1465-1474.
Guetzkow, J., Lamont, M., & Mallard, G. (2004). What is Originality in the Humanities and the Social Sciences?. American Sociological Review, 69(2), 190-212.
Jewitt, C., Xambo, A., & Price, S. (2016). Exploring methodological innovation in the social sciences: the body in digital environments and the arts. International Journal of Social Research Methodology, 1-16.
Judge, T. A., Cable, D. M., Colbert, A. E., & Rynes, S. L. (2007). What causes a management article to be cited—article, author, or journal? Academy of Management Journal, 50(3), 491-506.
Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic Inquiry. Newbury Park, CA: Sage Publications.
Mays, N., & Pope C. (2000). Assessing quality in qualitative research. BMJ, 320 (5).
McGilvray, D. (2010) Executing Data Quality Projects: Ten Steps to Quality Data and Trusted Information. Beijing: Publishing House of Electronics Industry.
Tracy, S.J. (2010). Qualitative quality: Eight “big-tent” criteria for excellent qualitative research. Qualitative Inquiry, 16(10) 837–851.
Competing interests: No competing interests
I read with interest Mays and Popes (1) discussion of the issues
surrounding the assessment of qualitative research, citing 'the
proliferation of guidelines for doing and judging qualitative research' as
a manifestation of the concern about assessing quality. I would propose
that many researchers also include 'quality filters' when searching
bibliographic databases in this list.
I have recently completed a survey of qualitative researchers
surrounding their perceptions of the use and relevance of 'quality
filters' when searching electronic databases. One of the main issues
arising from this survey was the tenet that 'quality filters', such as
those developed by the Cochrane Collaboration, McMasters University and
the NHS Centre for Reviews and Dissemination, aim to make assessments on
the basis of the quality of the research study. This is clearly an
impossibility; a misconception bound up with the use of terminology and
those associated with the received evidence hierarchy.
The term 'quality filter' is misleading in two respects. Firstly, it
suggests a filtering out or sifting of potentially irrelevant - perhaps
poor quality - materials. In reality the search strategies aim to be more
inclusive, aiming to identify journal papers more accurately by overcoming
the limitations of database indexing, such as the misapplication or non-
application of appropriate medical subject headings (MESH headings).
Secondly, there is the issue of quality. To date the majority of
published search strategies have been developed to overcome indexing
problems for randomised controlled trials, systematic reviews etc. Given
their position at the top end of the received evidence hierarchy, there is
often an implicit assumption made that these research methods are of a
'gold standard'. Thus, having identified studies of these types, there is
an implication that little appraisal is assumed to be necessary because
studies of a lesser quality, studies of a less 'rigorous' nature, have
been excluded. However, quality assessments remain of paramount
importance in evaluating the appropriateness of study design, the basis of
conclusions etc. wherever the study appears in the evidence hierarchy.
As such, I believe there is a serious need for the reconsideration of
the terminology used in relation to search strategies developed to
overcome deficiencies in database indexing. The wide spread adoption of
the term 'optimal search strategy' would be a more appropriate and more
accurately representative term than ‘quality filter’.
Only then, can the true nature of quality assessments be recognised.
Maria J Grant
Health Care Practice R&D Unit
University of Salford
Humphrey Booth House
Tel: 0161 295 0454
1. Mays N, Pope C. Qualitative research in healthcare: assessing
quality in qualitative research. British Medical Journal 2000; 320: 50-52.
Competing interests: No competing interests
We appreciated the paper by Mays and Pope but were struck by their
indulgent treatment of the antirealist position.(1) Antirealist
qualitative researchers contend there is no "social" reality or truth that
is independent of the observer. Antirealists, a species of post-
modernists, scoff at those naive enough to believe in the physical
reality of social world: "what the parochial view in the social,
behavioral, and service sciences has touted as ‘science' is historical and
practical myth...."(2) Presumably "social" reality consists of the
interactions of human beings, i.e., spoken or written words, and all human
actions that relate to other humans. Thus the antirealists apparently
would contend that this letter has no reality. Antirealists thus fall
headlong into self-contradiction. If no utterances, presumably including
their own, have reality, why should one read what they write? Further,
why should one pay any attention to the work of supposedly "scientific"
researchers who deny the independent reality of what they research?
The antirealist view seems to be at best an excuse for sloppy work.
Antirealists have argued that bias in research is good, "not something to
be eliminated, but is a productive element, a foundation for formulating
questions and understanding answers in the process of research."(3) They
have asserted that the traditional notions of methodologic rigor, "the
classical cannons of reliability, validity, and objectivity," are
irrelevant to their kind of qualitative research, to the point that a
"powerful case can be made for methodological anarchy...!"(4) In
retreating to ancient subjectivist attitudes, antirealists have renounced
the rigor, self-discipline, humility in the face of evidence, and
willingness to risk failure and blind alleys that the scientific attitude
Alan Sokal, the physicist whose parody of post-modernism in science
received wide attention, said it well. First he decried "a particular
kind of nonsense and sloppy thinking: one that denies the existence of
objective reality."(5) Then he wrote "intellectually, the problem with
such doctrines is that they are false. There is a real world; its
properties are not merely social constructions; facts and evidence do
matter. What sane person would contend otherwise?"
We applaud the efforts of Pope and May to bring more rigor to the
design, execution, and review of qualitative research. We fear, however,
that responsible qualitative researchers will have trouble convincing
others of the value of their field until they disavow the pseudo-
philosophic nonsense of antirealism.
Roy M. Poses MD
Brown University Center for Primary Care and Prevention
Pawtucket, RI, USA
Norman J. Levitt PhD
Department of Mathematics,
New Brunswick, NJ, USA
1. Mays N, Pope C. Assessing quality in qualitative research. Brit
Med J 2000; 320: 50-52.
2. Gubrium JF. Qualitative research comes of age in gerontology.
Gerontologist 1992: 32: 581-582.
3. Berkwits M, Aronowitz R. Different questions beg different
methods. J General Internal Medicine 1995; 10: 409-410.
4. Henwood KL, Pidgeon NF. Qualitative research and psychological
theorizing. Br J Psychology 1992;83:97-111.
5. Sokal A. A physicist experiments with cultural studies. Lingua
Franca 1996; 6: 62-64.
Competing interests: No competing interests