Evaluating complementary medicine: methodological challenges of randomised controlled trialsBMJ 2002; 325 doi: https://doi.org/10.1136/bmj.325.7368.832 (Published 12 October 2002) Cite this as: BMJ 2002;325:832
- Su Mason, honorary principal research fellowa (, )
- Philip Tovey, principal research fellowb,
- Andrew F Long, professorc
- aNorthern and Yorkshire Clinical Trials and Research Unit, Academic Unit of Epidemiology and Health Services Research, University of Leeds, Leeds LS2 9NG,
- bSchool of Healthcare Studies, University of Leeds
- cHealth Care Practice Research and Development Unit, University of Salford, Salford M5 4WT
- Correspondence to: S Mason
- Accepted 10 April 2002
Complementary medicine is increasingly popular for treating many different problems. Doctors and patients need evidence about complementary treatments, but randomised controlled trials need to be carefully designed to take holism into account and avoid invalid results
You think that by understanding one, you can understand two, for one and one is two. But to understand two, you must first understand “and.” Sufi saying1
Complementary medicine should be evaluated as rigorously as conventional medicine to protect the public from charlatans and unsafe practices,2–5 but many practitioners of complementary medicine are reticent about evaluation of their practice. Sceptics maintain that this is because of fear that investigations will find treatments ineffective and threaten livelihoods. In defence, many practitioners argue that research methods dissect their practice in a reductionist manner and fail to take into account complementary medicine's holistic nature leading to invalid evaluation.
Complementary medicine has a different philosophy from conventional medicine, presenting challenges to research methodology
Rigorous evaluation of complementary medicine could provide much needed evidence of its effectiveness
Good design of randomised controlled trials will avoid invalid results and misrepresentation of the holistic essence of complementary medicine
Practitioners need to be recognised as a component in or contributor to complementary treatment
Both specific and non-specific outcome measures with long follow up are needed to adequately encompass the essence of complementary medicine
Nature of complementary medicine
Complementary medicine comprises many different disciplines, a wide spectrum of practices and philosophies which differ from conventional medicine. Conventional medicine traditionally aims to diagnose illness and treat, cure, or alleviate symptoms. Many complementary disciplines aim not only to relieve symptoms and restore wellness but also to help individuals in a process of self healing within a holistic view of health. In this view, individuals are more than just mind, body, and spirit in a social—family or work—environment: as well as promoting wellness, some complementary medicine contains a philosophy that everything is interconnected, and consequently intrinsically bound in a therapeutic relationship between the individual and practitioner.1
Linked to the idea of self healing is a spiritual component: some complementary practitioners believe that illness has a corrective purpose, showing the underlying disharmony in people's lives and enabling identification of areas for change.6 Illness is, in addition, recognised as complex, embracing factors such as genetic predisposition, environment, and diet. Conventional medical relief of symptoms or even “cure” may be considered a temporary respite if the individual does not tackle the underlying cause—for example, by releasing suppressed anger, forgiving, or reducing stress by changing job or being more accepting of circumstances. Complementary treatment aims to be tailored to the individual's stage in life, exploring different underlying causes and solutions.
The challenge for research methodology is to fuse the philosophical concerns of stakeholders with the highest standards of methodological rigour. Taking complementary practitioners' criticisms of randomised controlled trials seriously enables methods to be modified so that the concerns of all stakeholders are taken into account. This ensures that as far as possible what is being assessed under experimental conditions is consistent with everyday complementary practice. In this paper, we consider the core methodological difficulties in assessing complementary medicine and offer practical solutions.7–9 Some evidence from unmodified trials can be rejected as invalid and is therefore ignored; anecdotes from practice are rejected as meaningless. Unless evidence is generated in a way that satisfies all parties, it is unlikely to have an impact on the objective of having evidence based decision making and practice.
Challenges in research methodology
Separation of organisations —Complementary care within European countries is generally separate to conventional medicine. Complementary medicine is not usually integrated into national health systems, and individuals pay for treatment. Funding for trials should include the additional costs of complementary treatment. Additional complexities require attention; these include legal indemnity for the practitioners taking part and assurances of their professional and technical competence.
Generalisability —Because complementary and conventional care are separate, patients recruited to a trial comparing the two may differ from typical complementary patients since they themselves did not decide which treatment they would receive. For example, belief systems or coping mechanisms might differ, affecting treatment outcomes. Incorporating preference in the design of the trial might help tackle and investigate this problem.
Therapeutic expectation—Choosing or preferring a particular complementary discipline carries an associated implicit belief in benefit. Such credibility and expectation of therapeutic gain can bias the results of trials, especially if they cannot be blinded (as in a trial of yoga practice or tai chi). Expectation and perceived credibility should be acknowledged and assessed at randomisation. Several controlled trials have used a credibility scale to check the equivalence of psychological treatments. 10 11
Wide range of symptoms—Complementary medicine and subsequent diagnosis are often not used because of a specific disease. Instead, users may feel tired, lack energy, or be unhappy. A trial about the effectiveness of an intervention based on a disease may not be relevant to complementary treatments. Trials may need to focus on problem areas which are independent of categories of disease. Broad and appropriate outcome measures should capture a wide range of symptoms.
Standardisation—Within a complementary discipline, treatment is individualised and includes care associated with an individual's “diagnosis” and the specific practitioner (skills, competence, manner, etc). The practitioner is explicitly recognised as a component of or contributor to the treatment. This may conflict with ideas of standardisation in conventional trials. Complex treatment interventions can be investigated pragmatically. Standardised treatment can be interpreted as treating to a standard regimen, defined in a protocol, rather than exactly the same treatment. Randomisation should include stratification by individual practitioner.
Influence of practitioner and user —Practitioners often claim that they do not just carry out a set of techniques—for example, the remedy in herbalism or the needles in acupuncture—on the user. They claim to understand how to improve the healing process, the user's role in the healing process, and the therapeutic relationship between the user and the practitioner. All aspects of complementary care should be assessed, and the human experience should be retained as central (not marginalised, as often occurs in reductionist methods).12 If a trial design includes modified therapies so that the holistic aspect is excluded, then this must be explicitly stated.
Controls —Choosing an appropriate comparison group in trials of complementary medicine trials is often problematic. Sham controls—for example, dummy acupuncture to create a placebo controlled comparison—should be treated with caution as they are artificial experiences and distort practice. Any benefit over standard medical treatment should be assessed and appropriate controls adopted. For some complementary treatments—for example, homoeopathy and herbalism—the traditional blinded randomised controlled trial is appropriate. For treatments in which blinding is not possible—for example, meditation—emphasis should be on pragmatic trials with conventional medical practice as controls.
Understanding complementary processes—It is difficult to define precisely which aspect of a complementary treatment provides benefit. During spiritual healing, is it the intent to heal, the preparation, the channelling of “energy,” the accompanying non-judgmental listening to the user, the relationship with the healer, the healing environment, users' expectations or attitudes, or all of these which are important? Complex interventions in conventional medicine result in similar problems.13 Too often trials provide both limited detail on the actual process and little insight into what component of the intervention leads to the effect. Qualitative methods should be used to investigate the process. The design of the trial should address the therapeutic relationship and non-specific effects, in addition to any specific treatment effects.14 At the same time, from the user's perspective, it is the beneficial effect itself that matters, not how it was brought about.
Appropriate outcome measures —Capturing the essence of how complementary medicine might help people is an important challenge. Noteworthy changes after treatment, such as spiritual change or personal growth, might not be measurable. Choice of outcome criteria in the evaluation of complementary therapies “may make a substantial difference to conclusions as to whether a particular intervention or therapy is effective or not.”14 Holism, which is central to the philosophy and practice of much complementary medicine, needs to be better understood. Detailed qualitative studies, accompanying a trial, could elicit these more elusive outcomes and aim to take account of experiential outcome criteria, such as increased self reflection.
Illness role —According to some complementary philosophies, illness indicates an imbalance in the body and symptoms of illness draw attention to this. For example, individuals develop colds, backache, or eczema when they are under stress or run down. If complementary medicine helps people recognise imbalance and adverse lifestyle factors that contribute to illness and promotes change and resultant improvements in health, then research needs to capture these changes.
Chronic illness —Complementary treatment is often sought by people with chronic illnesses who may not have responded to conventional treatment or may want to explore different forms of care. Any change is likely to be gradual and subtle and to have a relapsing and remitting pattern. As complementary interventions tend to aim for long term healing, and not necessarily cure, they are well suited to help people with chronic illness. Funding bodies must understand that long term follow up of outcomes (specific and non-specific) is needed for results to be meaningful. Randomised controlled trials work best for exploring short term effects. Over time, factors other than the trial treatments could bring about the observed effects.
Variations in experience —One person can have different experiences of complementary medicine at different times; different people can have widely different experiences. Designing a trial to investigate a treatment that might be effective immediately for one person but take months to be effective for another is a problem. A “:healing moment”: may occur at the time of treatment or some time after. Appropriate experiential outcome measures should take variations into account along with adequately long follow up periods.
Sick role —Some users do not respond to treatment. There are those who adapt to the sick role because they have an investment in staying ill. Others may be satisfied with the short fix, of feeling peaceful and relaxed, that is offered by many complementary treatments and may not take responsibility to make the necessary changes to improve their health. In principle, randomised controlled trials provide an efficient way to deal with this problem. People adopting the sick role should be distributed equally throughout the arms of a trial through the randomisation process.
Researchers need to tackle the complex issues in researching complementary medicine. Studies should be designed to enable a valid evaluation of complementary medicine, on multiple levels, in the human healing process.
People often turn to complementary treatments because of frustration with conventional medicine.15 They seek that “more”—the essence of what is valued—that complementary treatments offer. Research into complementary medicine has a challenge not to miss the “more” and not to design inappropriately to find it.
Contributors: SM conceived the idea for the paper and wrote the first draft which was rewritten with extensive contributions from PT and AL.
Funding No additional funding.
Conflict of interest SM is a spiritual healer.