Papers

Extent and determinants of error in doctors' prognoses in terminally ill patients: prospective cohort studyCommentary: Why do doctors overestimate?Commentary: Prognoses should be based on proved indices not intuition

BMJ 2000; 320 doi: http://dx.doi.org/10.1136/bmj.320.7233.469 (Published 19 February 2000) Cite this as: BMJ 2000;320:469
  1. Nicholas A Christakis, associate professor (nchrista{at}medicine.bsd.uchicago.edu)a,
  2. Elizabeth B Lamont, fellowb
  1. a Department of Medicine, University of Chicago Medical Center, Chicago, IL 60637, USA
  2. b Robert Wood Johnson Clinical Scholars Program, University of Chicago Medical Center
  3. Hospice of Rochester, Rochester, NY 14607, USA
  4. Chorleywood, Hertfordshire WD3 5AS
  1. Correspondence to: N A Christakis
  • Accepted 5 November 1999

Abstract

Objective: To describe doctors' prognostic accuracy in terminally ill patients and to evaluate the determinants of that accuracy.

Design: Prospective cohort study

Setting: Five outpatient hospice programmes in Chicago

Participants: 343 doctors provided survival estimates for 468 terminally ill patients at the time of hospice referral

Main outcome measures: Patients' estimated and actual survival.

Results: Median survival was 24 days. Only 20% (92/468) of predictions were accurate (within 33% of actual survival); 63% (295/468) were overoptimistic and 17% (81/468) were overpessimistic. Overall, doctors overestimated survival by a factor of 5.3. Few patient or doctor characteristics were associated with prognostic accuracy. Male patients were 58% less likely to have overpessimistic predictions. Non-oncology medical specialists were 326% more likely than general internists to make overpessimistic predictions. Doctors in the upper quartile of practice experience were the most accurate. As duration of doctor-patient relationship increased and time since last contact decreased, prognostic accuracy decreased.

Conclusion: Doctors are inaccurate in their prognoses for terminally ill patients and the error is systematically optimistic. The inaccuracy is, in general, not restricted to certain kinds of doctors or patients. These phenomena may be adversely affecting the quality of care given to patients near the end of life

Introduction

Although doctors commonly have to prognosticate, most feel uncomfortable doing so.1 Neither medical training 17 nor published literature 34 treat prognostication as important, and prognostic error is widespread.2 Unfortunately, prognostic error may have untoward effects on both patient care and social policy.

Parkes showed that doctors' predictions of survival in 168 cancer patients were often erroneous and optimistic,5 and these findings were confirmed by subsequent studies.610 However, previous work has been limited by use of small samples of patients and very small samples of prognosticators (typically fewer than four); failure to examine whether certain types of doctors are more likely to err in certain types of patients; and neglect of the possibility of different determinants of optimistic and pessimistic error. Therefore, we conducted a large, prospective cohort study of terminally ill patients to evaluate the extent and determinants of prognostic error.

Participants and methods

Our cohort consisted of all patients admitted to five outpatient hospice programmes in Chicago during 130 consecutive days in 1996. Participating hospices notified us about patients on admission,and we immediately contacted the referring doctors to administer a four minute telephone survey. Of the 767 patients (referred by 502 doctors), 65 did not meet the entry criteria (they were children, were denied hospice admission, or refused to give consent) and 51 died before we were notified (and thus survival predictions would be meaningless). Of the remaining 651 patients, for 66 (10%) we contacted the doctor only after the patient's death (and so could not get meaningful prognoses),for 14 (2%) the doctor refused to participate, and for 67 (10%) the doctor could not be contacted. We thus completed surveys with 365 different doctors caring for 504 patients (504/651=77%). Comparison of these 504 patients with the 147 excluded patients showed no important differences in patient or doctor characteristics. On 30 June 1999 we had dates of death for 486 of the 504 patients (96%).Because data were occasionally missing, not all totals in the analyses are equivalent.

We obtained the patients' age, sex, race, religion, marital status, diagnosis, and comorbidities from the hospice. From the survey,we obtained an estimate of how long the patient had to live;information about the patient, including Eastern Cooperative Oncology Group performance status11 and duration of illness; information about the doctor, including experience with similar patients and self rated dispositional optimism; and information about the doctor-patient relationship, including the duration, recentness, and frequency of contact. We obtained other data on the doctors, such as specialty,years in practice, and board certification from public records. Dates of patients' deaths were obtained from public death registries or the hospices.

We divided the observed by the predicted survival, and deemed prognoses “accurate” if this quotient was between 0.67 and 1.33. Values less than 0.67 were “optimistic” prognostic errors and those greater than 1.33 were “pessimistic.” We conducted analyses using different cut off points or more categories, as well as analyses that treated this quotient as a continuous measure, but these analyses did not contravene the results presented. To evaluate associations between categorical and continuous variables and the trichotomous prognostic accuracy variable, we used χ2 tests and analysis of variance respectively. We used multinomial logistic regression to assess the multivariate effect of patient and doctor variables on prognostic accuracy.

Results

The patients had a mean age of 69 (SD 17) years and 225/504 (45%)were men. The diagnosis was cancer in 326 (65%), AIDS in 62 (12%),and other conditions in 116 (23%). The mean duration of disease was 83.5 (135.8) weeks, and the median performance status was 3 (corresponding to >50% of the day spent bedridden). The doctors had a median duration of medical practice of 16 years; 291/363 (80%) were men; 293/365 (80%) were board certified; and 255/345 (74%) rated themselves optimistic. A total of 114/358 (32%) specialised in general internal medicine, 71/358 (20%) in non-oncological internal medicine subspecialties, 61/358 (17%) in oncology, 55/358 (15%) in family or general practice, 27/358 (8%) in geriatrics, and 30/358 (8%)were surgeons or practised other specialties. In the past year,the doctors had had experience caring for a median of five patients with the same diagnosis and had referred a median of eight patients to a hospice.They had known the patient an average of 159 (308)weeks; had 11 (14)contacts in the previous three months; and had examined the patient 14 (29) days before.

Doctors' prognostic estimates

In only 18 of 504 patients did the doctor refuse to predict survival to us. Eighteen of the remaining 486 had missing dates of death, leaving 468 cases referred by 343 doctors for analysis of prognostic accuracy. The figure illustrates the extent of the error.The median observed patient survival was 24 days. The mean ratio of predicted to observed survival was 5.3. The correlation between predicted and observed survival was 0.28 (P<0.01). When an accurate prediction was defined as between 0.67 and 1.33 times the actual survival, 20% (92/468) of predictions were accurate, 63% (295/468) optimistic, and 17% (81/468) pessimistic. When an accurate prediction was defined as between 0.50 and 2.0 times the actual survival, 34% (159/468) of predictions were accurate, 55% (256/468)optimistic, and 11% (53/468) pessimistic. Death occurred within one month of the predicted date for 42% (195/468) of patients, at least one month before the predicted date in 46% (214/468), and at least one month after the predicted date in 13% (59/468) of patients.

Figure1

Predicted versus observed survival in 468 terminally ill hospice patients. Diagonal line represents perfect prediction. Patients above diagonal are those in whom survival was overestimated; patients below line are those in whom survival was underestimated

The extent of prognostic error varied depending on both observed and predicted survival (table). The longer the observed survival (that is, the less ill the patient), the lower the error, and, conversely, the longer the predicted survival, the greater the error.

Doctor' overestimates of patient survival by observed and predicted survival

View this table:

Factors associated with prognostic accuracy

Bivariate analyses of the trichotomous accuracy variable and patient attributes showed no important differences with respect to patients' age, sex, race, religion, or marital status.However, cancer patients were the most likely to have overoptimistic predictions (220/301 (67%) v 37/58 (64%) of AIDS patients and 56/109 (51%) of other patients) and the least likely to have overpessimistic predictions (39/301 (13%) v 13/58 (22%) and 29/109 (27%)); AIDS patients were the least likely to have correct predictions (8/58 (14%) v 60/301 (20%)of cancer patients and 24/109 (22%) of others; P<0.01).

Bivariate analyses of the doctor attributes showed no important differences with respect to sex, years in medical practice, board certification, self rated optimism, number of hospice referrals in past year, or number of medically similar patients in the past year.However,doctors in non-oncological medical subspecialties were the least likely to give correct estimates (8/79 (10%) v 11/30 (37%) doctors in surgery or other, 18/76 (27%) in family or general practice, 24/105 (23%) in oncology, and 30/180 (17%) in geriatric or general internal medicine), and oncologists were the least likely to be overpessimistic in their estimates (10/105 (9%)v 21/79 (27%) in other internal medicine subspecialties, 13/67 (19%) in family or general practice,31/180 (17%) in geriatric or general internal medicine, and 4/30 (13%) in surgery or other; P<0.01).

Among the doctor-patient relationship variables (such as length of professional relationship, number of recent contacts, time since last examination), the interval since last examination was important:overpessimistic predictions were associated with the most recent examinations (7.5 days), overoptimistic predictions with the next most recent examinations (13.8 days), and the correct predictions with the longest interval since physical examination (19.5 days);P<0.05.

The trichotomous prognosis variable was regressed on patients' age, sex, race, diagnosis, duration of disease, and performance status and on doctors' experience, sex, optimism, board certification, specialty,related practical experience, duration of relationship, number of contacts, and interval since last examination (full results available on the BMJ's website). The model showed that doctors' prognostic accuracy was independent of most patient and doctor attributes.However, after other attributes were adjusted for, male patients were 58% less likely to have overpessimistic than correct predictions (odds ratio 0.42; 95% confidence interval 0.18 to0.99).Doctors in the upper quartile of practice experience were 63% less likely to make optimistic rather than correct predictions (0.37;0.19 to 0.74) and 78% less likely to make pessimistic rather than correct predictions (0.22; 0.08 to 0.61). Doctors with medical subspecialty training (excluding oncologists) were 3.26 times more likely than geriatricians and general internists to make pessimistic rather than correct predictions (3.26; 1.01 to 10.7). As the duration of the doctor-patient relationship increased, so too did the doctor's odds of making an erroneous prediction—for example, each one year longer that the doctor had known the patient resulted in a 12% increase in the odds of an overpessimistic prediction (1.12; 1.02 to 1.22). Also, as the interval since last physical examination increased, the odds of a doctor making a pessimistic rather than a correct prediction decreased;each day longer resulted in a 3% decrease in the odds (0.97;0.94 to 0.99).

Discussion

Our study of 365 doctors and 504 hospice outpatients found that only 20% of prognoses were accurate. Most predictions (63%) were overestimates, and doctors overall overestimated survival by a factor of about five. These prognoses were doctors' best guesses about their patients' survival prospects, objectively communicated to the investigators and not to patients themselves. Close multivariate examination showed that most doctor and patient attributes were not associated with prognostic error. However, the tendency of doctors to make prognostic errors was lower among experienced doctors. Moreover,the better the doctor knew the patient—as measured, for example, by the length and recentness of their contact—the more likely the doctor was to err.

These findings have several implications. Firstly, undue optimism about survival prospects may contribute to late referral for hospice care,with negative implications for patients.12 13 Indeed,although doctors state that patients should ideally receive hospice care for three months before death,14 patients typically receive only one month of such care.15 The fact that doctors have unduly optimistic ideas about how long patients have to live may partly explain this discrepancy. Doctors who do not realise how little time is left may miss the chance to devote more of it to improving the quality of patients' remaining life. Secondly, to the extent that doctors' implicit or explicit communication of prognostic information affects patients' own conceptions of their future, doctors may contribute to patients making choices that are counterproductive.Indeed, one study found that terminally ill cancer patients who hold unduly optimistic assessments of their survival prospects often request futile, aggressive care rather than perhaps more beneficial palliative care.16 Thirdly, our work hints at corrective techniques that might be used to counteract prognostic error.Disinterested doctors, with less contact with the patient, may give more accurate prognoses, perhaps because they have less personal investment in the outcome.17 Clinicians may therefore wish to seek “second opinions” regarding prognoses, and our work suggests that experienced doctors may be a particularly good source of opinion.Finally, our work suggests that prognostic error in terminally ill patients is rather uniformly distributed. This has implications for doctors' training and self assessment since it suggests that there is not one type of doctor who is prone to error, nor is there one type of patient in whom doctors are likely to err.

What is already known on this topic

Doctors' prognostic estimates are a central element of both patient and physician decision making, especially at the end of life

Doctors' prognostic estimates in their terminally ill patients are often wrong and usually optimistic

What this study adds

A prospective cohort study of 504 terminally ill patients and their 365 doctors found that only 20% of the doctors' predictions were accurate: 63% were overoptimistic and 17% overpessimistic

Multivariate modelling showed that most types of doctors are prone to error, in most types of patients

The greater the experience of the doctor the greater the prognostic accuracy, but a stronger doctor-patient relationship is associated with lower prognostic accuracy

Obtaining prognostic information is often the highest priority for seriously ill patients, eclipsing their interest in treatment options or diagnostic details.18 19 And reliable prognostic information is a key determinant of both doctors' and patients' decision making.16 20 21 Although some error is unavoidable in prognostication, the type of systematic bias towards optimism that we have found in doctors' objective prognostic assessments may be adversely affecting patient care.

Acknowledgments

We thank Elena Linden and Tammy Polonsky for help in administering the survey.

Contributors: NAC initiated the study, conceptualised the key questions, oversaw data collection, assisted in data analysis, and cowrote the paper. EBL conceptualised the key questions, performed most of the data analysis, and cowrote the paper. Both authors will act as guarantors.

Footnotes

  • Funding Soros Foundation Project on Death in America Faculty Scholars Program (NAC), the American Medical association Education and Research Foundation (NAC), and the Robert Wood Johnson Clinical Scholars Program (EBL).

  • Competing interests Both authors have occasionally received honorariums for speaking at events sponsored by hospices

References

  1. 1.
  2. 2.
  3. 3.
  4. 4.
  5. 5.
  6. 6.
  7. 7.
  8. 8.
  9. 9.
  10. 10.
  11. 11.
  12. 12.
  13. 13.
  14. 14.
  15. 15.
  16. 16.
  17. 17.
  18. 18.
  19. 19.
  20. 20.
  21. 21.

Commentary: Why do doctors overestimate?

  1. Julia L Smith, medical director (julia.smith{at}viahealth.org)
  1. a Department of Medicine, University of Chicago Medical Center, Chicago, IL 60637, USA
  2. b Robert Wood Johnson Clinical Scholars Program, University of Chicago Medical Center
  3. Hospice of Rochester, Rochester, NY 14607, USA
  4. Chorleywood, Hertfordshire WD3 5AS

    American doctors refer patients to hospice too late. Christakis and Lamont's research shows that doctors are poor prognosticians and tend to overestimate how long a person who is terminally ill will live.

    Most of the patients in their study had cancer (65%). This is a similar proportion to that found in hospice patients in a survey carried out in 1995 (60%), despite the fact that cancer is not the leading cause of death in the United States. Seven per cent of the patients referred to these hospices died within hours of admission.This eleventh hour referral pattern is at least partly due to doctors not recognising the nearness of death.

    A patient is eligible for hospice care if they have an estimated life expectancy of six months or less. As the authors point out,the actual length of stay is usually less than six weeks. Thus most patients come to hospice during a period of rapid physical change and often in crisis. And they don't live long beyond the crisis.

    At times of crisis, the immediate management of symptoms and relieving the family overshadows the need to address the emotional and spiritual issues of remembering, forgiving, and bringing to closure the issues of a person's life. Provision of a physically comfortable death is a worthy goal. It reduces regrets among survivors. Yet more time provides the opportunity for the dying person to participate directly in the process of validating the past and planning for the future and gives the family the chance to relish or repair bonds with the dying person.The National Hospice Organisation has tried to educate doctors on how to predict appropriate entry points to hospice for various conditions.1 These guidelines should be incorporated into the general education of doctors.

    The authors' suggestion that prognostication should be done by a “disinterested” experienced doctor hits near one common thread of late hospice referrals. Doctors may be reluctant to acknowledge that patients they know well are close to death. This can be compounded by the patient's and family's preference to keep hoping for the patient to live longer. Those of us who know our patients longer often become attached to them. We, too, hate to admit that death is near. I remember a woman in her 60s I was treating for metastatic breast cancer. She was admitted to the hospital with gastric bleeding that was thought to be unrelated to her cancer. I remember talking to her and her husband and being optimistic about the reversibility of the problem.Because I was trying not to scare her I did not discuss the issues of advanced directives and resuscitation. That night she went into shock,required intubation, and went to the intensive care unit. Her husband was devastated and angry that she had had such treatment. The next day he and I together decided that no additional treatment would be given to prevent her death. He sat with her for over 24 hours before she died.My desire to be optimistic prolonged her dying and added anguish to her husband. Doctors often rail against the denial of patients and their families. Yet we are not immune.

    Decisions at the end of life are not just guided by doctors. There is a complex interaction of doctor recognising and acting on accurate prognostication, what the doctor tells the patient and family, and what the patient and family actually hear. Christakis and Lamont have begun to help tease out the factors involved in end of life predictions. In oncology, the end of chemotherapy usually signals the terminal phase.In other specialties, prognosis is much less demarcated.Doctors should better define landmarks or turning points in prognosis and begin to acknowledge these to ourselves and to our patients. Only then can we adequately guide our patients through the dying process.

    Footnotes

    • Competing interests None declared.

    References

    1. 1.

    Commentary: Prognoses should be based on proved indices not intuition

    1. Colin Murray Parkes, consultant psychiatrist (cmparkes{at}aol.com)
    1. a Department of Medicine, University of Chicago Medical Center, Chicago, IL 60637, USA
    2. b Robert Wood Johnson Clinical Scholars Program, University of Chicago Medical Center
    3. Hospice of Rochester, Rochester, NY 14607, USA
    4. Chorleywood, Hertfordshire WD3 5AS

      The accurate prediction of survival is important for several reasons. Excessive optimism may cause us to wait too long to refer people for palliative care, we may delay the use of narcotic drugs for pain relief, and we may persist in unpleasant and pointless treatments aimed at curing or prolonging life when it would be kinder to stop.

      This being the case, it is disappointing to learn from Christakis and Lamont that doctors are still no better at predicting the length of survival of our terminally ill patients than they were 27 years ago.1 Experienced oncologists may be slightly less starry eyed than the rest of us, but even they are overoptimistic about the likely length of survival of their patients. If all predictions had been divided by two they would have been marginally more accurate.

      This failure has to be set against the recent success of research instruments such as Morita's palliative prognostic index2 2-44 and Maltoni's palliative prognostic score,5 5-66 both of which have been shown to predict short term survival reasonably well. These short and simple instruments make use of a mixture of performance measures and systematic symptom assessments rather than relying on intuition and clinical judgment alone.

      In the long term it may be possible to extract from the research those criteria that will enable us to make more reliable clinical predictions. Until that time arrives we would do better to stop guessing and, when predictions are needed, to make use of these indices.

      Footnotes

      • Competing interests None declared.

      • website extra A table showing the results of regression analysis is available on the BMJ's website. www.bmj.com

      References

      1. 1.
      2. 2.
      3. 3.
      4. 4.
      5. 5.
      6. 6.
      View Abstract