BMJ 1998;316:1701-1705 ( 6 June )

Papers

Reliability of league tables of in vitro fertilisation clinics: retrospective analysis of live birth rates

See Editor's choice

E Clare Marshall, research student,  David J Spiegelhalter, senior statistician. 

MRC Biostatistics Unit, Institute of Public Health, Cambridge CB2 2SR

Correspondence to: Dr Spiegelhalter david.spiegelhalter{at}mrc-bsu.cam.ac.uk

Objective: To determine to what extent institutions carrying out in vitro fertilisation can reasonably be ranked according to their live birth rates.
Design: Retrospective analysis of prospectively collected data on live birth rate after in vitro fertilisation.
Setting: 52 clinics in the United Kingdom carrying out in vitro fertilisation over the period April 1994 to March 1995.
Main outcome measure: Estimated adjusted live birth rate for each clinic; their rank and its associated uncertainty.
Results: There were substantial and significant differences between the live birth rates of the clinics. There was great uncertainty, however, concerning the true ranks, particularly for the smaller clinics. Only one clinic could be confidently ranked in the bottom quarter according to this measure of performance. Many centres had substantial changes in rank between years, even though their live birth rate did not change significantly.
Conclusions: Even when there are substantial differences between institutions, ranks are extremely unreliable statistical summaries of performance and change in performance, particularly for smaller institutions. Any performance indicator should always be associated with a measure of sampling variability.

Key messages Institutional ranks are extremely unreliable statistical summaries of performance Institutions with smaller numbers of cases may be unjustifiably penalised or credited in comparison exercises Additional statistical analysis may help to identify the few institutions worthy of review Any performance indicator should always have an associated statistical sampling variability

---

© BMJ 1998
CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    What's this?

Relevant Articles

Comparison of hospital episode statistics and central cardiac audit database in public reporting of congenital heart surgery mortality

Stephen Westaby, Nicholas Archer, Nicola Manning, Satish Adwani, Catherine Grebenik, Oliver Ormerod, Ravi Pillai, and Neil Wilson
BMJ 2007 335: 759. [Abstract] [Full Text] [PDF]

Paediatric cardiac surgical mortality after Bristol: Details of risk adjustment tools were not given

Gareth Parry, Elizabeth S Draper, and Patricia McKinney
BMJ 2005 330: 43. [Extract] [Full Text]

League tables of in vitro fertilisation clinics misinform patients

Robert Winston
BMJ 1998 317: 1593. [Extract] [Full Text]

Satisfaction with nurse specialists in breast care clinics

J M Dixon, J Lamb, G Stones, A Rahman, D Mitchell, M Bramley, G J Byrne, N J Bundred, L Garvican, P Littlejohns, and N P M Sacks
BMJ 1998 317: 1316. [Extract] [Full Text] [PDF]

The dark side of medicine
BMJ 1998 316: 0. [Full Text]

Too much attention should not be paid to ranking in league tables
BMJ 1998 316: 0. [Full Text]

Lessons from the Bristol case

Tom Treasure
BMJ 1998 316: 1685-1686. [Extract] [Full Text] [PDF]

This article has been cited by other articles:

• Castilla, J. A., Hernandez, J., Cabello, Y., Lafuente, A., Pajuelo, N., Marqueta, J., Coroleu, B., (Assisted Reproductive Technology Register of the, (2008). Defining poor and optimum performance in an IVF programme. Hum Reprod 23: 85-90 [Abstract] [Full text]  
• Moser, K., Frost, C., Leon, D. A (2007). Comparing health inequalities across time and place rate ratios and rate differences lead to different conclusions: analysis of cross-sectional data from 22 countries 1991 2001. Int J Epidemiol 36: 1285-1291 [Abstract] [Full text]  
• Westaby, S., Archer, N., Manning, N., Adwani, S., Grebenik, C., Ormerod, O., Pillai, R., Wilson, N. (2007). Comparison of hospital episode statistics and central cardiac audit database in public reporting of congenital heart surgery mortality. BMJ 335: 759-759 [Abstract] [Full text]  
• Lutfiyya, M. N., Bhat, D. K., Gandhi, S. R., Nguyen, C., Weidenbacher-Hoper, V. L., Lipsky, M. S. (2007). A comparison of quality of care indicators in urban acute care hospitals and rural critical access hospitals in the United States. Int J Qual Health Care 19: 141-149 [Abstract] [Full text]  
• Anderson, J., Hackman, M., Burnich, J., Gurgiolo, T. R. (2007). Determining Hospital Performance Based on Rank Ordering: Is It Appropriate?. American Journal of Medical Quality 22: 177-185 [Abstract]  
• Lemmers, O., Kremer, J. A.M., Borm, G. F. (2007). Incorporating natural variation into IVF clinic league tables. Hum Reprod 22: 1359-1362 [Abstract] [Full text]  
• Huang, I-C., Dominici, F., Frangakis, C., Diette, G. B., Damberg, C. L., Wu, A. W. (2005). Is Risk-Adjustor Selection More Important Than Statistical Approach for Provider Profiling? Asthma as an Example. Med Decis Making 25: 20-34 [Abstract]  
• Parry, G., Draper, E. S, McKinney, P. (2005). Paediatric cardiac surgical mortality after Bristol: Details of risk adjustment tools were not given. BMJ 330: 43-43 [Full text]  
• Rogers, C. A., Reeves, B. C., Caputo, M., Ganesh, J. S., Bonser, R. S., Angelini, G. D. (2004). Control chart methods for monitoring cardiac surgical performance and their interpretation. J. Thorac. Cardiovasc. Surg. 128: 811-819 [Full text]  
• Aylin, P., Bottle, A., Jarman, B., Elliott, P. (2004). Paediatric cardiac surgical mortality in England after Bristol: descriptive analysis of hospital episode statistics 1991-2002. BMJ 329: 825- [Abstract] [Full text]  
• Donnan, P T, Wei, L, Steinke, D T, Phillips, G, Clarke, R, Noone, A, Sullivan, F M, MacDonald, T M, Davey, P G (2004). Presence of bacteriuria caused by trimethoprim resistant bacteria in patients prescribed antibiotics: multilevel model with practice and individual patient data. BMJ 328: 1297- [Abstract] [Full text]  
• de Leval, M. R (2004). Facing up to surgical deaths. BMJ 328: 361-362 [Full text]  
• McKee, M. (2004). Not everything that counts can be counted; not everything that can be counted counts. BMJ 328: 153-153 [Full text]  
• Tekkis, P. P, Poloniecki, J. D, Thompson, M. R, Stamatakis, J. D (2003). Operative mortality in colorectal cancer: prospective national study. BMJ 327: 1196-1201 [Abstract] [Full text]  
• HOWLEY, P. P., GIBBERD, R. (2003). Using hierarchical models to analyse clinical indicators: a comparison of the gamma-Poisson and beta-binomial models. Int J Qual Health Care 15: 319-329 [Abstract] [Full text]  
• Tekkis, P. P, McCulloch, P., Steger, A. C, Benjamin, I. S, Poloniecki, J. D (2003). Mortality control charts for comparing performance of surgical units: validation study using hospital mortality data. BMJ 326: 786-788 [Abstract] [Full text]  
• Powell, A E, Davies, H T O, Thomson, R G (2003). Using routine comparative data to assess the quality of health care: understanding and avoiding common pitfalls. Qual Saf Health Care 12: 122-128 [Abstract] [Full text]  
• Sharif, K., Afnan, M. (2003). The IVF league tables: time for a reality check. Hum Reprod 18: 483-485 [Abstract] [Full text]  
• Lindstrom, M, Moghaddassi, M, Merlo, J (2003). Social capital and leisure time physical activity: a population based multilevel analysis in Malmo, Sweden. J. Epidemiol. Community Health 57: 23-28 [Abstract] [Full text]  
• Miles, H, Litton, E, Curran, A, Goldsworthy, L, Sharples, P, Henderson, A J (2002). The PATRIARCH Study. Using outcome measures for league tables: Can a North American prediction of admission score be used in a United Kingdom children's emergency department?. Emerg. Med. J. 19: 536-538 [Abstract] [Full text]  
• Dorsch, M F, Lawrance, R A, Sapsford, R J, Oldham, J, Greenwood, D C, Jackson, B M, Morrell, C, Ball, S G, Robinson, M B, Hall, A S (2001). A simple benchmark for evaluating quality of care of patients following acute myocardial infarction. Heart 86: 150-154 [Abstract] [Full text]  
• Localio, A. R., Berlin, J. A., Ten Have, T. R., Kimmel, S. E. (2001). Adjustments for Center in Multicenter Studies: An Overview. ANN INTERN MED 135: 112-123 [Abstract] [Full text]  
• Merlo, J, Östergren, P-O, Broms, K, Bjorck-Linné, A, Liedholm, H (2001). Survival after initial hospitalisation for heart failure: a multilevel analysis of patients in Swedish acute care hospitals. J. Epidemiol. Community Health 55: 323-329 [Abstract] [Full text]  
• McKinley, R. K, Fraser, R. C, Baker, R. (2001). Model for directly assessing and improving clinical competence and performance in revalidation of clinicians. BMJ 322: 712-715 [Full text]  
• Steyerberg, E. W., Ivanov, J., Tu, J. V., Naylor, C. D., Krumholz, H. M. (2000). Ranking of Surgical Performance Response Response. Circulation 102 : e61-e62 [Full text]  
• Giuffrida, A., Gravelle, H., Roland, M. (1999). Measuring quality of care with routine data: avoiding confusion between performance indicators and health outcomes. BMJ 319: 94-98 [Abstract] [Full text]  
• Steel, C M., Jackson, D., Sinclair, D. W, Magee, S. R, Levison, D A, Parratt, D, Bland, J M, Gore, S. M, McManus, C. (1999). Selection to medical school in Great Britain. BMJ 318: 937a-937 [Full text]  
• Winston, R. (1998). League tables of in vitro fertilisation clinics misinform patients. BMJ 317: 1593-1593 [Full text]  
• Dixon, J M, Lamb, J, Stones, G, Rahman, A, Mitchell, D, Bramley, M, Byrne, G J, Bundred, N J, Garvican, L, Littlejohns, P, Sacks, N P M (1998). Satisfaction with nurse specialists in breast care clinics. BMJ 317: 1316-1316 [Full text]