Ten steps towards improving prognosis research
==============================================

* Harry Hemingway
* Richard D Riley
* Douglas G Altman

Prognosis research should be a basic science in translational medicine, but methodological problems mean systematic reviews are unable to reach firm conclusions. **Harry Hemingway and colleagues** recommend action to improve the quality

Stemming the tide of low quality, low impact, prognosis research is an urgent priority for the medical and research community. Diverting currently wasted research resources into high quality prognosis research will require major changes, one of which is the implicit collusion between researchers, medical journal editors, and conference organisers: “If you agree to inflate the importance of your research, we will agree to showcase it.” We outline challenges facing prognosis research, and possible next steps, drawing on recent evidence from different clinical specialties and study designs.

## Problems with prognosis research

Prognosis research has been defined as the study of relations between occurrences of outcomes and predictors in defined populations of people with disease.1 It encompasses (ideally) prospective, observational research evaluating three broad questions—causes of disease progression, prediction of risk in individuals, and individual response to treatment. High quality prognosis research results in better understanding of disease progression, offers improved opportunities for mitigating that progression, and allows more reliable communication of outcome risk to patients.1 2 Prognosis research should be a basic science in translational medicine.

Analysing 168 reports, Malats and colleagues concluded that “after 10 years of research [including over 10 000 patients], evidence is not sufficient to conclude whether changes in P53 act as markers of outcome in patients with bladder cancer.”3 This is not an isolated example. Such concerns have been identified in systematic reviews of different types of prognostic biomarkers4 5 and across different clinical specialties and major global burdens of disease including cancer,6 coronary disease,7 stroke,8 trauma,9 and musculoskeletal disorders.10 11 Although some systematic reviews and meta-analyses of prognostic studies do reach clear conclusions, not all pay attention to the quality of the primary studies.12 13

It is inconceivable that 168 randomised controlled trials could fail to reach an answer on the effectiveness of an intervention. Why does the scientific community generate, and apparently tolerate, prognosis research with such limitations? Here we identify 10 areas where specific actions (table⇓) might make investments in prognosis research more effective (in terms of generating reliable new knowledge with benefits for patient outcomes) and more efficient (less redundant or misleading research).

View this table:
[Table 1](http://www.bmj.com/content/339/bmj.b4184/T1)

Table 1 
 Ten challenges facing prognostic research

## Purpose

In the absence of an accepted classification used across clinical specialties, we need to clarify the goals of prognosis research and thereby provide a framework with which to assess progress. Standard nomenclature is urgently needed. Broadly, three aims can be recognised: 

*   Identification of single biomarkers that have independent associations with outcome (relating to the causal pathway)

*   Development of multivariable risk prediction models (risk score or prognostic index) that predict an individual’s outcome, and 

Such a taxonomy could identify different goals at different stages in the translation of emerging putative prognostic biomarkers from the laboratory to the bedside. For example, early prognostic studies may aim at discovering possible prognostic biomarkers and will tolerate false positive results; later studies may evaluate the probability that such biomarkers are useful (in risk prediction models) and seek to minimise false positive results.14 Existing systematic reviews of prognostic biomarkers suggest that current prognosis research concentrates on the first goal. For example, in a systematic review of the prognostic value of C reactive protein on the prognosis of stable coronary disease,7 only three of the 77 studies reported a measure of its ability to discriminate risk in individuals.

A greater appreciation of the distinction between the three goals is required. For instance, it is wrong to assume that a biomarker that is (causally) related to incidence of disease (aetiology) is necessarily (causally) related to progression (prognosis). For example, body mass index is associated in aetiological studies with onset of coronary disease but not with subsequent fatal and non-fatal events among people with coronary disease.15 Risk prediction models are easy to produce, hard to validate,16 and harder still to implement in clinical practice. And, thus far, evidence of impact on decision making or prognosis is nearly always lacking.17 The next generation of such models needs to tackle these problems.

## Funding

Prognosis research has attracted much less funding than diagnostic and therapeutic research. As a crude marker of this, a search of the website of the US National Institutes of Health, globally one of the largest funding bodies, returns about 132 000 hits for the term “diagnostic,” 76 000 for “trial,” and only 4000 for “prognostic.” Indeed one reason for the large number of small, poor quality prognostic studies may be that many are conducted without peer reviewed external sources of funding. A “what’s in the freezer?” approach has been too common,18 in which the investigator apparently argues: given the data we already have, what abstract can be produced to allow a junior colleague to present at a conference. For example, in cancer biomarker studies, Kyzas and colleagues suggest “investigators may tend to conduct opportunistic studies on the basis of specimen availability rather than on thoughtful design.”19 Such an approach perpetuates poor quality research. 

Funders need a strategic framework to guide investment across complementary study designs. This will enable them to judge when it is best to set up bespoke investigator-led prognostic cohorts, to add biomarker or other measurements to existing clinical cohort collections (including registry data), to exploit linkages between different electronic health records (such as in primary care and disease registries), or to stimulate meta-analysis of data on individual participants.20

## Protocols

All research on humans should have a protocol,21 22 yet many current prognosis research studies seem not to be protocol driven. Most prognostic studies are retrospective in the sense that the investigator decides which analyses to do after the data have been collected. Just four of the 77 studies in the C reactive protein systematic review referred to a previously written study protocol.7 Thus the reader does not know whether the analyses were part of the rationale for entering patients into the study or were prespecified in a statistical analysis plan and there is large potential for selective and biased reporting. It should become mandatory for prognosis research to have a registered study protocol outlining the aims and detailing the methods of data collection and statistical analysis that will be used.23 Study registration and publication of analytical and study protocols should also help improve the quality of studies.

## Predictors

Given the wide range of factors that may influence prognosis—the social and healthcare environment, psychosocial factors, health behaviours, and biological factors**—**why is the focus of prognosis research so uneven? The “mile wide, inch deep” focus on circulating biomarkers is illustrated in a systematic review of 130 different factors in neuroblastoma in which the median number of publications per biomarker was 1 (fig 1⇓).24 By contrast, the prognostic importance of history, examination, and simple investigations has been relatively neglected.25 For example, whereas meta-analyses have examined the relation between alcohol consumption in initially healthy populations and subsequent death from coronary disease,26 there has been little research into the relation between alcohol and prognosis among people with cardiovascular disease,27 and no meta-analyses. This is a clinically important question because doctors need evidence on which to base advice to patients and a framework to evaluate new prognostic biomarkers in the context of existing knowledge. There is a need for clarity over the strength of evidence required for prognostic biomarkers to be considered “established” or “useful.”

![Figure1](http://www.bmj.com/https://www.bmj.com/content/bmj/339/bmj.b4184/F1.medium.gif)

[Figure1](http://www.bmj.com/content/339/bmj.b4184/F1)

**Fig 1** Mile wide inch deep focus of research shown by systematic review of studies of genetic and other circulating biomarkers for recurrence or death from neuroblastoma.24 130 different biomarkers were studied with a median of one study per marker

## Outcomes

Most prognosis research in cancer and cardiovascular disease fails to report suffering from symptoms, functional status, and quality of life. Mortality may not be the most important outcome to the patient, nor is it necessarily a good proxy for other outcomes. Patient values are a constituent, not a contingent, property of a full understanding of prognosis. Assessments of the impact of a particular disease on a patient’s life vary widely among patients, and are commonly discordant with the severity assessed by doctors.28

As most prognostic studies examine multiple outcomes, selective reporting, where only those outcomes found to be statistically or clinically significant are reported, is a concern. Selective reporting is a problem in cancer prognostic studies,29 but is likely to be prevalent in other fields too. This problem underscores the need for study and protocol registration, with pre-specification of the primary outcomes of interest.

## Methods

Prognosis research must catch up with the standards of high quality randomised trials or observational aetiological research, in terms of design, conduct, analysis, and reporting.20 Many studies are simply too small to provide reliable evidence—for example, a meta-analysis of 47 studies among patients with Barrett’s oesophagus reported a total of just 209 incident cases of oesophageal cancer (fig 2⇓).30 Prognosis research needs to be seen as a distinct field in order to foster scientifically justified, rather than idiosyncratic, methods. For example, in cancer research continuous biomarkers are almost always dichotomised, whereas in cardiovascular research this is much less common. 

![Figure2](http://www.bmj.com/https://www.bmj.com/content/bmj/339/bmj.b4184/F2.medium.gif)

[Figure2](http://www.bmj.com/content/339/bmj.b4184/F2)

**Fig 2** Systematic review of 47 studies investigating incidence of oesophageal cancer in patients with Barrett’s oesophagus found that most were too small to provide reliable evidence.30 Larger studies tended to show lower incidence of cancer

Hayden and colleagues list six stages in the design, conduct, and analysis of prognosis studies where bias may operate,31 but most primary studies inadequately protect against these threats to validity.10 

## Publication

A prudent default position would be to assume that prognosis research is seriously afflicted by publication bias, until there is evidence to the contrary. Evaluating 1575 articles on different prognostic biomarkers for cancer, Kyzas and colleagues found that almost all report significant results,6 signalling a major problem of publication bias. The C reactive protein systematic review found that publication bias was so large that different methods to adjust for its effect either substantially attenuated, or abolished, the apparent association between C reactive protein and outcome.7 Study and protocol registration would help with this problem because it would make it easier to identify unpublished studies.

## Reporting

Authors of prognosis research articles often omit key details, outcomes, and analyses and inflate the importance of their findings.32 Currently there are no generic reporting guidelines for prognosis research, which means that journal editors, peer reviewers, authors, and readers do not have a framework for distinguishing reliable observations from the merely new. An important start has been made by the REMARK guidelines for biomarkers in cancer,33 though lack of adherence to these guidelines has recently been noted.34 Prognostic studies share many methodological features with healthy population studies, but require reporting of additional items, such as the initial medical condition, its stage, and duration since onset; the translational clinical question examined; absolute risks; and the clinical outcomes that are more varied than the singular end points used in aetiological studies.

Importantly, there are currently no reporting standards for risk prediction scores,9 nor any central register where clinicians and researchers can access and compare these rapidly expanding technologies.35 We propose that reporting guidelines are developed that span the scope of prognosis research (perhaps using REMARK as a starting point). As a related but distinct exercise a checklist of quality criteria should be developed.

## Synthesis

Given the concerns about the quality of primary prognosis research, efforts at evidence synthesis should be viewed with caution. Evaluating 17 systematic reviews in the prognosis of low back pain, Hayden and colleagues concluded that “because of the methodological shortcomings . . . there remains uncertainty about the reliability of conclusions regarding prognostic factors.”10 Such a conclusion is common for systematic reviews of prognostic studies. The Cochrane Collaboration Prognosis Methods Group, established in 2008, aims to facilitate and improve the quality of systematic reviews of prognosis research.36 

Developments are required at multiple stages, starting with improvements in primary studies and working towards improving methodology for synthesis. Remarkably, electronic searches of publications on PubMed cannot distinguish studies among people with disease from studies among healthy people who go on to develop disease. We therefore need a standardised nomenclature for describing the results of studies. The term prognosis is used variably, with at least three meanings: any outcome study including those in initially healthy populations; synonymous with mortality; and as risk prediction (or prognostication). 

When high quality primary research studies exist, meta-analysis of individual participant data 20 is the most reliable method synthesis and is achievable.20 37 38 An emerging challenge is the synthesis of different types of evidence relating to one prognosis research question. For example, studies assessing whether a new prognostic biomarker is causally related to disease progression use different, and potentially complementary, methods for dealing with confounding (observational study designs use statistical adjustment and genetic study designs use mendelian randomisation).39

## Impact of research

However well prognosis research comes to protect against this range of biases, the “so what?” question demands answering. Prognosis research is not having the effect it should have both at early stages in the translational spectrum (for example, on informing the design and development of drug or other targets for patient management) or at later stages supporting clinical decision making (for example, in facilitating individualised or stratified medicine). Since 1991 there have been 102 risk prediction models reported for traumatic brain injury in 53 articles; in only five articles were models externally validated, and none has been widely implemented in clinical practice (fig 3⇓.9 From the perspective of a clinician and a patient, effectiveness means altered clinical decisions and consequent patient outcomes.17 The psychosocial effect of prognostic information on patients and their families also warrants consideration in such effectiveness criteria.

![Figure3](http://www.bmj.com/https://www.bmj.com/content/bmj/339/bmj.b4184/F3.medium.gif)

[Figure3](http://www.bmj.com/content/339/bmj.b4184/F3)

**Fig 3** Illustration of lack of clinical impact of some prognostic research. Systematic review identified 53 papers on 102 risk prediction models for death and disability in patients with traumatic brain injury published during 1991-2005, but only five were externally validated and none of the models is used in clinical practice9

From the perspective of a policy maker the impact of prognosis research should be made explicit in a cost effectiveness framework. For example, a recent study showed the value of cost effectiveness decision models for evaluating different prognostic risk scores to prioritise the waiting list for coronary surgery.40 From the perspective of a research funder, the cost of investing in new prognosis research and the impact of the resulting reduction in scientific uncertainty can be formally modelled.41 42

## Conclusion

Prognosis research across multiple disease areas faces challenges at each stage of the research process. We acknowledge that our backgrounds (in cardiovascular epidemiology and cancer biostatistics) both inform, and limit, our views. A systematic comparison of the state of the art of prognosis research across several clinical conditions would clarify the need for action and help prioritise our proposed 10 steps. Progress in prognosis research should be empirically demonstrated. One marker of progress is the emphasis that prognosis research commands in evidence based medicine; an influential book currently includes only 14 pages out of a total of 809.43 This needs to change.

#### Summary points

*   The quality of much prognosis research is poor 

*   Systematic reviews can often reach only limited conclusions because of variation in methods, poor reporting, and publication bias

*   Ten steps towards improving prognosis research are outlined

*   Study and protocol registration and guidelines for reporting are urgently required

## Notes

**Cite this as:** *BMJ* 2009;339:b4184

## Footnotes

*   Contributors: Discussions among the authors were facilitated by John Scadding and David Misselbrook (Royal Society of Medicine) and Trish Groves (*BMJ*). Each author contributed examples and critically commented on the text. HH wrote the first draft and is the guarantor. 

*   Competing interests: None declared.

*   Provenance and peer review: Not commissioned; externally peer reviewed.

## References

1.  Hemingway H. Prognosis research: why is Dr Lydgate still waiting? J Clin Epidemiol2006;59:1229-38.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2006.02.005&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=17098565&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000242136500001&link_type=ISI) 

2.  Moons KG, Royston P, Vergouwe Y, Grobbee DE, Altman DG. Prognosis and prognostic research: what, why, and how? BMJ2009;338:b375.
    
    [FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE2OiIzMzgvZmViMjNfMS9iMzc1IjtzOjQ6ImF0b20iO3M6MjM6Ii9ibWovMzM5L2Jtai5iNDE4NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

3.  Malats N, Bustos A, Nascimento CM, Fernandez F, Rivas M, Puente D, et al. P53 as a prognostic marker for bladder cancer: a meta-analysis and review. Lancet Oncol2005;6:678-86.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/S1470-2045(05)70315-6&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=16129368&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000231789400018&link_type=ISI) 

4.  Ntzani EE, Ioannidis JP. Predictive ability of DNA microarrays for cancer outcomes and correlates: an empirical assessment. Lancet2003;362:1439-44.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(03)14686-7&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=14602436&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000186356600008&link_type=ISI) 

5.  Nicholson A, Kuper H, Hemingway H. Depression as an aetiologic and prognostic factor in coronary heart disease: a meta-analysis of 6362 events among 146 538 participants in 54 observational studies. Eur Heart J2006;27:2763-74.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiZWhqIjtzOjU6InJlc2lkIjtzOjEwOiIyNy8yMy8yNzYzIjtzOjQ6ImF0b20iO3M6MjM6Ii9ibWovMzM5L2Jtai5iNDE4NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

6.  Kyzas PA, Denaxa-Kyza D, Ioannidis JP. Almost all articles on cancer prognostic markers report statistically significant results. Eur J Cancer2007;43:2559-79.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/j.ejca.2007.08.030&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=17981458&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000251706000027&link_type=ISI) 

7.  Hemingway H, Henriksson M, Chen R, Damant J Fitzpatrick N, Abrams K, et al. The effectiveness and cost effectiveness of biomarkers for the prioritisation of patients awaiting coronary revascularisation: a systematic review and decision model. Health Technol Assess (in press). 
    
    

8.  Whiteley W, Chong WL, Sengupta A, Sandercock P. Blood markers for the prognosis of ischemic stroke: a systematic review. Stroke2009;40:e380-9.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToic3Ryb2tlYWhhIjtzOjU6InJlc2lkIjtzOjk6IjQwLzUvZTM4MCI7czo0OiJhdG9tIjtzOjIzOiIvYm1qLzMzOS9ibWouYjQxODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

9.  Perel P, Edwards P, Wentz R, Roberts I. Systematic review of prognostic models in traumatic brain injury. BMC Med Inform Decis Mak2006;6:38.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1186/1472-6947-6-38&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=17105661&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 

10. Hayden JA, Chou R, Hogg-Johnson S, Bombardier C. Systematic reviews of low back pain prognosis had variable methods and results-guidance for future prognosis reviews. J Clin Epidemiol2009;62:781-96.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2008.09.004&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=19136234&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000268200300002&link_type=ISI) 

11. Pengel LH, Herbert RD, Maher CG, Refshauge KM. Acute low back pain: systematic review of its prognosis. BMJ2003;327:323.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjEyOiIzMjcvNzQxMC8zMjMiO3M6NDoiYXRvbSI7czoyMzoiL2Jtai8zMzkvYm1qLmI0MTg0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

12. Williams MD, Harris R, Dayan CM, Evans J, Gallacher J, Ben-Shlomo Y. Thyroid function and the natural history of depression: findings from the Caerphilly Prospective Study (CaPS) and a meta-analysis. Clin Endocrinol (Oxf)2009;70:484-92.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1111/j.1365-2265.2008.03352.x&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=18681859&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 

13. Zandbergen EG, de Haan RJ, Hijdra A. Systematic review of prediction of poor outcome in anoxic-ischaemic coma with biochemical markers of brain damage. Intensive Care Med2001;27:1661-7.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1007/s001340101076&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=11685309&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000172000600017&link_type=ISI) 

14. Hayden JA, Cote P, Steenstra IA, Bombardier C. Identifying phases of investigation helps planning, appraising, and applying the results of explanatory prognosis studies. J Clin Epidemiol2008;61:552-60.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2007.08.005&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=18471659&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000256164700007&link_type=ISI) 

15. Romero-Corral A, Montori VM, Somers VK, Korinek J, Thomas RJ, Allison TG, et al. Association of bodyweight with total mortality and with cardiovascular events in coronary artery disease: a systematic review of cohort studies. Lancet2006;368:666-78.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(06)69251-9&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=16920472&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000239849200034&link_type=ISI) 

16. Altman DG, Royston P. What do we mean by validating a prognostic model? Stat Med2000;19:453-73.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1002/(SICI)1097-0258(20000229)19:4<453::AID-SIM350>3.0.CO;2-5&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=10694730&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000085613900005&link_type=ISI) 

17. Reilly BM, Evans AT. Translating clinical research into clinical practice: impact of using prediction rules to make decisions. Ann Intern Med2006;144:201-9.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.7326/0003-4819-144-3-200602070-00009&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=16461965&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000235074800007&link_type=ISI) 

18. Schmitz-Drager BJ, Goebell PJ, Ebert T, Fradet Y. p53 immunohistochemistry as a prognostic marker in bladder cancer. Playground for urology scientists? Eur Urol2000;38:691-9.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1159/000020364&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=11111186&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000165692800010&link_type=ISI) 

19. Kyzas PA, Denaxa-Kyza D, Ioannidis JP. Quality of reporting of cancer prognostic marker studies: association with reported prognostic effect. J Natl Cancer Inst2007;99:236-43.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiam5jaSI7czo1OiJyZXNpZCI7czo4OiI5OS8zLzIzNiI7czo0OiJhdG9tIjtzOjIzOiIvYm1qLzMzOS9ibWouYjQxODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

20. Riley RD, Sauerbrei W, Altman DG. Prognostic markers in cancer: the evolution of evidence from single studies to meta-analysis, and beyond. Br J Cancer2009;100:1219-29.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1038/sj.bjc.6604999&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=19367280&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000265575000002&link_type=ISI) 

21. World Medical Association. Declaration of Helsinki. WMA, 2008.
    
    

22. Council for International Organizations of Medical Sciences. International ethical guidelines for biomedical research involving human subjects. 2nd ed. Geneva: WHO, 2002.
    
    

23. Rifai N, Altman DG, Bossuyt PM. Reporting bias in diagnostic and prognostic studies: time for action. Clin Chem2008;54:1101-3.
    
    [FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiRlVMTCI7czoxMToiam91cm5hbENvZGUiO3M6ODoiY2xpbmNoZW0iO3M6NToicmVzaWQiO3M6OToiNTQvNy8xMTAxIjtzOjQ6ImF0b20iO3M6MjM6Ii9ibWovMzM5L2Jtai5iNDE4NC5hdG9tIjt9czo4OiJmcmFnbWVudCI7czowOiIiO30=) 

24. Riley RD, Burchill SA, Abrams KR, Heney D, Lambert PC, Jones DR, et al. A systematic review and evaluation of the use of tumour markers in paediatric oncology: Ewing’s sarcoma and neuroblastoma. Health Technol Assess2003;7:1-162.
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=12583822&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 

25. Sutcliffe P, Hummel S, Simpson E, Young T, Rees A, Wilkinson A, et al. Use of classical and novel biomarkers as prognostic risk factors for localised prostate cancer: a systematic review. Health Technol Assess2009;13:1-242.
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=19624978&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000265139300001&link_type=ISI) 

26. Corrao G, Rubbiati L, Bagnardi V, Zambon A, Poikolainen K. Alcohol and coronary heart disease: a meta-analysis. Addiction2000;95:1505-23.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1046/j.1360-0443.2000.951015056.x&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=11070527&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000089790500006&link_type=ISI) 

27. Muntwyler J, Hennekens CH, Buring JE, Gaziano JM. Mortality and light to moderate alcohol consumption after myocardial infarction. Lancet1998;352:1882-5.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/S0140-6736(98)06351-X&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=9863785&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000077544600008&link_type=ISI) 

28. Nease RF Jr, Kneeland T, O’Connor GT, Sumner W, Lumpkins C, Shaw L, et al. Variation in patient utilities for outcomes of the management of chronic stable angina. Implications for clinical practice guidelines. JAMA1995;273):1185-90.
    
    

29. Kyzas PA, Loizou KT, Ioannidis JP. Selective reporting biases in cancer prognostic factor studies. J Natl Cancer Inst2005;97:1043-55.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiam5jaSI7czo1OiJyZXNpZCI7czoxMDoiOTcvMTQvMTA0MyI7czo0OiJhdG9tIjtzOjIzOiIvYm1qLzMzOS9ibWouYjQxODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

30. Yousef F, Cardwell C, Cantwell MM, Galway K, Johnston BT, Murray L. The incidence of esophageal cancer and high-grade dysplasia in Barrett’s esophagus: a systematic review and meta-analysis. Am J Epidemiol2008;168:237-49.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NzoiYW1qZXBpZCI7czo1OiJyZXNpZCI7czo5OiIxNjgvMy8yMzciO3M6NDoiYXRvbSI7czoyMzoiL2Jtai8zMzkvYm1qLmI0MTg0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

31. Hayden JA, Cote P, Bombardier C. Evaluation of the quality of prognosis studies in systematic reviews. Ann Intern Med2006;144:427-37.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.7326/0003-4819-144-6-200603210-00010&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=16549855&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000236251800007&link_type=ISI) 

32. Riley RD, Abrams KR, Sutton AJ, Lambert PC, Jones DR, Heney D, et al. Reporting of prognostic markers: current problems and development of guidelines for evidence-based practice in the future. Br J Cancer2003;88:1191-8.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1038/sj.bjc.6600886&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=12698183&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000182891200007&link_type=ISI) 

33. McShane LM, Altman DG, Sauerbrei W, Taube SE, Gion M, Clark GM. Reporting recommendations for tumor marker prognostic studies (REMARK). J Natl Cancer Inst2005;97:1180-4.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiam5jaSI7czo1OiJyZXNpZCI7czoxMDoiOTcvMTYvMTE4MCI7czo0OiJhdG9tIjtzOjIzOiIvYm1qLzMzOS9ibWouYjQxODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

34. Gould Rothberg BE, Bracken MB, Rimm DL. Tissue biomarkers for prognosis in cutaneous melanoma: a systematic review and meta-analysis. J Natl Cancer Inst2009;101:452-74.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiam5jaSI7czo1OiJyZXNpZCI7czo5OiIxMDEvNy80NTIiO3M6NDoiYXRvbSI7czoyMzoiL2Jtai8zMzkvYm1qLmI0MTg0LmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ==) 

35. Hlatky MA, Greenland P, Arnett DK, Ballantyne CM, Criqui MH, Elkind MS, et al. Criteria for evaluation of novel markers of cardiovascular risk: a scientific statement from the American Heart Association. Circulation2009;119:2408-16.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MTQ6ImNpcmN1bGF0aW9uYWhhIjtzOjU6InJlc2lkIjtzOjExOiIxMTkvMTcvMjQwOCI7czo0OiJhdG9tIjtzOjIzOiIvYm1qLzMzOS9ibWouYjQxODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

36. Riley RD, Ridley G, Williams K, Altman DG, Hayden J, de Vet HC. Prognosis research: toward evidence-based results and a Cochrane methods group. J Clin Epidemiol2007;60:863-5.
    
    [CrossRef](http://www.bmj.com/lookup/external-ref?access_num=10.1016/j.jclinepi.2007.02.004&link_type=DOI) 
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=17606185&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 
    
    [Web of Science](http://www.bmj.com/lookup/external-ref?access_num=000248138700016&link_type=ISI) 

37. Thakkinstian A, Dmitrienko S, Gerbase-Delima M, McDaniel DO, Inigo P, Chow KM, et al. Association between cytokine gene polymorphisms and outcomes in renal transplantation: a meta-analysis of individual patient data. Nephrol Dial Transplant2008;23:3017-23.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoibmR0IjtzOjU6InJlc2lkIjtzOjk6IjIzLzkvMzAxNyI7czo0OiJhdG9tIjtzOjIzOiIvYm1qLzMzOS9ibWouYjQxODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

38. Look MP, van Putten WL, Duffy MJ, Harbeck N, Christensen IJ, Thomssen C, et al. Pooled analysis of prognostic impact of urokinase-type plasminogen activator and its inhibitor PAI-1 in 8377 breast cancer patients. J Natl Cancer Inst2002;94:116-28.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NDoiam5jaSI7czo1OiJyZXNpZCI7czo4OiI5NC8yLzExNiI7czo0OiJhdG9tIjtzOjIzOiIvYm1qLzMzOS9ibWouYjQxODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

39. Kuper H, Nicholson A, Kivimaki M, Hemingway H, et al. Evaluating the causal relevance of diverse risk markers: horizontal systematic review. BMJ 2009;339:b4265. 
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6MzoiYm1qIjtzOjU6InJlc2lkIjtzOjE3OiIzMzkvbm92MDVfMS9iNDI2NSI7czo0OiJhdG9tIjtzOjIzOiIvYm1qLzMzOS9ibWouYjQxODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

40. Henriksson M, Palmer S, Chen R, Damant J, Fitzpatrick N, Abrams KR, et al. Assessing the cost-effectiveness of prognostic biomarkers: a case study in prioritising patients waiting for coronary artery surgery. BMJ (in press). 
    
    

41. Fenwick E, Claxton K, Sculpher M. The value of implementation and the value of information: combined and uneven development. Med Decis Making2008;28:21-32.
    
    [Abstract/FREE Full Text](http://www.bmj.com/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6NToic3BtZG0iO3M6NToicmVzaWQiO3M6NzoiMjgvMS8yMSI7czo0OiJhdG9tIjtzOjIzOiIvYm1qLzMzOS9ibWouYjQxODQuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 

42. Claxton K, Ginnelly L, Sculpher M, Philips Z, Palmer S. A pilot study on the use of decision theory and value of information analysis as part of the NHS health technology assessment programme. Health Technol Assess2004;8(31):1-103, iii.
    
    [PubMed](http://www.bmj.com/lookup/external-ref?access_num=14960258&link_type=MED&atom=%2Fbmj%2F339%2Fbmj.b4184.atom) 

43. Guyatt G, Rennie D, Meade M, Cook D. Users’ guides to the medical literature: a manual for evidence-based clinical practice. 2nd ed. AMA Press, 2008.