Rapid Responses to:

RESEARCH: Cynthia Lokker, K Ann McKibbon, R James McKinlay, Nancy L Wilczynski, and R Brian Haynes Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: retrospective cohort study BMJ 2008; 336: 655-657 [Abstract] [Full text]

Rapid Responses: Submit a response to this article

Rapid Responses published:

Validation of a prediction model: what’s in a name?

Ewout W Steyerberg, Hester F. Lingsma   (29 March 2008)

Validating Metrics, Early and Late

Stevan Harnad   (21 April 2008)

Validation of a prediction model: what’s in a name? 29 March 2008
Ewout W Steyerberg, Prof of Medical Decision Making Dept of Public Health, Erasmus MC, PO Box 2040, 3000 CA Rotterdam, the Netherlands, Hester F. Lingsma Send response to journal: Re: Validation of a prediction model: what’s in a name?	A recent paper presented an interesting model to predict citation counts for clinical articles 1. This topic is that important that we can predict that the paper will likely attract many citations. We first want to clarify some of the nomenclature of validation of prediction models, to avoid confusion in future reporting. The authors randomly divided 1274 articles into a derivation data set of 757 articles for development of a prediction model and a validation dataset for testing of 504 articles, after exclusion of outliers with >150 citations. This procedure is an example of a ‘split-sample’ approach. The authors however refer to it as ‘cross-validation’. Cross-validation would mean that we develop a model in the first part of the data and test it in the second part, and then repeat the procedure with development in the second part and testing in the first. The authors report that explained variation (R2) decreased from 0.60 at development to 0.56 at validation, and refer to this decrease as ‘shrinkage’. Shrinkage is not an appropriate term for this decrease; a better label is ‘optimism’ 2 3. Optimism is the phenomenon that prediction models tend to perform poorer in new data than in the data where the model was developed; it occurs especially when many predictors are considered in relatively small data sets 4. Ironically, a need for ‘shrinkage’ is well illustrated in Fig 2, where we note that the residuals are generally positive for low predictions (which were often too low), and generally negative for high predictions (which were often too high) 1. Shrinkage should be applied to the regression coefficients for more reliable predictions 2 4 5 6. How valid is this model to predict citations? First, the authors did not shrink regression coefficients, which implies that high predictions will be too high and low predictions too low for articles fulfilling the inclusion criteria. Second, for a future article we cannot know beforehand whether the article is an outlier, i.e. having more than >150 citations. Exclusion of outliers at validation is artificial and should not have been done; it has inflated the R2 of the model. As always with prediction models, future validation is required and may reveal disappointing performance. Ewout W Steyerberg Hester Lingsma References 1. Lokker C, McKibbon KA, McKinlay RJ, Wilczynski NL, Haynes RB. Prediction of citation counts for clinical articles at two years using data available within three weeks of publication: retrospective cohort study. Bmj 2008;336(7645):655-7. 2. Harrell FE, Jr., Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996;15(4):361-87. 3. Steyerberg EW, Harrell FE, Jr., Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. J Clin Epidemiol 2001;54(8):774-81. 4. Steyerberg EW, Eijkemans MJ, Harrell FE, Jr., Habbema JD. Prognostic modeling with logistic regression analysis: in search of a sensible strategy in small data sets. Med Decis Making 2001;21(1):45-56. 5. Copas JB. Regression, prediction and shrinkage. J R Stat Soc, Ser B 1983;45(3):311-354. 6. van Houwelingen JC, Le Cessie S. Predictive value of statistical models. Stat Med 1990;9(11):1303-25. Competing interests: None declared
Validating Metrics, Early and Late 21 April 2008
Stevan Harnad, Canada Research Chair in Cognitive Sciences Universite du Quebec a Montreal and University of Southampton, UK Send response to journal: Re: Validating Metrics, Early and Late	Here is another study that found a high correlation between early downloads and later citations (Brody et al 2006). It will be important to validate all these metrics (including citations) against peer evaluation too (Harnad 2007). Brody, T., Harnad, S. and Carr, L. (2006) Earlier Web Usage Statistics as Predictors of Later Citation Impact. Journal of the American Association for Information Science and Technology (JASIST) 57(8) pp. 1060 -1072. http://eprints.ecs.soton.ac.uk/10713/ Harnad, S. (2007) Open Access Scientometrics and the UK Research Assessment Exercise. In Proceedings of 11th Annual Meeting of the International Society for Scientometrics and Informetrics 11(1), pp. 27- 33, Madrid, Spain. Torres-Salinas, D. and Moed, H. F., Eds. http://eprints.ecs.soton.ac.uk/13804/ Competing interests: None declared