Speed of updating online evidence based point of care summaries: prospective cohort analysis
BMJ 2011; 343 doi: https://doi.org/10.1136/bmj.d5856 (Published 23 September 2011) Cite this as: BMJ 2011;343:d5856- Rita Banzi, researcher12,
- Michela Cinquini, statistician2,
- Alessandro Liberati, associate professor13,
- Ivan Moschetti, general practitioner1,
- Valentina Pecoraro, researcher1,
- Ludovica Tagliabue, medical resident14,
- Lorenzo Moja, assistant professor14
- 1Italian Cochrane Centre, Mario Negri Institute for Pharmacological Research, Milan, Italy
- 2Mario Negri Institute for Pharmacological Research, Milan
- 3Department of Oncology, Hematology and Respiratory Diseases, University of Modena and Reggio Emilia, Modena
- 4Department of Public Health, Microbiology and Virology, University of Milan, Milan
- Correspondence to: Rita Banzi Mario Negri Institute for Pharmacological Research, Via G. La Masa 19, 20156 Milan, Italy rita.banzi{at}marionegri.it
- Accepted 5 August 2011
Abstract
Objective To evaluate the ability of international point of care information summaries to update evidence relevant to medical practice.
Design Prospective cohort bibliometric analysis.
Setting Top five point of care information summaries (Clinical Evidence, EBMGuidelines, eMedicine, Dynamed, UpToDate) ranked for coverage of medical conditions, editorial quality, and evidence based methodology.
Main outcome measures From June 2009 to May 2010 we measured the incidence of research findings relating to potentially eligible newsworthy evidence. As samples, we chose systematic reviews rated as relevant by international research networks (such as, Evidence-Based Medicine, ACP Journal Club, and the Cochrane Collaboration). Every month we assessed whether each sampled review was cited in at least one chapter of the five summaries. The cumulative updating rate was analysed with Kaplan-Meier curves.
Results From April to December 2009, 128 reviews were retrieved; 53% (68) from the literature surveillance journals and 47% (60) from the Cochrane Library. At nine months, Dynamed had cited 87% of the sampled reviews, while the other summaries had cited less than 50%. The updating speed of Dynamed clearly led the others. For instance, the hazard ratios for citations in EBM Guidelines and Clinical Evidence versus the top performer were 0.22 (95% confidence interval 0.17 to 0.29) and 0.03 (0.01 to 0.05).
Conclusions Point of care information summaries include evidence relevant to practice at different speeds. A qualitative analysis of updating mechanisms is needed to determine whether greater speed corresponds to more appropriate incorporation of new information.
Introduction
As biomedicine evolves with the accumulation of new research and publications, promising healthcare interventions can emerge while others become out of date or suboptimal.1 2 Sound evidence—together with contextual factors, values, resources, etc—forms the basic framework on which healthcare decisions should rest. Failure to incorporate results of new research into practice can affect individual and population outcomes. This is the main reason for updating any medical information sources such as systematic reviews, guidelines, and clinical summaries.
Comprehensive presentation of new findings from research against the background of what is already available is essential to meet doctors’ needs for evidence during clinical consultations: which interventions work, which don’t work, which are additional or alternative, which need more investigation, and which might be harmful. For internet based information in particular, doctors and health professionals expect to rapidly find the latest knowledge to answer their information needs.
Point of care information summaries are web based compendiums designed to provide health professionals with comprehensive evidence condensed into easily digestible formats. The innovative aspect of these tools relies on how the information is engineered to be used at the point of care, when patient and practitioner interact. Point of care content is logically grouped around common medical scenarios and translated into alternative options related to diagnosis, treatment, and management.3 Publishers encourage physicians to use them during consultations or as a second opinion in their clinical decision making.
To make them attractive to final users, all publishers claim these products are regularly updated. Some even make direct reference to the dynamic incorporation of the latest evidence in their commercial names. To determine how long it takes for the latest research findings to make their way into a point of care summary we conducted a bibliometric analysis to examine the speed of updating—that is, the time between a paper’s publication and its citation in a point of care summary. For this analysis, we considered only papers with implications relevant to practice.
Methods
Out of 18 point of care information services available in 2008, we selected Clinical Evidence, Dynamed, EBM Guidelines, eMedicine, and UpToDate. These were ranked in the top quarter for at least two desirable dimensions: coverage of medical conditions (volume) and editorial quality and evidence based methodology.3 Our reasoning was that updating is a further desirable dimension of point of care summaries on top of others, and it would have been useless to look at the updating speed of products that were suboptimal in other dimensions on the basis of our evaluation. The decision to limit our analysis to the top ranking summaries reflected the aim of our research, which was to help users select one product over others.
For each of the five point of care information summaries we collected data on the updating mechanism by examining the free access web pages and sending emails to the information request service and editorial teams, as needed. This cross sectional qualitative analysis was done only once, in December 2009.
To evaluate updating speed we measured the incidence of research findings cited in point of care information summaries on potentially eligible newsworthy pieces of information. As samples of information relevant to practice we chose systematic reviews, which aim to provide a comprehensive appraisal of evidence. Findings from a single clinical trial are often rapidly contradicted by subsequent studies and low bias systematic reviews could help to get closer to the unknown “true evidence.”4 5 6 Systematic reviews have also gained acceptance as a starting point in the development of evidence based clinical practice guidelines.7 High quality systematic reviews are used more and are rated more highly by physicians in terms of relevance to clinical practice than other designs of articles.8 Four of the five point of care summaries we included clearly give priority to systematic reviews (and in general a cumulative approach to evidence) than other types of publications.3 We were not able to retrieve this information for eMedicine.
We selected all the systematic reviews signalled by the American College of Physicians (ACP) Journal Club and Evidence-Based Medicine Primary Care and Internal Medicine from April to December 2009. These two literature surveillance journals survey a wide range of international medical journals, applying strict criteria for the quality and validity of research articles. Practising clinicians assess studies that meet the basic validity criteria for relevance of clinical implications for practice and newsworthiness and a summary is then produced for the top rated articles. In the same period (April to December 2009) we selected all the Cochrane systematic reviews labelled as “conclusion changed” in the Cochrane Library. These are new citation versions of updated reviews that warrant additional highlighting in the Cochrane Library (for example, with a flag), indicating that they should be read again.9 We assumed that this sampling frame was representative of systematic reviews that meet explicit quality standards and are deemed directly relevant to clinical practice.
To evaluate how fast point of care summaries are updated we used a prospective cohort design over a one year period from June 2009 to May 2010. The follow-up started two months after the collection period to allow the potential citation of the most recent systematic reviews. Two reviewers independently checked whether each sampled systematic review was cited in at least one chapter of the five point of care information summaries. This was done monthly at the same time for each product. Disagreements were resolved by discussion between the two reviewers.
For each systematic review we defined “birth” as the publication date in one of the two literature surveillance journals or in the Cochrane Library and “death” (that is, event) as its citation in the monitored summaries. When the two reviewers agreed on the inclusion of that evidence in a summary the follow-up for that systematic review was terminated by the event. We censored systematic reviews when they had not been cited by the end of follow-up or if there was clear evidence that the topic was not covered by a given summary, similar to losses at follow-up in survival analyses. Two independent reviewers defined loss to follow-up. We excluded citations in additional reference lists, such as further or external readings and alert systems. We kept an archive of all the reference web pages citing the sampled systematic reviews.
We did not attempt any formal sample size calculation because information about the baseline incidence rates of citation was not available. Instead, we conducted an interim analysis after six months to determine the length of the collection period (that is, a small difference would have required an extended collection period and hence more systematic reviews). At the interim analysis we found substantial differences between the top performer and the other summaries, dramatically boosting the power of the study. The collection period was then stopped at nine months (December 2009).
We assessed the cumulative rate of updating using Kaplan-Meier survival analyses. As there were substantial differences between the top performer and the other summaries, we calculated the hazard ratios and 95% confidence intervals for each comparison using a univariate random Cox model. As we conducted an interim analysis to drive the length of the collection period, P≤0.025 was considered significant.
We further explored whether systematic reviews were more likely to be cited by the point of care information summaries on the basis of their source (literature surveillance journals or the Cochrane Library). As we observed different patterns of citation between the two second to top summaries, we compared the proportions of systematic reviews retrieved from literature surveillance journals or the Cochrane Library in these two summaries. Because this exploratory analysis did not aim to compare the citation rates but only the proportions, we used logistic regression and have reported the results as odds ratios.
Results
Table 1⇓ describes the updating mechanism for product. For EBM Guidelines information was obtained after contacting the editors by email, while for eMedicine we were unable to retrieve any details on updating. Clinical Evidence declares a target updating cycle of one year and alerts readers of each specific chapter about potentially relevant new publications, providing links to the full reference (BMJ Updates). These alerts, however, are not inserted in the chapters or evaluated together with the existing body of evidence. EBM Guidelines, UpToDate, and Dynamed refer to “a continuous update,” meaning that new research findings are incorporated into the summaries every time they are published. UpToDate is the only product that clearly reports quantitative data on the topic updated (35% of all contents during a four month cycle).
From April to December 2009, we retrieved 128 systematic reviews, 68 from the two literature surveillance journals (53%) and 60 (47%) from the Cochrane Library. The complete list is available in the appendix on bmj.com. Overall, 114 systematic reviews (89%) had been cited by at least one point of care summary. The median follow-up time was 33 weeks (range 1-60). Table 2⇓ reports the proportions of citations by summaries over time and the hazard ratio for each summary compared with the top performer. Dynamed has an updating process that markedly led the others (fig 1⇓). For instance, the hazard ratios for citation for EBM Guidelines and Clinical Evidence versus the top performer were 0.22 (95% confidence interval 0.17 to 0.29) and 0.03 (0.01 to 0.05), respectively. This means that the updating speed of Dynamed is 78% and 97% greater than those of EBM Guidelines and Clinical Evidence, respectively. The median time to citation was 7.7 weeks (range 7-8.2) for Dynamed and 42 weeks (range 34-maximum not reached) for EBM Guidelines. Dynamed has a median citation rate of around two months, EBM Guidelines is around 10 months but quite close to the limit of our follow-up. The citation rate of the other three point of care summaries (UpToDate, eMedicine, Clinical Evidence) were so slow that they exceeded the follow-up period and we could not compute the median.
Dynamed was also the first when we separately analysed the updating rate for systematic reviews retrieved through the Cochrane Library (fig 2⇓) and the literature surveillance journals (fig 3⇓). The two second point of care summaries (EBM Guidelines and UptoDate) had similar updating rates when we considered the whole sample of systematic reviews but differed when we took the origin of the systematic reviews into account. Cochrane systematic reviews were more likely to be cited by EBM Guidelines than by UpToDate (odds ratio 0.02, 0.01 to 0.10; P<0.001, logistic regression). EBM Guidelines has a formal agreement with the Cochrane Collaboration to use Cochrane contents and label its summaries as “Cochrane inside.”
Discussion
Evidence held to be relevant to clinical practice is inserted at different rates in point of care information summaries, and these products vary widely in their speed at updating content. Our citation analysis showed that Dynamed clearly dominates the other products (Clinical Evidence, EBMGuidelines, eMedicine, and UpToDate). Slowness in updating could mean that new relevant information is ignored and could thus affect the validity of point of care information services. Ultimately, whenever the transfer of relevant information is inappropriately slow, this can affect the care of patients, potentially denying treatments of proved benefit. This happens despite the fact that many of these products promote themselves to the clinical community as being regularly updated with the latest evidence.
When should point of care information content be updated?
A few studies have looked into strategies for updating clinical guidelines2 10 11 12 13 and systematic reviews,14 15 but no definitive conclusions have been reached on the best approach. A bottom line common to these studies was that updating is costly and time consuming. As far as we know, no data are available on how quickly point of care information content is updated and so publishers seem to adopt empirical approaches in managing their updating schedule. Even without an optimal approach, the updating of point of care information summaries should be evaluated bearing in mind that these online tools are largely intended to be used by an audience sensitive to brand new information.
Reasons for different updating speeds
Differences in updating ability are possibly justified by different approaches to content development. According to Shekelle et al, the updating process is based on two phases: identifying important new evidence and assessing whether it offers new information that might change recommendations for clinical practice.2 In addition, a third phase exists in which the new evidence should be included in the “old” body of knowledge. Citing a single trial or a systematic review without appraising and interpreting this new evidence in the light of existing knowledge is not enough.16 In other words, updating is not only a matter of literature surveillance but implies a critical evaluation of what a new item of knowledge adds to other works and what that means for clinical practice.6
Referring to these three phases, do these point of care information summaries differ in their approaches? Some of the products we analysed identify important new evidence by regular systematic searches or active surveillance of published journals and other information sources (such as reports from drug regulatory agencies, public health entities, World Health Organization, etc). In this phase we detected no major differences between products. How this new evidence is deemed relevant and then incorporated into the body of the summary probably largely dictates the different updating speeds. In Dynamed, the top ranked summary, updating is done centrally by the editorial team (supported by McMaster University’s Health Information Research Unit since the end of 2010), and this might make for more prompt inclusion of evidence. In Clinical Evidence, one of the lowest ranked, the authors of chapters are involved and often a new peer review process is required (R Minhas, editor of Clinical Evidence, personal communication). This is time consuming so content is likely to be updated more slowly or, in the worst case, to simply become out of date. In 2009, the BMJ Group launched the BMJ Best Practice product by engineering the contents of Clinical Evidence to fit the purpose of better use at the point of care, but we did not include it as it was not evaluated in our previous work.3 As little information on updating mechanisms was available for some summaries, our ability to further explore possible differences in updating approaches is limited. Publishers should fully elucidate information about their updating mechanisms.
Limitations
We chose a citational approach to measure updating speed, though there are shortcomings with this approach. Firstly, the total number of citations in the point of care information products should have been taken into account. Secondly, citational analysis counts only bibliographic references without going deeply into the content of the citation. This criticism, widely raised when citational analysis is used to evaluate scientific productivity and quality,17 18 also applies to our assessment. We did not attempt to go beyond the empirical number of citations found. In fact we did not judge the appropriateness of the update but simply used the updating speed as its proxy. Qualitative analysis of the updating process and how new evidence is incorporated and affects recommendations should also be taken into account in assessing whether one summary is better than others. Thus we cannot say that Dynamed is superior to the other products in terms of the appropriateness of the updating process or that Clinical Evidence compensates the limitations of its updating speed by offering deeper and more insightful updating.
We did not directly assess how many systematic reviews in our sample called for a change in clinical practice as we assumed that all our sources (ACP Journal Club, Evidence-Based Medicine, and the Cochrane Library) highlight newsworthy and relevant evidence through well established selection processes. Furthermore, these are considered authoritative international networks that close the gap between medical literature and clinical practice. We cannot exclude that relevant messages requiring urgent action might be given priority by the publishers, thus preserving the quality of the point of care summary. We chose Cochrane systematic reviews with “conclusion changed,”—that is, those that should be read again.9 If a point of care information summary still cited the old version of the Cochrane systematic review this was considered not updated, regardless of the nature and impact of the change in conclusions. We believe this conservative approach, which might have partially influenced the citing speed of Cochrane systematic reviews, was appropriate as knowing that a Cochrane systematic review has been updated could be important for readers. We are also aware that we relied on sources of pre-appraised evidence mainly devoted to general practice and internal medicine, though specialty topics are provided too (see appendix on bmj.com). Finally, we did not consider the updating of results from studies with other designs (such as randomised clinical trials) as we think that systematic reviews are preferable to support decision making at the point of care.
Conclusions
Updating is only one aspect of the overall quality of a point of care product. Other studies have assessed other dimensions such as user’s satisfaction, how well different online point of care services answered questions arising in daily clinical work, content development, and evidence based soundness.19 Findings from both user and content centred analyses need to be combined if one has to choose one product rather than another. Readers should be aware that point of care information summaries vary widely in their updating ability, and, in some cases, it might be unsatisfactory in relation to what users expect and what publishers advertise. In the context of an editorial market with rapidly evolving point of care summaries, our specific intent was to provide a snapshot assessment of the updating speed of point of care services with recently published relevant systematic reviews. The quantitative findings should be considered together with a qualitative analysis of updating methods that can be done only if updating mechanisms are described in greater detail by publishers.
The process leading from evidence to clinical recommendation and then to changes in behaviour is affected by many factors besides access to the latest studies.20 21 22 Nevertheless, appropriate promotion of progressing evidence is essential to provide patients with better healthcare.
What is already known on this topic
Few studies have compared the quality of point of care summaries
Most looked at user satisfaction as the outcome measure and none evaluated their speed of updating and appropriateness
What this study adds
Point of care information summaries insert latest evidence relevant to practice at different speeds
Notes
Cite this as: BMJ 2011;343:d5856
Footnotes
We thank Richard Smith for helpful discussion in devising the study and Judith Baggott for editing.
Contributors: RB, AL, LM, and IM devised and designed the study; RB, VP, and LT extracted data; MC, RB, and LM analysed and interpreted data; RB and LM wrote the first draft; all authors contributed to subsequent versions and approved the final article. LM is guarantor.
Funding: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors. It was part of the AIFA’s contract to run a series of investigations about point of care products, and this is one of the studies that have emerged from this activity.
Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work. In 2003-8 the Italian Cochrane Centre received grants from the Italian Medicines Agency (AIFA) for the Italian translations of one of the products assessed in the study (Clinical Evidence). The Italian Cochrane Centre is part of the Cochrane Collaboration, which forms a publishing partnership with Wiley-Blackwell to deliver the Cochrane Library through Wiley InterScience. The content of the paper does not represent any official view of the Cochrane Collaboration but solely that of the authors.
Ethical approval: Not required.
Data sharing: The dataset is available from the corresponding author at rita.banzi{at}marionegri.it.
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.