Research

Relation of study quality, concordance, take home message, funding, and impact in studies of influenza vaccines: systematic review

BMJ 2009; 338 doi: http://dx.doi.org/10.1136/bmj.b354 (Published 12 February 2009) Cite this as: BMJ 2009;338:b354
  1. T Jefferson, coordinator,
  2. C Di Pietrantonj, statistician,
  3. M G Debalini, researcher,
  4. A Rivetti, researcher ,
  5. V Demicheli, director of health, Piemonte region
  1. 1Cochrane Vaccines Field, ASL (Azienda Sanitaria Locale) AL 20, 15100 Alessandria, Italy
  1. Correspondence to: T Jefferson jefferson.tom{at}gmail.com
  • Accepted 3 November 2008

Abstract

Objective To explore the relation between study concordance, take home message, funding, and dissemination of comparative studies assessing the effects of influenza vaccines.

Design Systematic review without meta-analysis.

Data extraction Search of the Cochrane Library, PubMed, Embase, and the web, without language restriction, for any studies comparing the effects of influenza vaccines against placebo or no intervention. Abstraction and assessment of quality of methods were carried out.

Data synthesis We identified 259 primary studies (274 datasets). Higher quality studies were significantly more likely to show concordance between data presented and conclusions (odds ratio 16.35, 95% confidence interval 4.24 to 63.04) and less likely to favour effectiveness of vaccines (0.04, 0.02 to 0.09). Government funded studies were less likely to have conclusions favouring the vaccines (0.45, 0.26 to 0.90). A higher mean journal impact factor was associated with complete or partial industry funding compared with government or private funding and no funding (differences between means 5.04). Study size was not associated with concordance, content of take home message, funding, and study quality. Higher citation index factor was associated with partial or complete industry funding. This was sensitive to the exclusion from the analysis of studies with undeclared funding.

Conclusion Publication in prestigious journals is associated with partial or total industry funding, and this association is not explained by study quality or size.

Introduction

Healthcare workers wanting to keep up to date with recent advances in their specialty must deal with the quantity and quality of information sources.1 2 One study estimated that every month journals publish 7287 items (studies, letters, and editorials) relevant to primary care. These would take physicians trained in epidemiology 627 hours to read and appraise. Few healthcare workers have the time and skills to carry out in depth critical appraisal of published articles.3 Given the pressures of everyday work, the time available for reading a scientific article might be as little as 22 minutes.4

To inform their conduct and update their knowledge physicians might rely on a brief (two minute) scan of material.5 6 This probably includes either browsing the abstract (the “shop window” of an article) or the conclusions paragraph (the “take home message”), especially those published in prestigious journals (that is, those with the highest journal impact factor) that are most readily available or of their digests. The impact factor was specifically developed to facilitate prioritisation of subscription resources, allowing targeting on the most cited journals, considered to be most read and the ones that publish articles of the best quality. These journals are thus more likely to be accessible and their content widely disseminated.7 8

Methodological quality of studies is the other important aspect. Several items, all affecting the reliability of what presents to healthcare workers, are related in a complex fashion. Such quality is associated with the size of the effects reported in a study. The lower the quality, the greater the effect of the intervention seems.9 10

Funding of meta-analyses of antihypertensive drugs by individual pharmaceutical companies has been shown to be inversely associated with whether data presented supported the authors’ conclusions (concordance)11 and directly associated with the probability that study conclusions did or did not support the drugs being evaluated (take home message).12 The relation between the items of quality, concordance, take home message, and funding of single studies, however, is thought to be specific to the context, drugs evaluated, and outcomes assessed. This implies that generalisation of quality across study designs, interventions, and outcomes cannot be made on current evidence. In addition we found no studies looking at the relation of all these variables with that of study dissemination and possible impact.13 14

We explored the relation between study concordance, take home message, funding, and citation (as a proxy for dissemination) in studies on influenza vaccines by reviewing all published comparative studies. Our choice was motivated by the global nature of the intervention, the almost exclusive analysis by analytical studies of quality items in randomised controlled trials or their meta-analyses (in influenza vaccines there is an abundance of non-randomised evidence), and by the recent publication of studies suggesting poor quality and contradictory evidence from field studies supporting vaccination.15 16 17 18 19 20

Methods

Evidence searches

We systematically reviewed comparative studies on influenza vaccines. We used the bibliography of available up to date systematic reviews of influenza vaccines to identify studies comparing the effects of influenza vaccines with placebo or no intervention (“primary studies”) and carried out supplementary searches for all other comparative primary studies published by the end of 2006 (see appendix on bmj.com for search strategy and procedure).

Data extraction and quality assessment

We extracted primary study data in duplicate according to a list of variables (box 1).

Box 1 Variables extracted from primary studies

  • Study identification

  • Year(s) of execution of the study (the end of the past influenza season cited in the study)

  • Year of publication

  • Study design and location

  • Number of arms

  • Comparison and type of outcome assessed within the same study (papers divided into separate studies or datasets if they contained more than one study or comparison)

  • Study population

  • Types of influenza vaccine used

  • Content of influenza vaccines used

  • Match between content and circulating viruses in the community

  • Types of outcome assessed

  • Intervention arm size

  • Number of events observed in the intervention arms

  • Types of control

  • Size of control arms

  • Number of events observed in the control arms

  • Type of funder

  • Impact factor assigned to the journal for the year after the study was published, number of times the articles have been cited by other sources (citation index factor), conclusions of article (classified according to the categories reported in box 2)

Two reviewers, independently unblinded to the authors and institutions of the article, extracted data from the studies in three phases. Firstly, they read each study and compiled an abstract describing study content. Data presented in the study were summarised on a computerised database based on the Cochrane vaccines field study register. This process involved understanding the design and intent of the authors as well as reading and reconciling each part of the study. We then compared our abstract with the authors’ presentation and summaries of results. To classify concordance and take home message definitions we asked whether data presented in the study supported the authors’ conclusions and then whether the conclusions did or did not support the vaccine(s) being evaluated. We classified take home message of all studies as supporting, neutral, or critical of the effects of influenza vaccines (box 2). The definitions were based on work carried out on study concordance and take home message.11 12 We grouped funding sources into government/private/unfunded or industry/mixed, concordance into yes or no/partially/unclear, and take home message into favourable or mixed/unfavourable/unclear.

Box 2 Classification of study take home message

Supportive
  • The authors state the dominance of the vaccines even without supporting statistical conclusions

  • The study reports statistically significant vaccine effects compared with control intervention(s)

  • The authors report the main study end points as statistically significantly dominant (P<0.05) compared with the control arm(s)

  • The authors state that vaccination reduces disease burden and/or treatment costs and/or resource consumption

  • The authors stress the efficacy, effectiveness, and safety of the vaccines compared with control intervention(s)

  • The authors conclude that vaccination should be undertaken

  • The authors criticise dissenters on the effects of vaccination

  • The authors support at least one of the producers’ or programme sponsors’ statements on the vaccines

Neutral
  • The authors conclude that there are insufficient data to assess one or all of the effects of the vaccines

  • The authors make no recommendations but seem fairly to weigh up pros and cons of each course of action

  • The authors conclude that vaccination is cost neutral

  • The authors conclude that the data presented are insufficient to answer the study question

  • The authors state that after vaccination the evidence for reduction of disease burden and/or treatment costs and/or resource consumption is slight or non-existent

  • The authors dissent from at least one of the producers’ or programme sponsors’ statements on the vaccines but do not reach a definite conclusion

  • The authors conclude that the evidence shows efficacy, effectiveness, and safety of the vaccines compared with control intervention(s) for some outcomes but not for others

  • The authors conclude that the evidence shows efficacy, effectiveness, and safety of one type of vaccine compared with another and with control intervention(s)

Critical/not supportive
  • The authors report no statistically significant efficacy or effectiveness results and/or significant harms

  • The authors clearly state the vaccines’ inferiority and present data showing significant vaccine inferiority compared with control(s)

  • The authors state that vaccination does not reduce disease burden and/or treatment costs and/or resource consumption

  • The authors emphasise concerns over efficacy and/or effectiveness and/or safety and/or recommend alternative interventions to control the burden of disease

  • The authors criticise those supporting influenza vaccination or state that vaccination should not be introduced or continued

  • The authors are critical of all the producers’ or programme sponsors’ statements on the vaccines

Finally, we assessed quality of randomised and non-randomised studies separately. Randomised studies were assessed according to randomisation method, generation of the allocation sequence, allocation concealment, blinding, and follow-up. Non-randomised studies were assessed for the presence of potential confounders with the appropriate Newcastle-Ottawa scales29 for case-control and cohort studies. We assigned risk of bias (low, moderate, and high risk of bias) as described by the Cochrane Handbook30 by design category (randomised and non-randomised).

Statistical methods

We used χ2 tests to assess the association between items such as quality of methods, funding sources, coherence, and take home message.

To assess the level of dissemination we identified the impact factor of the journals publishing each of the studies in the review. A journal’s impact factor is calculated from the number of citations in the current year to items published in the previous two years (numerator), divided by the number of substantive articles and reviews published in the same two years (the denominator). This is proposed as a proxy for journal quality and, by implication, for quality of the research articles in that journal.7 8 The impact factor was designed to identify journals worthy of subscription, and its rankings fit well with the perceived prestige of journals8 and their circulation.

We identified the citation index of each article (the cumulative total of the number of times the article has been cited in journals indexed in the ISI Web of Science database). The latest impact factor and citation indices were taken from the ISI website http://isiwebofknowledge.com/.

We also tested the association between delay in publication (difference between the publication date and the date the study was completed), the journal impact factor, or the citation index factor against the items described above by Wilcoxon and Kruskal-Wallis non-parametric tests (SAS/STAT 9.1, SAS, Cary, NC). Other studies have described an association between type of take home message and citation and type of funder and citation.12 21 22 23 24 25 26 27 28

Results

Characteristics of primary studies

The figure shows the flow of the included studies.

Figure1

Flow of primary studies in review (RCT=randomised controlled trial; CCT=clinical controlled trial, semi-randomised trial)

We included 274 comparative studies/datasets (from 259 papers) in our population. These were mostly randomised or semi-randomised (that is, the methods of allocation were alternation or birth date) studies (138/274, 50%) or prospective cohort studies (69/274, 25%). Most studies were carried out on healthy or general populations (203/274, 74%) and were government financed (48%). Seventy per cent of studies reported conclusions favourable to the vaccines but only 18% showed complete concordance between data reported and study conclusions. Over half (56%) of studies were at high risk of bias, with only 4% being at low risk. We did not include any vaccine registration trials.

Association between study quality and concordance, take home message, and funding

Table 1 summarises our main findings. All comparisons assessing the association with study quality were carried out after we excluded studies for which we had insufficient data (4/274). We calculated χ2 on the basis of the three methodological quality categories of high, moderate, and low risk of bias. We calculated the χ2 for all comparisons of funding sources for three categories (government, industry, missing). Because a high proportion of studies had missing funding information (64/274, 23%) we carried out a full sensitivity analysis to assess robustness of observed associations by systematically assigning studies with unclear funding source to the other known funding categories, thus covering all possible scenarios.

Table 1

 Summary of main findings of assessment of possible association between methodological quality, concordance between data presented and conclusions reported, take home message of article conclusions, and funding source in comparative studies of influenza vaccines

View this table:

Our analysis shows the presence of a strong positive association between methodological quality and concordance between results presented and study conclusions (when we aggregated studies with moderate and high risk of bias: odds ratio 16.35, 95% confidence interval 4.24 to 63.04). In other words, the higher the study quality (and the lower the risk of bias), the higher the probability of concordance. In addition, the higher the probability of concordance, the lower the probability that a study’s conclusions were in favour of vaccines’ effectiveness (0.04, 0.02 to 0.09); table 2 shows examples of discordance between results and conclusions and take home message. We found no association between funding source and methodological quality (excluding studies with unknown funding: 0.74, 0.19 to 2.84), but there was an inverse association between conclusions in favour of the vaccines’ effectiveness and government funding (0.45, 0.26 to 0.90). This finding confirms the established association between funding source and type of study conclusions.21 22 23 24 25 28

Table 2

 Examples of discrepancy between results and conclusions

View this table:

Association between study quality items, funding, and dissemination

Statistical tests show no evidence of association between the mean time lapsed from the end of the study to the date of publication (publication delay), methodological quality, type of study design, concordance, coherence, and funder. There was no evidence of association between either journal impact factor or citation index factor and study design, methodological quality, and concordance.

The mean journal impact factor of 92 government funded studies was 3.74, and their mean citation index factor was 33.75. We calculated the mean impact factor on 114 studies, as the oldest studies were published in journals for which no impact factor was available. For the 52 studies wholly or partly funded by industry, the mean impact factor was 8.78 and the mean citation index factor was 58.39. We calculated the mean citation index for 74 studies as the oldest studies were published in journals for which no citation index was available.

Non-parametric testing shows that the impact factor score of studies with complete or partial industry funding was significantly higher than that of studies funded by government and other funders (table 3).

Table 3

 Analysis of relation between journal impact factor, citation index factor, study sample size, publication delay, methodological quality, take home message content, concordance, and source of funding

View this table:

Non-parametric testing for article citation index showed a similar significant difference, which was, however, sensitive to the exclusion of studies with no declared funding source. In this case we found no difference in citation between studies with complete or partial funding from industry and studies funded by government and other funders (table 3).

There was no evidence of association between sample size of comparator and study design, methodological quality, concordance, and take home message.

We conclude that study size is an unlikely explanation for the association between industry sponsorship and publication in journals with higher impact factors and that industry funding is associated with a higher probability of publication in journals with a higher impact factor and possibly a higher citation index.

The differences in impact factor means between studies funded by government and those funded by industry or mixed sponsors were significant and sufficiently robust. Whole or part industrial funding was associated with publication in journals with higher impact factors.

Discussion

In the studies we included, poor methodological quality was associated with a discrepancy between results and conclusions, and this in turn was associated with optimistic conclusions in non-government sponsored studies. We found no direct association between these findings and industry funding, but this might have been affected by the sizeable number of studies with undeclared sponsors (23%).

Studies partly or completely sponsored by industry, however, were published in more prestigious journals and are probably cited more, although their methodological quality and size were similar. Some of these findings might help to explain the continuation of a near global policy, despite growing doubts as to its scientific basis.

We reasoned that the combined impact factor and citation index would give us a good idea of the circulation and dissemination of the study (that is, the interest its publication generated). Most of our studies (70%) were of poor quality with overoptimistic conclusions—that is, not supported by the data presented. Those sponsored by industry had greater visibility as they were more likely to be published by high impact factor journals and were likely to be given higher prominence by the international scientific and lay media, despite their apparent equivalent methodological quality and size compared with studies with other funders. Although differences in citation index by study funding are sensitive to the inclusion of the large number of studies with undeclared sources of funding, the higher impact factor and citation index are probably a reflection of a higher profile of industry sponsored studies and a more thorough dissemination of their content.

There are two possible mechanisms involved. Firstly, the same studies are cited more than the others, possibly because of the systematic nature of the dissemination of their results by industry. We have personally observed this on three recent occasions in which industry representatives presented abstracts or reprints of industry sponsored influenza vaccine studies to decision makers, their advisors, and local researchers in an effort to influence their decisions. Symposiums, conferences, and other types of publication further enhance the dissemination process. Often the abstracts were expensively bound and translated into the local language, a tangible sign of their importance to industry.

We cannot say for certain why industry sponsored studies are more attractive to more prestigious journals, but such journals are preferentially targeted by all studies because of their prominence and prestige, so industry sponsored studies might have a higher probability of acceptance. The two mechanisms might be linked, but further research, especially in other specialties, is required. As a measure of transparency for readers and authors, however, we recommend that once a year editors and publishers should post all sources of income related to the running of the journal.

Our finding of lack of concordance between results and conclusions is similar to those of Yank and colleagues in an industry sponsored meta-analysis of antihypertensive drugs.11 In our studies, however, we found a lack of concordance between results and conclusions associated with poor quality rather than any specific sponsorship. Given our findings of lack of concordance in primary studies, the content of current policy might reflect the gap between results and conclusions—that is, synthesis of evidence for policy making might be carried out at two independent levels: that of results and that of conclusions. In most cases, what you see is not necessarily what you get.

What is already known on this topic

  • Study sponsorship is associated with optimistic results

  • Influenza vaccination continues to be recommended globally, despite growing doubts about the validity of the scientific evidence underpinning policy recommendations

What this study adds

  • Evidence is of poor quality, and studies with conclusions in favour of vaccines are of significantly lower methodological quality

  • Influenza vaccines studies sponsored by industry are published in journals with higher impact factors and are cited more but are of similar size and quality to the others

Notes

Cite this as: BMJ 2009;338:b354

Footnotes

  • Karen Davies, Peter Doshi, Peter Gøtzsche, and Iain Chalmers commented on previous drafts.

  • Contributors: TJ, CDP, and VD designed the study. TJ and CDP wrote the protocol. AR and MGD carried out supplementary searches. TJ, AR, and MGD applied inclusion criteria and extracted data. CDP checked data extraction and carried out data analysis and statistical testing. TJ, CDP, and VD wrote the final report. All authors contributed to both protocol and final report. VD is guarantor.

  • Funding: This study was funded by ASL AL, Alessandria, Piemonte, Italy. The funding source had no role in the study design, collection, analysis, and interpretation of results, in the writing of the report, or in the decision to submit the paper for publication.

  • Competing interests: TJ owned shares in GlaxoSmithKline and has received consultancy fees from Sanofi-Synthelabo (2002) and Roche (1997-9).

  • Ethics approval: Not required.

This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

References

View Abstract