References that anyone can edit: review of Wikipedia citations in peer reviewed health science literature

Strengths and limitations

As the first study to describe the citation practices of Wikipedia (how, where, when, and so on) in indexed health science literature, our study adds insight into the role that Wikipedia may play in academic literature. We also provide information about the quality of the journals that cite Wikipedia, measured through impact factors from the Journal Citation Reports. The study searched citations from two well recognised databases (Scopus and Web of Science) and included all English language journals available in full text at Canadian university libraries, so some citations may have been excluded from the search because of the search criteria. However, the study used two comprehensive databases that probably captured most citations from journals with reaching influence as well as high impact factors. Although we report an increasing number of Wikipedia citations, the health science literature is expansive; therefore, papers that have cited Wikipedia remain a small minority of all published papers, albeit with increasing frequency.

Comparison with other studies

Active research on Wikipedia has examined the role of academic citations in Wikipedia, content validity, and interactions between Wikipedia and user groups.5 The only other study to examine Wikipedia in the scholarly references was completed by Park in 2011.30 In this bibliometric analysis, Park also used the ISI Web of Science and Scopus databases to identify the total number of studies (n=1746), as well as leading authors, their institutional affiliations, most frequent publication sources, main academic fields, and other statistics on the frequency with which scholarly articles cite Wikipedia. Our study took Park’s analysis further. Firstly, we used Ulrich’s database to refine search findings to be specific to health sciences literature. Secondly, we examined how Wikipedia is being cited. Both studies show a general increase in frequency of citations over time, suggesting that if Park’s data were to extend beyond 2010 they would continue to show increasing Wikipedia citations. While Park identified the publications that cited Wikipedia most often, our study also provides impact factors, adding information about journals’ impact.

Contextualisation and policy implications

Just because more researchers are citing Wikipedia does not necessarily justify it as a valid source of information for citation. Wikipedia itself has cautioned against using Wikipedia as a source,33 and some universities have gone so far as to ban students from citing Wikipedia.34 Recognising that learning modalities are changing and evolving increasingly towards online and e-resources,21 23 35 36 we believe that ensuring that peer reviewed academic literature aligns with the International Committee of Medical Journal Editors (ICMJE) guidelines is still important. Relevant to this study, the ICMJE guidelines state that “Readers should therefore be provided with direct references to original research sources whenever possible.”29 Although people have been argued that Wikipedia is comparable to an encyclopedia in terms of accuracy,8 37 this argument overlooks the literary problems associated with citing a tertiary source intended to direct researchers to an appropriate reference.34 We echo the findings of other researchers who believe that Wikipedia should not be cited when a more authoritative—that is, primary, permanent, peer reviewed, evidence based—source exists6; however, Wikipedia may be the most appropriate source to cite for a definition of Wikipedia and in situations in which Wikipedia is used as part of the scientific methods (for example, a search strategy). Outside of those rare instances, arguing that citation of Wikipedia in the academic literature is appropriate is difficult. Our “original research” category further illustrates how Wikipedia was often cited when the authors could have cited an original research study. For example, one article stated that “Some researchers propose that vitamin D supplementation may be beneficial in the treatment and prevention of some types of cancer,”38 and rather than citing the original sources provided on the Wikipedia page, the author instead cited Wikipedia.

Evidence based medicine requires that clinical decisions are made using professional expertise in conjunction with replicable evidence from systematic research.39 As Wikipedia entries are constantly changing, they may be difficult to access in the form that was originally cited by the time the work is published, reducing the reader’s ability to reproduce the author’s process.27 In a study by Peoples,6 most citations to Wikipedia did not reference the date and time at which the entry was visited. Although Wikipedia does have a history function that allows users to access previous versions of the page, this function may not be obvious to most users, and determining which version was accessed for citations that do not denote the date and time of the visit can be difficult. If Wikipedia is to be cited in the future, not only the date accessed but also the time should be included so that future researchers can access the entry as it was originally viewed. Thus, the process of peer review should include close attention to the references that appear in articles before recommending them for publication and potentially approving information that contradicts evidence based research.

Future directions

Although the proportion of retrieved Wikipedia citations in the indexed health science literature was relatively small, the increasing trend of citation is still important to note. The study of how Wikipedia is used in the academic literature is relatively young, and beginning an open discourse around the appropriateness of using non-permanent Wikipedia citations in the academic literature seems timely. The relationship between academic publication and Wikipedia remains largely understudied, and international guidelines such as those of the ICJME, World Association of Medical Editors, and Council of Science Editors lack editorial guidance on the subject. Although our study does not provide evidence of harm by these citations, even those that substitute for a permanent primary research source, we emphasise the need for a consistent voice on how Wikipedia should be used in the academic literature. This study begs the broader question of “what is an appropriate reference source?” Is it something that is generated by the scientific method, is replicable, and undergoes rigorous peer review or is it something that is collaboratively generated, readily open to editing, and broadly accepted? We call attention to the need to work with the Wikipedia community to establish guidelines not only for reviewers and editors but also for academics drafting publications.

Conclusion

An increasing number of peer reviewed academic papers in the health sciences are citing Wikipedia. The apparent increase in the frequency of citations of Wikipedia may suggest a lack of understanding by authors, reviewers, or editors of the mechanisms by which Wikipedia evolves. Although only a very small proportion of citations are of Wikipedia pages, the possibility for the spread of misinformation from an unverified source is at odds with the principles of robust scientific methodology and could potentially affect care of patients. We caution against this trend and suggest that editors and reviewers insist on citing primary sources of information where possible.