Televised medical talk shows—what they recommend and the evidence to support their recommendations: a prospective observational study
BMJ 2014; 349 doi: https://doi.org/10.1136/bmj.g7346 (Published 17 December 2014) Cite this as: BMJ 2014;349:g7346
All rapid responses
Rapid responses are electronic comments to the editor. They enable our users to debate issues raised in articles published on bmj.com. A rapid response is first posted online. If you need the URL (web address) of an individual response, simply click on the response headline and copy the URL from the browser window. A proportion of responses will, after editing, be published online and in the print journal as letters, which are indexed in PubMed. Rapid responses are not indexed in PubMed and they are not journal articles. The BMJ reserves the right to remove responses which are being wilfully misrepresented as published articles or when it is brought to our attention that a response spreads misinformation.
From March 2022, the word limit for rapid responses will be 600 words not including references and author details. We will no longer post responses that exceed this limit.
The word limit for letters selected from posted responses remains 300 words.
I read with great interest the paper by Korownyk et al. BMJ 2014;349:g7346. Although I agree with the general conclusion of the paper regarding the lack of evidence to support several particular recommendations, I found methodological deficiencies or uncertainties that were not properly conducted or explained by authors. This situation harms the validity of the main findings.
Searching evidence for recommendations is an objective process. However, the assessment of quality of the evidence is subjective and should be systematic and transparent; the information provided by authors as supplement is largely insufficient.
Several evaluated recommendations were indeed vague, and should be delivered with the warning that there is no enough evidence supporting them. Recommendations backed by scientific organizations and only based on expert opinions are not the best evidence. Importantly, the strength of recommendations is based not only on efficacy or effectiveness evidence from studies, but also on costs and patient values and preferences (1).
Developing searchable questions for vague recommendations is difficult. Authors made a good effort developing those questions, but several of the questions seem incomplete or erroneous in a closer examination of the results. Authors explained their broad and inclusive approach, but still some questions failed in finding the best evidence.
Finally, according to authors conflicts of interest were not declared by most of the guests of the medical show. I understand that US network broadcasters are compelled to report any conflicts of interest, and not revealing a conflict is illegal. I think this had to be explained to the non-American readers given the worldwide focus of the BMJ.
1. Andrews JC et al. GRADE guidelines: 15. Going from evidence to recommendation-determinants of a recommendation’s direction and strength. J Clin Epidemiol 2013;66:726-35.
Competing interests: No competing interests
During my academic career and in a leadership role at the US Food and Drug Administration, the usefulness of data always related to the rigor in its collection and analysis. Korownyk and colleagues tackled important issues in this paper. Unfortunately, the methods undermine its credibility.
The authors develop a new set of tools to evaluate health recommendations using methods that they do not validate. Without full transparency as to the specifics of the tools/processes, readers cannot allow themselves to be seduced to believe the authors’ conclusions. By applying these tools to conventional treatment guidelines or the dialog in practitioners’ offices, greater confidence in the approaches would be possible, though those experienced in Evidence Based Medicine would recognize much more work would be required before the methods would be considered validated.
Korownyk and colleagues make the mistake of serving as judge and jury without managing the inherent conflict that exists by making themselves the unblinded arbiters of believability, where that definition also is hidden from the readers.
Until the authors provide validation and/or share the information with sufficient transparency to allow for others to do so, this publication can only be considered a nidus for conversation, and neither support for their methods nor as an appropriate means to evaluate these television shows.
Competing interests: No competing interests
I personally have not watched Oz or the Doctors shows, but I am now intrigued to do so and conduct my own 'biased' study. It seems like the Oz show started in September 2009, and I can only assume that it evolved into its current form over the years. So a sampling from previous years would give different results than more current ones.
I agree with some of the commentators here about the methodological flaws of the study despite the lengthy response that is provided by the author, which seems dismissive of the validity of points that have been raised.
If I am understanding correctly, it is being argued that the purpose of this study was to describe the content of EBM in these shows. Considering the misinformation and misuse this publication received, it is important to point the fact that some of the recommendations categorized as 'no evidence' might be just because they were not previously investigated or might have conflicting evidence, which does not mean recommendations are made against existing evidence.
My understanding from all my readings is that these shows provide answers to some of the mundane questions of daily life. In these situations, do we know what an average physician recommends? One example of 'no evidence' from the authors was covering one's mouth with an arm while sneezing prevents the transmission of flu. What would an EBM-practicing physician recommend in this instance? If your sample contained many subjective or common sense recommendation examples, then what did we really learn from this investigation? A more robust design would have been to compare the same recommendations with several comparable medical practices. At this point, things would, of course, get very difficult because you would need to go back and ascertain data from an actual practice as you have done with the TV show. It is not a secret that many physicians have different practice patterns, and you will find many don't agree on certain treatment recommendations - just covering another physician's practice for a brief period of time can be quite educational.
Were these doctors making recommendations that jeopardized the public's lives and were harmful? Were they giving advice that was unequivocally shown to be wrong? Otherwise, if the shows' recommendations were assessed by the three experts in your study, all this publication presents is the opinion of those experts, not a systematic review or a prospective review (whatever that really means in epidemiological design sense).
Anyways, a good effort to get public attention. As an epidemiologist I always am wary about the inferences of my studies; hence, I hope to be cognizant of these design issues. Considering the professional discourse building around these TV shows and the fear of duping public with quack medicine, I sense the authors and the editor felt responsibility to get this information out to the public. However, sometimes it is better not to pursue a study than to misinform the public because that's how we end up with those recommendations that keep getting changed with each new study not because of the new scientific discoveries but inherent design flaws. Well, just because a study got published in The BMJ doesn't mean it is a great study as we have seen many examples in this and many other respectable journals in recent history.
Competing interests: No competing interests
We wish to respond to the letter from the Li’s pertaining to our study. It provides an excellent example for us to further discuss the importance of having a thorough understanding of the process of evidence evaluation when it comes to making medical recommendations and critiquing studies in the medical literature.
We have responded to each point in their letter below.
1) The title inaccurately claims it to be a “prospective observation study.” According to the U.S. Agency for Healthcare Research and Quality, a prospective observational study is: “a clinical research study in which people who presently have a certain condition or receive a particular treatment are followed over time and compared with another group of people who are not affected by the condition .” The article does not describe a clinical study, nor does it compare cohorts of individuals with and without a condition over time. Rather, the authors watched TV, documented statements, conducted Internet research, and tallied findings, conducting market research, not scholarly medical research.
Our Response: We prospectively recorded the TV programs as they were released, observed what was said and then studied/evaluated the evidence for the recommendations – therefore we called our investigation a prospective observation study.
The authors of the letter correctly state the USAHRQ definition of a prospective observational clinical study. However, this definition is irrelevant in this case because, our study was not a clinical study.
Given that most if not all of the top medical journals have over the last few years published reviews of media coverage we think it is safe to assume the medical community thinks such assessments constitute scholarly medical research. Other studies that address these types of research questions are also called cohort studies1,2
The authors of the letter suggest we “conducted internet research” to find evidence. We modeled our approach to finding evidence based on searches for systematic reviews (the highest level of evidence): searching “Pubmed/Medline, Embase, Cochrane Database of Systematic Reviews.” We added Natural Standard Database as it was recommended on the Dr Oz Show for natural products. We also searched Google because the “grey literature” (unpublished research) can sometimes only be found on the Internet.
2) The author’s statement “Consumers should be skeptical about any recommendations provided on television medical talk shows, as details are limited and only a third to one half of recommendations are based on believable or somewhat believable evidence” is flawed because only two television shows were examined yet their conclusions are generalized to the entire spectrum of such shows.
Our Response: As we stated clearly in our methods, we searched for internationally syndicated medical/health shows and we found only these two shows.
3) The authors assume that statements made on medical talk shows constitute medical recommendations. Clearly, not all statements they examined are medical recommendations, such as: “Give your self permission to experience emotional pain …”; “Look for sneaky words like homemade and legendary, on restaurant menus…”; “Laugh 15 minutes a day to decrease stress…”; “Eat foods from the fridge, not canned …”; “If you have thin nails, avoid nail gels …” Non-medical recommendations less likely to be supported by peer-reviewed research publications. Therefore, calculations of % recommendations supported by evidence as a reflection of show credibility are deeply flawed.
Our Response: First, most of the above mentioned recommendations have clear medical context. Emotional pain is often called adjustment disorder and is commonly seen in primary care. The reference to restaurant menus has to do with eating healthy and not gaining weight. Stress impacts health and reducing stress is thereby important. Most people should be able to easily see a medical connection to these statements. Second, due to the types of statements made on medical talk shows, we decided to record all recommendations, as excluding some could introduce bias at the earliest level of data collection. The study evaluated all recommendations and that is why the title of our article included only the word “recommendations”, not medical recommendations.
Our premise was that viewers watching a health show hosted by a medical doctor would perceive their recommendations as something they should consider for their health or well-being. We believe viewers need to realize that just because they hear something on TV from an expert doesn’t mean that adequate information was provided or the statements are backed up by solid evidence.
4) On the basis of watching only 2 episodes of each television show, the authors estimated 50% of recommendations would not be supported by evidence. They inappropriately applied a statistical calculation suggesting that “158 recommendations would give an 80% chance that the 95% confidence interval would have a precision within + 10%.” Applying good statistical design to the wrong type of study leads to a Type I error.
Our response: When we designed our study we wanted to ensure we included enough recommendations so that our point estimate of the number of recommendations with or without evidence would be fair and reasonable and by that we meant a precision of +/-10%. That is the only reason we provided a sample size calculation and made a statistical attempt to identify an estimate of the number of recommendations we needed to review.
The authors of the letter suggest we made a type 1 error but that is impossible because the term “type 1 error” refers specifically to what one does in “statistical hypothesis testing”. We did not do any statistical comparisons between the two shows or between anything – our results were strictly descriptive. The authors of the letter seem to have unfortunately conflated important critical appraisal issues for RCTs with our non-clinical study. When one does critical appraisal, by obvious necessity the assessment varies considerably depending on the type of study one wishes to assess. Even though statistics were only used for sample size estimation, we included a statistician in our study. Competent researchers know that sample size calculations are frequently based on estimation from limited previous data (in our case, there was none so we had to watch some episodes ahead of time to make an estimate). Even with lots of previous data, estimated event rates are frequently faulty. In our case, our estimation turned out to be very close to what we found.
5) The “recommendation categories” used were crude, overlapping, and inadequately defined. One recommendation category was “non-drug medical advice”, yet other categories were named “screening”, “infection prevention”, “dietary advice” which could be considered non-drug medical advice, making analysis uninterpretable
Our Response: We agree there was some subjectivity as to how we chose the categories. The broad variety and non-specific nature of many of the recommendations in the shows severely limited our ability to be more specific. Clearly if screening, infection prevention, etc were specific categories, than any recommendations included in non-drug medical therapy would not include screening, infection prevention, etc.
However, regardless of any categorization we used, it in no way affects any analysis we did of the evidence-base around the recommendations or the details of the information provided with recommendations. The bottom-line is these categories were used simply to provide readers with a rough idea of the types of recommendations made.
6) The accurate assessment of information presented on the shows requires expertise the authors do not appear to possess across biochemistry, dermatology, diabetology, dietetics, gastroenterology, immunology, infectious disease, psychiatry, and metabolomics. Oddly, the authors deliberately excluded expert opinion from their consideration of evidence “because all evidence in the programs were, by definition, being made by experts.” The authors thus take the position of being non-experts who are judging experts, a precarious position.
Our Response: We are not sure how the authors of the letter determined our level of expertise. Many of the investigators of this study regularly publish scholarly articles in high profile medical journals in the areas of diabetes, cardiology, infectious disease, and gastroenterology and participate in guideline development – but the definition of an expert is very much in the eye of the beholder.
Nonetheless, to properly do this study, evidence evaluation skills are the key skills required – not expertise in any specific medical area. Being an expert in a specific medical discipline does not ensure that a person is an expert at evidence evaluation. In fact, it may be just the opposite. An examination of clinical reviews found that experts performed reviews of inferior quality and the higher the level of expertise, the worse the evaluation of the literature.3
Expert opinion in medicine is important, but even non-experts in critical appraisal should be aware that virtually every hierarchy places “expert” opinion as the lowest form of evidence. Furthermore, in our opinion we were very liberal and fair in the level of evidence we required to be considered believable. In fact, peer review comments suggested that we were too liberal in considering what was evidence-based.
7) Throughout the article, the authors refer to “efficacy”, “safety”, “risk of harm”, and “disclosure of conflicts of interest” terms rarely applicable to television. By superimposing academic standards, the authors inadvertently applied bias using incorrect measures that affect their conclusions, for example “the near absence of conflict of interest reporting (<1%) further challenges viewer’s ability to balance the information provided.” This reveals their lack of understanding of U.S. Federal Government rules governing network broadcasters. The U.S. Federal Trade Commission regulates "unfair, deceptive or misleading" endorsements and mandates disclosure, while U.S. Federal Communications Commission rules require all network television show employees who receive a benefit of any sort for the inclusion of a good, product, brand or service on-air to disclose this fact.
Our Response: We agree that efficacy, safety and risk of harm are terms rarely used on television. Therein lies one of the biggest problems and risks associated with people on TV providing health recommendations. We think almost every recommendation that has to do with health should be associated with a discussion about efficacy, safety and risk of harm. If that is not done, as we describe in our paper, in our opinion these shows should be considered as strictly entertainment.
What standard TV shows should adhere to, is a discussion worth having. Regardless of whether we, or anyone, understand the U.S. Federal Government rules governing network broadcasters, we were still interested to see if during the shows potential conflicts were mentioned. Rules are often about setting minimal standards while conflict of interest pertains to ethical issues that we feel are always relevant when making recommendations regarding health.
Interestingly, the authors of the letter state they have no competing interests pertaining to their letter. At least one of the authors of the letter has appeared on the Dr Oz show as an “expert.” In addition their Angiogenesis foundation lists Lilly, Sanofi, General Mills and Bayer among others as sponsors. Certainly in a medical context and in our opinion any context, these competing interests are well above any reasonable threshold for disclosure.
8) Inexact and subjective measures related to the spectrum of “believability” were applied to references retrieved from the evidence search. The authors defined believability as the synthesis of “quality” “quantity” “type of evidence” as determined by a “vote” of the authors (self-proclaimed non-experts). A consensus on whether evidence was “believable” “somewhat believable” or “not believable” was generated, yet the authors admit there was a “wide diversity of reviewer interpretation of the evidence.”
Our Response: As was stated in the methods, there initially was a “wide diversity of reviewer interpretation of the evidence” and therefore we modified the protocol and the four primary investigators reviewed the evidence as a team. Occasionally there was discussion about what constituted believable evidence but 91% of the categorizations were unanimous.
It is unfortunate that the authors of this letter appear to be unfamiliar with the process of evidence review. Even experienced reviews using the easiest evidence to review and extract information from (randomized controlled trials), have high-levels of disagreement.4,5
9) Using such non-rigorous approaches, the stated conclusion is “roughly a third of the recommendations on The Dr. Oz Show and half of the recommendations on The Doctors were based on believable or somewhat believable evidence” and that in “over 1-in-3 … of the recommendations for The Dr. Oz Show … no evidence could be found.”
Our independent evidence search revealed two major systematic flaws in that invalidate the authors’ results and conclusions:
i) Incorrect questions were formulated for evidence search. Of 80 recommendations selected from The Dr. Oz Show, 24 (30%) of the questions were incorrectly developed by the authors. The results of these queries based these questions are erroneous.
Our Response: We find this comment intriguing. The manuscript authors have years of clinical practice and academic medicine. The authors of the letter assert we incorrectly developed questions from the recommendations. If this is the case, then how will the public, with no medical training, no clinical experience and no proficiency in the medical literature stand any chance of correctly interpreting the real meaning of the recommendations in these shows. The comment that we incorrectly formulated questions simply strengthens our argument that the recommendations often lacked adequate information to understand the meaning of the shows recommendations.
Regardless, we have reviewed the questions again and feel they reflect the recommendations. As a reminder, in a few cases, the recommendation was not clear enough to generate a question on it’s own so we had to review additional show content to develop a reasonable question. This may explain why some questions do not directly mirror the recommendations.
ii) The authors could not find evidence that was located by us with ease. For 41 instances in which the authors found “no evidence” supporting The Doctor Oz Show recommendations, our independent search found credible evidence for 22 (54%). Therefore, of 80 questions for The Doctor Oz Show, 76% are supported by evidence. This is in contrast to the authors’ conclusion that “over 1-in-3 ... of the recommendations for The Dr. Oz Show ... no evidence could be found.” Moreover, the evidence we found emanated from highly credible sources, such as Academy of Nutrition and Dietetics, American Association of Family Practitioners, American Cancer Society, American Heart Association, Centers of Disease Control, Harvard Medical School, National Institutes of Aging, National Institutes of Mental Health, and others.
Our Response: In the spirit of collegial transparency, when requested we provided each show with information about our findings. Along with that we stated “As per standard protocols in the sharing of research data, we would trust that only your office would use this information and it may not be disseminated further without our permission.”
The authors of this letter appear to have obtained access to the research information that we provided to the Dr. Oz show. We would be very interested to learn how they obtained this information.
It should also be noted the “highly credible sources” the authors of the letter mention are organizations, not specific published original research. It is unfortunate that despite our clear description in article, the authors of the letter seem to struggle with the definition of “evidence.” During our literature search we found similar statements by “highly credible sources” for a number of the recommendations, many of which were not supported by published evidence.
10) As reviewers for many prestigious medical journals, we are familiar with the scholarship standards, intellectual rigor, and investigator qualifications expected of research in major peer-reviewed medical journals. We are frankly surprised this article was published in BMJ. Medical journals must uphold scholarly standards so that readers are presented with strong and objective rationale, well-designed methodology, and expert interpretation of results that can impact on public health and patient outcomes. This article by C Korownyk, et. al. is so flawed it merits a withdrawal.
Our Response: A number of the investigators for this study are reviewers for medical journals and as a group have published over 100 peer-reviewed articles. Why that is relevant is unclear. What is relevant is the ethical integrity of the research. There are limits to our study, which we were forthright in detailing. In fact, our study was thoroughly peer reviewed and we received positive comments on the full disclosure of the limits in our study. Despite the authors of the letter’s assertions of their expertise and their many comments on our study, we feel they have failed to raise any valid concerns that we did not outline in the original article.
Given all the above and at least one of the authors of the letter’s undisclosed affiliation with the aforementioned medical talk show, one may wonder if their recommendation for manuscript withdrawal may stem from something other than the methodological quality of our study.
1. Yavchitz A, Boutron I, Bafeta A, Marroun I, Charles P, Mantz J, et al. Misrepresentation of randomized controlled trials in press releases and news coverage: a cohort study. PLoS Med. 2012;9(9):e1001308.
2. Schwartz LM, Woloshin S, Andrews A, Stukel TA. Influence of medical journal press releases on the quality of associated newspaper coverage: retrospective cohort study. BMJ. 2012 Jan 27;344:d8164.
3. Oxman AD, Guyatt GH. The science of reviewing research.
Ann N Y Acad Sci. 1993 Dec 31;703:125-33; discussion 133-4.
4. Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2013 Sep;66(9):973-81.
5. Hartling L, Fernandes RM, Seida J, Vandermeer B, Dryden DM. From the trenches: a cross-sectional study applying the GRADE tool in systematic reviews of healthcare interventions. PLoS One. 2012;7(4):e34697.
Competing interests: No competing interests
We reviewed C Korownyk, et. al. (BMJ 2014;349:g7346) and found the study to be flawed in its conception, design, execution, analysis, and conclusions:
1) The title inaccurately claims it to be a “prospective observation study.” According to the U.S. Agency for Healthcare Research and Quality, a prospective observational study is: “a clinical research study in which people who presently have a certain condition or receive a particular treatment are followed over time and compared with another group of people who are not affected by the condition .” The article does not describe a clinical study, nor does it compare cohorts of individuals with and without a condition over time. Rather, the authors watched TV, documented statements, conducted Internet research, and tallied findings, conducting market research, not scholarly medical research.
2) The author’s statement “Consumers should be skeptical about any recommendations provided on television medical talk shows, as details are limited and only a third to one half of recommendations are based on believable or somewhat believable evidence” is flawed because only two television shows were examined yet their conclusions are generalized to the entire spectrum of such shows.
3) The authors assume that statements made on medical talk shows constitute medical recommendations. Clearly, not all statements they examined are medical recommendations, such as: “Give your self permission to experience emotional pain …”; “Look for sneaky words like homemade and legendary, on restaurant menus…”; “Laugh 15 minutes a day to decrease stress…”; “Eat foods from the fridge, not canned …”; “If you have thin nails, avoid nail gels …” Non-medical recommendations less likely to be supported by peer-reviewed research publications. Therefore, calculations of % recommendations supported by evidence as a reflection of show credibility are deeply flawed.
4) On the basis of watching only 2 episodes of each television show, the authors estimated 50% of recommendations would not be supported by evidence. They inappropriately applied a statistical calculation suggesting that “158 recommendations would give an 80% chance that the 95% confidence interval would have a precision within + 10%.” Applying good statistical design to the wrong type of study leads to a Type I error.
5) The “recommendation categories” used were crude, overlapping, and inadequately defined. One recommendation category was “non-drug medical advice”, yet other categories were named “screening”, “infection prevention”, “dietary advice” which could be considered non-drug medical advice, making analysis uninterpretable
6) The accurate assessment of information presented on the shows requires expertise the authors do not appear to possess across biochemistry, dermatology, diabetology, dietetics, gastroenterology, immunology, infectious disease, psychiatry, and metabolomics. Oddly, the authors deliberately excluded expert opinion from their consideration of evidence “because all evidence in the programs were, by definition, being made by experts.” The authors thus take the position of being non-experts who are judging experts, a precarious position.
7) Throughout the article, the authors refer to “efficacy”, “safety”, “risk of harm”, and “disclosure of conflicts of interest” terms rarely applicable to television. By superimposing academic standards, the authors inadvertently applied bias using incorrect measures that affect their conclusions, for example “the near absence of conflict of interest reporting (<1%) further challenges viewer’s ability to balance the information provided.” This reveals their lack of understanding of U.S. Federal Government rules governing network broadcasters. The U.S. Federal Trade Commission regulates "unfair, deceptive or misleading" endorsements and mandates disclosure, while U.S. Federal Communications Commission rules require all network television show employees who receive a benefit of any sort for the inclusion of a good, product, brand or service on-air to disclose this fact.
8) Inexact and subjective measures related to the spectrum of “believability” were applied to references retrieved from the evidence search. The authors defined believability as the synthesis of “quality” “quantity” “type of evidence” as determined by a “vote” of the authors (self-proclaimed non-experts). A consensus on whether evidence was “believable” “somewhat believable” or “not believable” was generated, yet the authors admit there was a “wide diversity of reviewer interpretation of the evidence.”
9) Using such non-rigorous approaches, the stated conclusion is “roughly a third of the recommendations on The Dr. Oz Show and half of the recommendations on The Doctors were based on believable or somewhat believable evidence” and that in “over 1-in-3 … of the recommendations for The Dr. Oz Show … no evidence could be found.”
Our independent evidence search revealed two major systematic flaws in that invalidate the authors’ results and conclusions:
i) Incorrect questions were formulated for evidence search. Of 80 recommendations selected from The Dr. Oz Show, 24 (30%) of the questions were incorrectly developed by the authors. The results of these queries based these questions are erroneous.
ii) The authors could not find evidence that was located by us with ease. For 41 instances in which the authors found “no evidence” supporting The Doctor Oz Show recommendations, our independent search found credible evidence for 22 (54%). Therefore, of 80 questions for The Doctor Oz Show, 76% are supported by evidence. This is in contrast to the authors’ conclusion that “over 1-in-3 ... of the recommendations for The Dr. Oz Show ... no evidence could be found.” Moreover, the evidence we found emanated from highly credible sources, such as Academy of Nutrition and Dietetics, American Association of Family Practitioners, American Cancer Society, American Heart Association, Centers of Disease Control, Harvard Medical School, National Institutes of Aging, National Institutes of Mental Health, and others.
As reviewers for many prestigious medical journals, we are familiar with the scholarship standards, intellectual rigor, and investigator qualifications expected of research in major peer-reviewed medical journals. We are frankly surprised this article was published in BMJ. Medical journals must uphold scholarly standards so that readers are presented with strong and objective rationale, well-designed methodology, and expert interpretation of results that can impact on public health and patient outcomes. This article by C Korownyk, et. al. is so flawed it merits a withdrawal.
Sincerely,
William W. Li, M.D.
President and Medical Director The Angiogenesis Foundation
Vincent W. Li, M.D., MBA
Chief Scientific Officer
The Angiogenesis Foundation
Competing interests: No competing interests
To the Editor:
The article by Korownyk et al regarding the credibility of medical TV talk shows has gathered significant coverage in the lay press. In my opinion, this is unfortunate, as the broad and sound-bite worthy conclusions of the study, while dramatic and provocative, are fundamentally flawed and do a disservice both to the physicians on the shows as well as the shows themselves. Statements such as "consumers should be skeptical about any recommendations provided on television medical talk shows" may make headlines, but before making such statements (and before publishing them), one should be assured the data upon which they stand are credible. And the data detailed by Korownyk et al are not.
The methods and description of the data acquisition raise significant questions about the validity of the conclusions drawn. Numerous problems exist: this was not a prospective study, the outcomes measured are vague, the protocol was varied (“iterative”?), data points “voted” upon, and medical and non-medical advice mixed and compared. Additionally, the schema for classifying program statements as either upheld by standards or not is difficult to follow and flawed; “believability” is simply not an accepted classification for medical evidence.
Fundamentally, this study attempts to compare apples and oranges and suggests preconceived bias. Medical TV talk shows are not guidelines or a forum for the dispensation of personal medical advice (as the authors document, “consult your physician” is frequent, and good, advice). Medical TV talk shows discuss and educate in an entertaining format and reach millions of people daily. The authors, as do I, care for and treat patients one at a time. The Doctors and The Dr. Oz Show stimulate millions of people to think about their health and hopefully encourage them to seek consultation with their physician when needed. Criticizing these shows for not adhering to the same standards as the sacrosanct patient-physician relationship is condescending and suggests an elitist attitude that fails to recognize the worth of these educational and entertaining programs to their audience. The Dr. Oz Show and The Doctors have probably helped more people over time than any single physician (now that would be an interesting hypothesis for study). And as we all know, if we don’t like the show, we can always turn off the TV.
Competing interests: No competing interests
The laborious attempt by Korownyk et al. to examine the recommendations made on two popular television series, The Dr Oz Show and The Doctors, highlights a lack of supporting ‘believable evidence’ and recommendation details that would allow for informed decisions by viewers. In response to the further ensuing discussion, the authors cautioned against jumping to conclusions as some recommendations, while seemingly reasonable are not supported by research, reminding everyone the important caveat that lack of evidence does not equate to ineffectiveness or harm.
As medical students, we find this article and the ensuing discussion refreshing as it revolves around two important themes in clinical practice - evidence-based medicine (EBM) and physician responsibility. The lack of suitable evidence found for many of these recommendations represent areas of medicine and healthy living that require scientific scrutiny. It also serves as a reminder of the need to re-examine conventional wisdom and assumptions within reason for the benefit of patients.
Physician responsibility in communicating information in such a manner as to facilitate an informed choice is also pertinent and goes hand-in-hand with EBM. This is especially relevant in the context of televised medical talk shows given the wide viewership of such shows and the influence of celebrity endorsement on health and lifestyle choices. This is in lieu of several studies showing the effect of celebrity endorsement on lifestyle choices,[1–3] with the effect possibly greater for medical celebrities. In light of this, it may be worth considering the introduction of relevant guidelines and regulations specific to medical talk shows to ensure that even when in front of an audience, we as physicians do not forget our ethical duty to patients.
1 Boyland EJ, Harrold JA, Dovey TM, et al. Food choice and overconsumption: effect of a premium sports celebrity endorser. J Pediatr 2013;163:339–43. doi:10.1016/j.jpeds.2013.01.059
2 Dixon H, Scully M, Wakefield M, et al. Parent’s responses to nutrient claims and sports celebrity endorsements on energy-dense and nutrient-poor foods: an experimental study. Public Health Nutr 2011;14:1071–9. doi:10.1017/S1368980010003691
3 Sterling KL, Moore RS, Pitts N, et al. Exposure to celebrity-endorsed small cigar promotions and susceptibility to use among young adult cigarette smokers. J Environ Public Health 2013;2013:520286. doi:10.1155/2013/520286
Competing interests: No competing interests
To the Editor:
It is exceptionally heartening to finally see a scientific investigation into the claims by one of the most popular medical professionals on mass media. In light of the previous rapid responses, perhaps it would be useful to have one from a young trainee in the medical field – an individual’s perspective on our training and aspirations.
The article’s insight is especially relevant considering the recent incidences of significant scientific fraud. The recent stimulus-triggered acquisition of plurarity (STAP) stem cells (1–3) and the constant indictment of mispublished articles (4) highlights the issue of integrity in the basic training of doctors and scientists. The recent resurgence of measles in America (5) is partially due to parents mistakenly buying into the fateful Wakefield paper, published thirteen years ago in the Lancet and ostensibly linking autism and vaccination (6). The wake left by scientific misinformation, especially at the highest levels, is long indeed. Two issues are central to this article – professional integrity and being conversant in the scientific process.
In this era of evidenced-based medicine, medicine and science are inextricably intertwined. Considering the tremendous role of medical science and technology in the lives of people, it is imperative we respect the duty inadvertently accorded to us as scientists and doctors. It is thus alarming that even from just two medical talk shows, this study found that ‘approximately half of the recommendations have either no evidence or are contradicted by the best available evidence. Potential conflicts of interest are rarely addressed’. The mass media has not taken kindly to this, and deservedly so (7–12).
Given the increasing integration of science with medicine in the professional lives of doctors, it is imperative we continue reflecting on the selection and development of character and values amongst medical trainees from the get go. We as trainees do not often appreciate this whilst in medical school, missing a valuable opportunity for character development. Be it the increasing role of doctors in advancing the science of medicine, or the enhanced visibility on national TV, we have a duty to ensure the public and our patients are provided with evidence-based recommendations. Any deviation undermines the trust we are accorded - a doctor that lies to his patient loses far more than the patient’s trust.
Yet we must never forget that doctors, be they on national TV or in the clinic, are still results of both nature and nurture.
It is a curious interplay. Even at medical school, we are held to higher standards. We are hyperselected for academic capacity. We have to demonstrate competence with a significant body of knowledge, ensure we can effectively communicate with our patients and colleagues, conduct research, prepare for competitive applications, be able to withstand long hours and never-ending work, while remaining in tip-top performance with impeccable ethics. And this does not yet factor having to consistently excel pre-medical school, the staggering debt many occur, and the significant impact on our personal lives, health and mental well-being. We are all in this gutter, and not everyone has the time or the wherewithal to look at the stars.
Yet in this race to help others, we often forget to live. We regularly cite this tremendous investment as reason for the rewards we assume we are beholden to.
Coupled with overly-driven individuals that tick all the boxes for professional success, these external pressures can possibly accentuate this desire for success. The public recognition and fame that such professional success brings within grasp makes this all the more concerning. Are we all truthfully the individuals held in high regard by the general public?
Dr Mehmet Oz has had a fantastic, inspirational, enviable career. The son of immigrant Turkish parents, he is a renowned cardiac transplant surgeon.
He has, in no uncertain terms, saved thousands of lives, and in so doing, helped countless families. And coupled with his public work, is at the forefront of the public eye.
Such individuals are the heroes we medical initiates seek to emulate. Their capabilities and track record as doctors are what almost every medical student and junior doctor aspires too. But it is incongruous what happens when such esteemed individuals come into the limelight. Dr Oz, in addition to claims investigated in this paper, faced a congressional hearing into the scientific validity of the weight loss products he has endorsed as ‘miracles’ (13). The individuals who promoted green coffee extract as a weight loss solution on the Dr Oz show have just been fined $9 million by the Federal Trade Commission (14). Senator Claire Mccaskill, head of the senate panel on consumer protection, succinctly addresses the crux of the congressional hearing, reflecting what both the public and many of us in medicine and science think, 'Why is it that when you have this amazing megaphone, and this amazing ability to communicate, why would you cheapen your show by saying things like that.' The actions of a minority, much less the public champions of medicine, undermine the tireless, selfless work of the majority.
It takes inner courage to stand up and point out the fallacies that our medical training was supposed to avoid. Even as students, in the rush to complete our firm sign offs, to master the five-minute history, we forget that patients are human. We forget we are human. How then could we even be expected to remember to reflect? How then, can we lead the doctors and the public of tomorrow? Those who are then fortunate to finally make it past this incredible rat race and into the position of Dr Oz, what then?
As medical professionals, we truly need to accord our very own with the same sincerity, dedication and nurture we afford our patients. We are not simply training technicians. We are, after all, human.
REFERENCES
1. Cyranoski D. Papers on “stress-induced” stem cells are retracted [Internet]. Nature. 2014 [cited 2014 Jul 2]. Available from: http://www.nature.com/news/papers-on-stress-induced-stem-cells-are-retra...
2. Obokata H, Wakayama T, Sasai Y, Kojima K, Vacanti MP, Niwa H, et al. Stimulus-triggered fate conversion of somatic cells into pluripotency. Nature. Nature Publishing Group; 2014 Jan 29;505(7485):641–7.
3. Obokata H, Sasai Y, Niwa H, Kadota M, Andrabi M, Takata N, et al. Bidirectional developmental potential in reprogrammed cells with acquired pluripotency. Nature. Nature Publishing Group; 2014 Jan 29;505(7485):676–80.
4. Qiu J. Publish or perish in China. Nature. 2010;463(January):142–3.
5. Healy J, Paulson M. Vaccine Critics Turn Defensive Over Measles [Internet]. The New York Times. 2015 [cited 2015 Jan 30]. Available from: http://www.nytimes.com/2015/01/31/us/vaccine-critics-turn-defensive-over...
6. Wakefield AJ, Murch SH, Anthony A, Linnell J, Casson DM, Malik M, et al. Ileal-lymphoid-nodular hyperplasia, non-specific colitis, and pervasive developmental disorder in children. Lancet. 1998;351:637–41.
7. Novak M. Dr. Oz Is Still Full of Shit [Internet]. Gizmodo. 2015 [cited 2015 Jan 26]. Available from: http://gizmodo.com/breaking-dr-oz-is-still-full-of-shit-1681865968?utm_c...
8. Terrence McC. Half of Dr. Oz’s medical advice is baseless or wrong, study says [Internet]. The Washington Post. 2014 [cited 2015 Jan 25]. Available from: http://www.washingtonpost.com/news/morning-mix/wp/2014/12/19/half-of-dr-...
9. Park A. Here’s What Experts Say About the Advice on Dr. Oz and The Doctors [Internet]. Time. 2014 [cited 2015 Jan 25]. Available from: http://time.com/3642533/dr-oz-advice/
10. Kaplan K. Real-world doctors fact-check Dr. Oz, and the results aren’t pretty [Internet]. LA Times. 2014 [cited 2015 Jan 25]. Available from: http://www.latimes.com/science/sciencenow/la-sci-sn-dr-oz-claims-fact-ch...
11. Kleeman S. We Now Have Scientific Proof That Dr. Oz Is Full of It [Internet]. Science Mic. 2014 [cited 2015 Jan 25]. Available from: http://mic.com/articles/106802/we-now-have-scientific-proof-that-tv-doct...
12. Abel J. Study: less than half of Dr. Oz’s recommendations are actually supported by evidence [Internet]. Consumer Affairs. 2014 [cited 2015 Jan 25]. Available from: http://www.consumeraffairs.com/news/study-less-than-half-of-dr-ozs-recom...
13. Christensen J, Wilson J. Congressional hearing investigates Dr. Oz “miracle” weight loss claims [Internet]. CNN. 2014 [cited 2015 Jan 25]. Available from: http://edition.cnn.com/2014/06/17/health/senate-grills-dr-oz/
14. Federal Trade Commission. Federal Trade Commission v Genesis Today Inc., Pure Health LLC, Lindsay Duncan. Washington D.C.: Federal Trade Commission; 2015. p. 23.
Competing interests: No competing interests
As “alternative medicine” migrated to become “integrative medicine” and integrative medicine grew from a few isolated centers to become a daily feature on the Dr. Oz show, concerns about claims led to a British Medical Journal article published last month. The study is entitled: “Televised medical talk shows—what they recommend and the evidence to support their recommendations: a prospective observational study.”
Not a bad subject to research, given the influence of Dr. Oz and that of a second show examined, The Doctors. According to the analysis, Dr. Oz offers an average of 12 recommendations per show. The authors' findings, and to the BMJ’s credit, the ensuing BMJ discussion, point out what glass houses we all live in. This is especially so as we shift focus from disease suppression to health creation.
The University of Alberta research team concluded: “Recommendations made on medical talk shows often lack adequate information on specific benefits or the magnitude of the effects of these benefits. Approximately half of the recommendations have either no evidence or are contradicted by the best available evidence. Potential conflicts of interest are rarely addressed. The public should be skeptical about recommendations made on medical talk shows.”
In fact, the researchers found that for 46% of the roughly 50-60 recommendations a week, there was some kind of evidence from case reports on up. They then applied a higher but still what they characterized as a "liberal” standard toward evidence and concluded that 33% made the grade.
In a BMJ forum, a reader quickly responded: “One reaction that one could have to this article is just how evidence-based the medical talk shows are compared to standard physician clinical practice.”
The authors responded: “As we note in the paper, in reviews of practicing doctors, only about three quarters of what is recommended is supported by evidence. In fact some research suggests that even for guidelines, moderate or good evidence is present for only about 50% of their recommendations. The highest level of evidence, randomised controlled trials, is present for only 10-15% of the recommendations.”
In short, the authors might have concluded that like the rest of medicine, approximately half of the recommendations have either no evidence or are contradicted by the best available evidence. The public should be skeptical about recommendations made by any healthcare professional in any circumstance.
So where does this leave us? Clearly, humility on evidentiary claims is a must for all of us. When it comes to high level evidence, we all are finding our way amidst levels of ambiguity. A recorded discussion on the BMJ site of the paper by the authors notes that they would expect significant evidentiary problems “if we record what is said in medical offices and pharmacies.”
Dr. Oz typically makes recommendations that would be expected to have a lower level of evidence than, for instance, the use of pharmaceuticals. His role is in opening listeners to options that are typically newer choices. Not only are they new. They typically have lower financial backing than conventional drugs.
Perhaps even more important is that Dr. Oz, like most of us in integrative medicine, comes from a philosophical approach that seeks to create health. What is a clinician to do in working with a patient if he or she has high level evidence for how to suppress symptoms of disease yet only an impressionistic or montage of evidence on the steps toward health creation?
As the authors state in the podcast discussion, Dr. Oz “haters” can find plenty to support their perspectives. Those for whom Dr. Oz is a champion can marshal evidence that makes the evidence-base for his claims look decent, relatively. Notably, the authors state that they are "disappointed that the overwhelming commentary seems to be that our study somehow proves that Dr. Oz or the Doctors are quacks or charlatans or worse. Our data in no way supports these conclusions.”
In all cases, we must consider the color of tinting any discussant has chosen for the evidentiary walls of his or her fragile dwelling.
Competing interests: No competing interests
Critique by Woolf (Part 1)
The recent upsurge in criticisms of Dr. Oz has led to unfounded claims based on the flawed study by Korownyk et al. in the December 2014 issue of the BMJ. The extensive deficiencies in the study could be ignored if it were not being used as an ad homonym tool to cast aspersions on a physician, but its continued use for just this purpose requires honest disclosure of the numerous problems with the paper. I have spent my career in the critical appraisal of evidence and the production of evidence-based practice guidelines and am disturbed to see these terms used improperly by the authors to cloak a study that rigorous reviewers would send back for revision. I fully support the goal of exposing incompetent medical advice, but this study cannot be cited as empirical support for any such claim given its weaknesses. The problems are too extensive to fit within the 1000-word limit for Rapid Responses so I am submitting my critique in three parts. Part 1 covers General Problems with the paper:
Two fundamental premises for the study are dubious at their core.
First, the authors make the assumption that the standards of evidence we use to critique physician guidance (e.g., at the bedside or in clinical practice guidelines) are appropriate for a televised talk show. Television—and certainly a talk show—is a wholly different medium, intended to mix medical advice with a spectrum of content by inviting guests with different perspectives to both educate and entertain viewers. Applying the rules of evidence that academics and professional societies use for clinical practice guidelines to the conversational content of a talk show is a misguided application of a tool not designed for this purpose. As noted below, academics with more experience than the authors know that conversational analysis requires a different methodology than the clumsy attempts the authors employed to evaluate 80 broadcasts. Reasons for describing their methods as clumsy are detailed below.
Second, even if one equates talk shows with physician counseling, it’s fair to say that the conversations that occur at the bedside or the surgery are also of mixed content, only some of which can be reduced to explicit statements supported by published evidence. The core findings of Korownyk et al.—e.g., that only 46% of recommendations were supported by evidence—have little meaning without a controlled comparison with the content of what physicians communicate at the bedside. David Sackett often estimated that only 10% of medical practice is based on evidence. I posit that the percentages would be far lower than 46% if the authors had applied the same methodology to 80 audio recordings of physicians advising patients.
Like Dr. Oz, physicians deliver informal conversational advice and use that connection to build their relationships with patients. A physician at the bedside who censored his/her statements to include only assertions supported by RCTs would be a robot and would fail as an effective medical historian, diagnostician, and counselor—unable to establish a foundation of trust to be persuasive either in explaining scientific evidence or convincing patients to obtain evidence-based services. Good doctors do this to connect with their patients, and the same is no doubt true for television doctors like Dr. Oz who need to connect with their audience. In either context it is misguided to judge the conversation by conducting searches on PubMed. I have a lengthy career background in the critical appraisal of evidence and the application of evidence-based medicine in clinical practice guidelines, but this is a misuse of these tools. Asking about the family or showing empathy for recent stresses are as important with patients as delivering dry facts about RCT results. Patients are more likely to agree to tests or treatments based on trust and respect, which comes from more than parroting published evidence.
While I laud the goal of trying to identify misinformation circulated by the media, I am puzzled that the authors—family physicians themselves—would design a study that so naively ignores this context. For example, they admit to being “unaware of the non-specific nature of many statements and recommendations given on medical television talk shows.” It should be obvious from clinical experience—even for someone who has never watched television—that what physicians say is a mixture of specific and non-specific content, some of it instructive, some empathic, some reflective, etc. It’s unclear what would possess investigators to hypothesize that every form of advice uttered by a physician, on matters that can stray as far from disease didactics as marital stress and obtaining insurance preauthorization for procedures, should be defended by a peer-reviewed journal article.
The unintended consequence of undermining the credibility of a television physician, as has occurred here, is the adverse impact on population health in a country that, perhaps unlike the UK, is heavily influenced by the medium. In the United States, a popular spokesman like Dr. Oz likely has greater influence on patient behavior and uptake of evidence-based services than the stacks of journal articles and practice guidelines that fill professional publications. The 80 recommendations examined by the authors include questionable treatments but also valuable admonitions to obtain recommended cancer screening tests, to eat a heart-healthy diet, to be physically active, to seek help for emotional disorders, and other centerpieces of health promotion and disease prevention. Denigrating the reputation of the spokesman comes at a cost of engendering reluctance among viewers to accept the program’s core advice to adopt healthier lifestyles.
Denigration by the BMJ is fair game if the evidence is solid, but this BMJ paper suffers from glaring methodological errors that would be problematic for any study, regardless of its topic. Red flags that apparently escaped the attention of reviewers included inadequate documentation of sampling methods; reliance on opaque, subjective, and inconsistent criteria for outcome measurement (what the authors called “an iterative approach to the study design”); use of invalid instruments; and omission of source data. It is our understanding that the authors are unwilling to share their source data publically, which only serves to make their research more suspect.
Further comments follow in Part 2.
Sincerely,
Steven H. Woolf, MD, MPH
Professor, Department of Family Medicine and Population Health
Virginia Commonwealth University
Competing interests: No competing interests