Televised medical talk shows—what they recommend and the evidence to support their recommendations: a prospective observational study

I appreciate the response from Dr. Korownyk and, while I continue to disagree on the science, I agree with her advice to not belabor this debate on these pages. My concerns with her methods and conclusions, which persist, are documented sufficiently in my lengthy Rapid Responses. However, I feel compelled to defend myself against her suggestion that I am not a “trusted professional” and her unfounded claim that I had a conflict of interest in reviewing her study. I am an elected member of the Institute of Medicine and have published more than 170 peer-reviewed papers. My publications (especially in my early career) reveal extensive writing on the proper methods of critical appraisal of scientific evidence and evidence-based practice guidelines, but I have never written about talk shows or television doctors to justify a claim of a conflict of interest. Throughout my entire career I have steadfastly refused funding from pharmaceutical companies or any other industry, nor did I receive a penny to review Dr. Korownyk’s article. Her claim that I have an “association with the Dr. Oz show” is false and asserted without evidence. I know Dr. Oz professionally through a mutual colleague and he shared Dr. Korownyk’s data file after asking me to conduct an independent review of the study. I was informed that Dr. Korownyk had provided the data file for independent review, a customary practice among the researchers I associate with, but I was never informed of Dr. Korownyk’s request that it “not be disseminated further without permission.” I agree with Dr. Korownyk that there should be room for open scientific discourse, but I see no need for the pages of the BMJ to host ad hominum accusations, whether it be the use of arguable analytic methods to assail television doctors or the use of speculation to challenge the integrity of those who question the science behind those analytic methods.

In February 2015, in response to a request from The Dr. Oz Show, we provided the show’s producer with the recommendations and the clinical questions created from our evidence review. With this data they had enough information to perform their own systematic evidence review of the shows’ recommendations. This information was provided with the following statement that “As per standard protocols in the sharing of research data, we would trust that only your office would use this information and it may not be disseminated further without our permission.”
The show then shared our data (without permission) with “4 trusted professionals”. We contacted The Dr Oz show and they wrote to us saying these individuals included Dr. Steven Woolf, Dr. Adrian Hernandez Diaz, Dr. Cornelius Dyke and Dr. William Li (names published at the request of the BMJ). We do not know if these “trusted professionals” were made aware of the fact this was confidential research information and should not be published without our permission.
All four of these “trusted professionals" proceeded to publish critiques of our research in the BMJ electronic letters section; and some of them actually included some of our research data in their response. At the time of our original submission of this response, (May 19), none of these “trusted professionals” declared their association with The Dr. Oz Show as a potential conflict of interest nor that they attained our data from The Dr. Oz Show. Furthermore, a few of their responses seem not intended for scientific debate or constructive suggestions for improvement. Dr. Hernandez has just (June 3) submitted a second critique, this time acknowledging his receipt of confidential data although does not declare from where he obtained this data. From these named “trusted professionals” 7 negative responses have been submitted. Ironically, all accuse us of inadequate transparency while none have declared their conflict of interest, and all have had contact with The Dr. Oz Show. It is not clear if any other letter writers have any affiliation with The Dr. Oz Show.

Nonetheless, in the spirit of collegial debate, we will address the key issues raised in the most recent rapid responses, some of whom have received our data from The Dr. Oz Show.
With respect to Dr. Woolf’s review, readers should bear in mind the aforementioned undisclosed interaction with the Dr. Oz Show. We note that he focuses solely on The Dr. Oz Show and never once mentions the other show we reviewed. Dr. Woolf’s review of our study is extensive, as he comments on anything potentially related no matter how peripheral, in his attempt to discredit the study.
His concerns do not relate to the findings of our study but rather focus on every aspect of our methods, mostly suggesting we did not provide enough information (even for results not presented in our study). If we had followed Dr. Woolf’s suggestions, our methods would have been even longer than his letter. Interestingly, the peer reviewers of our study had suggested we shorten and simplify the methods. Such is the reality of peer-review publication.
Dr. Woolf’s first fundamental argument is we are being unfair in assuming the “standards of evidence we use to critique physician guidance… are appropriate for a television show. ” Similarly Ayse Tezcan suggests the comparator standard should not be “evidence but what standard doctors do.” (PhD candidate Tezcan was not listed by the Dr. Oz Show as one of the “trusted professionals” with whom they shared data and therefore, to our knowledge, has no association with that show.) Dr. Woolf suggests our data contributes to “denigrating the reputation of a spokesman.” It is imperative for us to stress that in no way did we imply any value judgment to the findings we reported. We have also repeated this in previous responses. We have many times stated that sometimes guidelines and other recommendations have similar levels of evidence to what we found on these shows. We find it baffling that simply reporting on the evidence available to support these shows is considered denigrating. Other studies have compared clinical practice and guidelines to evidence (research) and we felt a similar comparison was the most reasonable. While some of our critics suggest the bar should be lowered for television, given the impressive viewership for these shows we strongly disagree.
Their second concern is that we should have compared these results to practicing clinicians. No one has so much as described the recommendations for medical shows before let alone completed an evidence review for a sample of the recommendations. This is an example of Dr Woolf’s faulty argument. Simply stating he feels we should have done more or done something differently does not invalidate nor change what we observed in our review.
Dr. Woolf found our sampling via “internet search” inadequate to determine which medical talk shows to review but failed to indicate the programs we missed.
Certainly his suggestion that we should have used qualitative tools, is one way to analyze this type of data. However, ours was a quantitative approach, another reasonable option. Our approach makes his suggestions regarding specific qualitative tools somewhat nonsensical.
Dr. Woolf’s comment that the data was difficult to obtain is interesting as he never approached us to request more data. He raises a number of captious issues: our title (which we addressed previously); the classification (which we acknowledged was tricky but did not in anyway affect the analysis of the evidence nor our assessment of the reporting of both benefits and harms); that we broke one recommendation into two (there were 160 recommendations, thus effecting 0.6% of our data); etc. Dr. Woolf hunts for faults while ignoring strengths like our standard data extraction techniques. He does not mention our dual independent review but implies our research assistants lacked skills to perform the task without ever having met them or assessing their skills. He states that David Sackett estimated only 10% of practice is evidence based. David Sackett and colleagues have indeed published on this topic, and found 82% of interventions offered to patients by a general medical team were evidence based.1 We would commend Dr. Woolf to a re-reading of Sackett’s paper as he would find that Sackett’s methods of evidence classification were not dissimilar to ours and not nearly as robust as he suggests they must be.
Dr. Sackner-Bernstein and Dr. Woolf were correct in stating we used unvalidated tools in our assessment. Given that this is the first evaluation of the evidence pertaining to televised medical shows, it is not surprising there were no validated tools to use. However, it is important to realize that even “validated” tools in evidence-based medicine, are not that reliable. In fact, there is poor agreement and inconsistency (often Kappa <0.5) in tools like GRADE and Cochrane Risk of Bias.2,3 We noted this in the original dual review of our study and attempted to improve this by using a committee of four experienced evidence-based reviewers and requiring agreement in three of four reviewers. We decided to use the term “believable” as we felt it would be more understandable to non-scientific readers than other categorizations that were never developed to assess medical recommendations on a TV show. Ideally a full systematic review for each of the evidence questions would have been ideal but the time commitment to complete 160 systematic reviews is simply prohibitive.
In the first Tezcan seems unclear on the objective of the study. The objective of our study, as with most research papers, is at the end of the introduction. Tezcan is concerned that our sampling was from a previous year of shows. A project of this magnitude cannot be accomplished and published in a few months. Clearly, only future research would be able to determine whether current shows are similar to past shows. With regard to what “no evidence” meant: it meant there was no evidence. That means research has not been published on that specific clinical question. However, it does not mean there is conflicting evidence. When there was conflicting evidence pertaining to a particular recommendation we deferred to the highest level of evidence we could find. If conflict occurred at a similar level of evidence, we gave the benefit of the doubt to the shows and concluded the evidence supported the recommendation. In the second posting 10 days later, Tezcan states she has now watched more episodes of The Dr. Oz Show than our team did (40 episodes). She then suggests another idea for a study of these shows, which we encourage her to pursue.
We have been exceptionally clear about the limits of our study including the challenges of researching this previously unexamined area, particularly the non-specific and subjective nature of the recommendations. Complaints, such as from Dr. Woolf, about how we generated different questions compared to how he would, simply reinforces our findings regarding the subjective nature of the recommendations. It should be noted we also used other information from the show to try to more accurately define the questions and represent the context of each recommendation more fairly. Regardless, the exercise by our critics demonstrates why this research should be replicated from the start. Simply jumping in at the end does not provide Dr. Oz’s “trusted professionals” with adequate context of these recommendations to develop adequate questions, nor a complete understanding of the process.
We would now like to address the request for the public release of our data. In addition to the normal peer review process that occurred before the study was published, we have offered full disclosure of our data to the BMJ to verify any aspect of our research.
We have concerns about full public release of our data. First, a number of the requests for release of data have not been related to scientific purposes. For example, we have had requests to use our data for online games, website development and to further particular agendas. This, as was pointed out in our first rapid response, was neither the purpose nor would it meaningfully add to the scientific discussion around our paper.
Second, we believe people with a stance for or against the shows may seek to repeat our specific evaluation with the result of verifying their predisposition bias. Indeed, the most caustic responses have come as a result of our data being shared (without our permission) by The Dr. Oz show, without any declaration of potential competing interests. We have strong concerns that any further analysis of our data will likely be biased to assert a specific agenda.
Third, these data were very time consuming to collect and process. We are still considering further evaluations and possible future publications. Complete data release could potentially negate this.
Fourth, some of the criticisms about our study arise from our methodology and possible subjectivity in our interpretations. We have been clear about our methodology and its limitations and have encouraged others to repeat the research in its entirety including: independent dual review of episodes, identifying recommendations, delineating stronger recommendations, elucidating the details around the recommendations, generating clinical questions from the recommendations, searching evidence for the recommendations, and finally assigning the final results of the evidence assessment. We would encourage authors such as Dr. Woolf to develop a framework for this type of study. Replication from the start is the most reliable way to verify and improve upon research. We would be more than happy to assist other truly independent academic researchers to replicate our study, which could include a consideration for sharing data with independent scientists who present a research and analysis plan.
Finally, everyone should be reminded that these shows are public and the data is readily available for people willing to watch and systematically record the data as we did. To assist in the evaluation of these shows, we would encourage all televised medical shows to publish transcripts and make them readily available.
As one last note, we hope others can soon move on from this debate. Most of the critics of our paper have raised similar issues and we have addressed them. We recognize that some people will still be dissatisfied but further debate is unlikely to change that. Furthermore, several of the critics have communicated with at least one of the shows and failed to disclose this, leaving us uncertain if the motivation for correspondence stems from interest in the research or enthusiasm to defend the shows. These debates are time consuming and have little further scientific merit. We hope those keen on future letters will direct that effort to repeating and improving on our study.
Interestingly, some of the key findings of our study, such as the relative lack of information on the magnitude of the benefit, potential harms, and costs provided with the recommendations, have been completely ignored by our critics. We hope future health recommendations in all media avenues will consistently include the:
• specific details of the proposed benefit
• magnitude of the proposed benefit
• potential harms and cost(s) of the intervention
• evidence supporting the recommendation, and
• disclosure of potential conflicts of interest.
Only then will the viewers/readers (our patients) be able to decide whether they should consider adopting televised medical shows recommendations.
References
1. Ellis J, Mulligan I, Rowe J, Sackett DL. Inpatient general medicine is evidence based. A-Team, Nuffield Department of Clinical Medicine. Lancet. 1995;346(8972):407-10.
2. Hartling L, Fernandes RM, Seida J, Vandermeer B, Dryden DM. From the trenches: a cross-sectional study applying the GRADE tool in systematic reviews of healthcare interventions. PLoS One. 2012;7(4):e34697.
3. Hartling L, Hamm MP, Milne A, Vandermeer B, Santaguida PL, Ansari M, et al. Testing the risk of bias tool showed low reliability between individual reviewers and across consensus assessments of reviewer pairs. J Clin Epidemiol. 2013;66(9):973-81.

I have read with interest the article authored by C. Korownyk, et al, and appreciate their attempt to take on this controversial topic. I am Boarded in Diagnostic Radiology, Nuclear Medicine as well as Integrative Medicine and, am familiar with rigorous scientific inquiry as well as the challenges of publishing in the Alternative Medicine realm. I, therefore, have questions regarding to the data used for this investigation.

Without clear understanding of the exact topics and the specific questions posed to investigate each topic, it is impossible to determine whether the conclusions drawn are appropriate. More accurate interpretation of this article would by facilitated by the inclusion of a table listing the searched topics as well as the exact search terms used. Was this information available to the reviewers of this article, as I cannot comprehend how a decision to publish could be made without access to the raw data.

Following is Part 3 of my critique of Korownyk et al:

Examination and search of the evidence: The authors provide few details on their search methods. They say little about precisely how they converted statements made on air into “searchable questions,” saying only “we developed a search question.” There are grounds to question the validity of their choices. For example, the supplement mentions that a statement—“you want your CRP numbers to be less than 1.0 mg/dL”—implies that “you have to change your CRP,” arguing that this justified a literature search for evidence of benefit from lowering CRP levels. It could be argued, however, that the statement is actually a description of the normal range of CRP, and the normal range derives not from intervention studies but from observational data on population risks associated with values outside the normal distribution (e.g., the 95% confidence interval). Normal distribution data are the basis for the Recommended Daily Allowances (RDA) for nutrients on every food package in the United States, along with the normal ranges for electrolytes, blood counts, urine sediment, and any other reference range on laboratory reports; on all of these, “you want your numbers to be less than” the elevated values associated with increased risk. The searchable question for such statements ought to be “is there evidence that values above the threshold are associated with significantly increased risk of disease?”

I have obtained the author’s spreadsheet, which raises further doubts about the formation of the search questions and an apparent tendency to create straw men by setting more narrow boundaries for the search than the reported statement covered. Consider the first 10 statements in their list of 80 recommendations, in which I found 5 instances in which the question was narrowly reframed:

• Reported broadcast statement: “Give yourself permission to have your [emotional] pain, not to be afraid of it.” The research question should be a variant of “Does acceptance of emotional pain improve wellbeing?” The authors constructed a narrower question: “Does allowing yourself to grieve prevent physical pain?”

• Reported broadcast statement: “Start your day green: wheat grass, vegetables etc...” The research question should address the spectrum of outcomes associated with fresh produce intake. The authors framed a narrower question: “Does eating green food oxygenate your body/blood?”

• Reported broadcast statement: “Look for the sneaky words on the menu. Want to avoid - cheesy, creamy, buttery. But look out for Homemade, legendary, luscious, mama's.” The research question should be whether foods with these labels contain higher levels of unhealthy nutrients. The authors’ reframed the question as if the statement alluded to weight gain: “Does food in restaurants with the word homemade, legendary, luscious, or mama's cause weight gain?”

• The broadcast statement reportedly outlined six symptoms of toxocariasis (abdominal pain, headache, coughing bouts, SOB, wheezing, some chest pain). The authors narrowed the question: “Is abdominal pain a symptom of toxocariasis infection?”

• Reported broadcast statement: “Don't start with laxatives, they destroy you inside, hair, teeth, family and home.” The authors narrowed the question to: “Does chronic laxative use negatively impact the GI tract?”

The authors looked only at bibliographic sources to find corroborating evidence but seem to have ignored evidence-based practice guidelines and evidence-based policies issued by government panels and agencies. A U.S. physician who advises women to get a mammogram—as does Dr. Oz and thousands of other physicians—is often basing this advice on the guidance of panels like the U.S. Preventive Services Task Force or the American Cancer Society. Turning to PubMed for the answer buries the physician in a literature beset with controversy and debate about the dozens of RCTs and observational studies of mammography. By ignoring this larger literature, the authors seem to chide a physician for reflecting the recommendations of panels that have studied this evidence and issued national guidelines rather than limiting their advice to what they can find in individual papers.

Even for the latter, the authors are vague in defining what they mean by “believable” evidence, other than some generalities: “the quality, quantity, and type of evidence available.” Without supplying metrics for defining what constitutes good quality, adequate quantity, etc., one has to assume the judgment was highly subjective and idiosyncratic. A voting process is mentioned, but the details of that voting process were not provided.

The supplement says decisions were made about the “ideal study design” for each question but offers no details on how that choice was made (or which study design was chosen for each of the 80 statements). The supplement alludes parenthetically to “cohort or RCT,” but more classes of evidence are pertinent to the breadth of remarks uttered in a broadcast. For example, neither a cohort study nor a RCT is the ideal standard for test performance characteristics (e.g., sensitivity or specificity). The same is true for the nutrient content of foods, a topic covered, for example, in a broadcast statement about the glucose content of ripe bananas, information intended for patients with diabetes. Population studies are an inappropriate source on this question, yet the authors sought prospective studies of “whether ripe bananas affect blood glucose or insulin levels.”

In conclusion, the weaknesses in this study raise serious questions about the validity of the percentages that drew attention in the abstract and that went viral on social media. It would do well for the BMJ do be transparent about the questions surrounding the accuracy of these results and to move the direction of future research on this topic away from ad homonym frameworks. The journal should encourage a more systematic examination of this medium by investigators who understand the topic and are willing to hold their own research to a modicum of the evidence standards they seem willing to apply to the work of others.

Sincerely,

Steven H. Woolf, MD, MPH
Professor, Department of Family Medicine and Population Health
Virginia Commonwealth University