I have reposted it to this article by Heneghan and colleagues (discussed in the box "What the research found" in Cohen's feature) because it is more relevant to the full online text.
A shorter version of this rapid response was published as a letter in the print issue of 18 August 2012 http://www.bmj.com/content/345/bmj.e5425 using the online article by Heneghan and colleagues as reference 1 rather than the feature by Cohen.
Sharon Davies
Letters editor
8 October 2012
------
To the Editor,
I read with interest the recent BMJ series including the feature ‘Forty years of Sports Performance Research and Little Insight Gained’ and associated editorial ‘The Truth About Sports Drinks’. A number of my research papers and review articles were included in this analysis and I also contributed to the upheld EFSA claims substantiation for rehydration solutions. Based upon my experiences working with colleagues in industry and those eminent scientists to have dedicated a life’s work to advancing understanding in this field, I feel these recent articles provided a wholly unbalanced account both of the quality of science available and, moreover, of the integrity of those involved.
One of the features of the recent analysis was that so many highly regarded studies within the field were deemed to be of low quality and at high risk of bias. It is therefore essential to consider whether the criteria by which studies were evaluated was appropriate. The method employed to determine risk of bias is more commonly used to evaluate clinical trials and therefore places heavy emphasis on controlling for deciphering treatment/group allocation, blinding and thus avoidance of demand characteristics or placebo effects. Such issues are undoubtedly of primary importance in large clinical trials under free-living conditions but not always so relevant for acute laboratory controlled trials (where blinding alone is often sufficient and knowledge of treatment can be integral to study design). Even so, some of the studies included in this analysis have been labeled as ‘non-randomised trials’ or with a ‘lack of blinding’ when both randomisation and blinding were clearly reported in the relevant papers.
Another issue was the perplexing view that any study with fewer than one-hundred participants should be considered ‘small’. This statement was supported by reference to a clinical review that derived this number based on the specific variability associated with therapeutic interventions for osteoarthritic joint pain. Major journals in the exercise sciences call for sample size estimates to determine whether the outcomes specific to each study require tens, hundreds or thousands of participants (the authors state that only four studies they reviewed provided power calculations, yet others in their list have been overlooked). Given the tight laboratory controls applied in acute experiments, we tend to find that between ten and twenty participants in a cross-over design are adequate to detect effects (as evidenced by the numerous positive effects reported). It would therefore seem excessive (unethical?) to test one-hundred or more individuals for studies that typically involve intense physical exertion and multiple tissue samples when hypotheses can be supported or refuted with just ten or twenty.
There was also a mismatch in these articles between the critique of how scientific claims may be used in the media and the quality of the science itself. This appears to underpin the authors’ negative view of ‘surrogate outcomes’, which may not directly support some of the advertising claims but rarely is that the focus of a scientific research question. For example, the authors state “…we are not aware of any major sporting events that use time to exhaustion as the outcome”. Many studies are mechanistic in nature, so time to exhaustion may be preferred to enable steady-state measurements at fixed intensity and/or to better understand physiological responses at the point of fatigue. It is interesting that the authors chose on this occasion to base their argument on sporting events, when their broader critique had targeted the inappropriate marketing of these claims from athletes to the general public. The ability to exercise at a set pace without fatigue might be considered more relevant than time-trials for the wider-majority of individuals, whose exercise intensity is rarely maximal race-pace but rather dictated by the treadmill, running partners and/or team-mates. The military provides an even more poignant example of the value in understanding the ability of all members within a group to sustain a collective pace rather than for the group break ranks according to individual ability.
Headlines questioning the value of sports products in general are understandably newsworthy in the weeks before the Olympic games but such a broad conclusion is unlikely to apply equally to all supplements, outcomes or populations. A theme running throughout these recent reviews is that large scale trials using a heterogeneous cohort are necessarily superior to multiple smaller but more tightly controlled experiments using a clearly defined homogenous cohort. While the former approach is often necessary for testing the efficacy of drugs or other treatments to be prescribed on a population scale, the use of clearly defined study cohorts not only provides sensitivity to the specific responsiveness of different populations but also permits systematic manipulation of other variables between studies. This approach often reveals far more about the likely mechanisms of action and thus how to maximise the desired outcomes, culminating in more individualised prescriptions. The eventual message is not then so straightforward (‘sports supplements’ do or do not ‘work’) but requires more detailed ‘commentary’ reviews to independently evaluate studies on individual merit and communicate how each contributes to a more complex picture. There are excellent examples of such reviews throughout the sport and exercise science literature.
I fully recognise that the ability to openly express opinions contributes towards healthy scientific debate. However, if the entirety of a body of work is to be labeled of poor quality and at high risk of bias (especially if bringing into question the integrity of those involved) it is essential that the underpinning evaluation is itself comprehensive, of high quality and balanced. In this case, the evidence provided certainly does not justify the strength with which such non-specific and unbalanced opinions were allowed to be communicated on this platform.
Competing interests:
A number of my published papers were funded by GlaxoSmithKline and/or included their products. I also produced a report subitted to EFSA on behalf of the British Specialist Nutrition Association to evaluate claims regarding oral rehydration solutions.
Rapid Response:
Re: Forty years of sports performance research and little insight gained
Editorial note
This rapid response http://www.bmj.com/content/345/bmj.e4737/rr/595874 was originally posted on 27 July 2012 in response to the feature by Cohen entitled "The truth about sports drinks" http://www.bmj.com/content/345/bmj.e4737
I have reposted it to this article by Heneghan and colleagues (discussed in the box "What the research found" in Cohen's feature) because it is more relevant to the full online text.
A shorter version of this rapid response was published as a letter in the print issue of 18 August 2012 http://www.bmj.com/content/345/bmj.e5425 using the online article by Heneghan and colleagues as reference 1 rather than the feature by Cohen.
Sharon Davies
Letters editor
8 October 2012
------
To the Editor,
I read with interest the recent BMJ series including the feature ‘Forty years of Sports Performance Research and Little Insight Gained’ and associated editorial ‘The Truth About Sports Drinks’. A number of my research papers and review articles were included in this analysis and I also contributed to the upheld EFSA claims substantiation for rehydration solutions. Based upon my experiences working with colleagues in industry and those eminent scientists to have dedicated a life’s work to advancing understanding in this field, I feel these recent articles provided a wholly unbalanced account both of the quality of science available and, moreover, of the integrity of those involved.
One of the features of the recent analysis was that so many highly regarded studies within the field were deemed to be of low quality and at high risk of bias. It is therefore essential to consider whether the criteria by which studies were evaluated was appropriate. The method employed to determine risk of bias is more commonly used to evaluate clinical trials and therefore places heavy emphasis on controlling for deciphering treatment/group allocation, blinding and thus avoidance of demand characteristics or placebo effects. Such issues are undoubtedly of primary importance in large clinical trials under free-living conditions but not always so relevant for acute laboratory controlled trials (where blinding alone is often sufficient and knowledge of treatment can be integral to study design). Even so, some of the studies included in this analysis have been labeled as ‘non-randomised trials’ or with a ‘lack of blinding’ when both randomisation and blinding were clearly reported in the relevant papers.
Another issue was the perplexing view that any study with fewer than one-hundred participants should be considered ‘small’. This statement was supported by reference to a clinical review that derived this number based on the specific variability associated with therapeutic interventions for osteoarthritic joint pain. Major journals in the exercise sciences call for sample size estimates to determine whether the outcomes specific to each study require tens, hundreds or thousands of participants (the authors state that only four studies they reviewed provided power calculations, yet others in their list have been overlooked). Given the tight laboratory controls applied in acute experiments, we tend to find that between ten and twenty participants in a cross-over design are adequate to detect effects (as evidenced by the numerous positive effects reported). It would therefore seem excessive (unethical?) to test one-hundred or more individuals for studies that typically involve intense physical exertion and multiple tissue samples when hypotheses can be supported or refuted with just ten or twenty.
There was also a mismatch in these articles between the critique of how scientific claims may be used in the media and the quality of the science itself. This appears to underpin the authors’ negative view of ‘surrogate outcomes’, which may not directly support some of the advertising claims but rarely is that the focus of a scientific research question. For example, the authors state “…we are not aware of any major sporting events that use time to exhaustion as the outcome”. Many studies are mechanistic in nature, so time to exhaustion may be preferred to enable steady-state measurements at fixed intensity and/or to better understand physiological responses at the point of fatigue. It is interesting that the authors chose on this occasion to base their argument on sporting events, when their broader critique had targeted the inappropriate marketing of these claims from athletes to the general public. The ability to exercise at a set pace without fatigue might be considered more relevant than time-trials for the wider-majority of individuals, whose exercise intensity is rarely maximal race-pace but rather dictated by the treadmill, running partners and/or team-mates. The military provides an even more poignant example of the value in understanding the ability of all members within a group to sustain a collective pace rather than for the group break ranks according to individual ability.
Headlines questioning the value of sports products in general are understandably newsworthy in the weeks before the Olympic games but such a broad conclusion is unlikely to apply equally to all supplements, outcomes or populations. A theme running throughout these recent reviews is that large scale trials using a heterogeneous cohort are necessarily superior to multiple smaller but more tightly controlled experiments using a clearly defined homogenous cohort. While the former approach is often necessary for testing the efficacy of drugs or other treatments to be prescribed on a population scale, the use of clearly defined study cohorts not only provides sensitivity to the specific responsiveness of different populations but also permits systematic manipulation of other variables between studies. This approach often reveals far more about the likely mechanisms of action and thus how to maximise the desired outcomes, culminating in more individualised prescriptions. The eventual message is not then so straightforward (‘sports supplements’ do or do not ‘work’) but requires more detailed ‘commentary’ reviews to independently evaluate studies on individual merit and communicate how each contributes to a more complex picture. There are excellent examples of such reviews throughout the sport and exercise science literature.
I fully recognise that the ability to openly express opinions contributes towards healthy scientific debate. However, if the entirety of a body of work is to be labeled of poor quality and at high risk of bias (especially if bringing into question the integrity of those involved) it is essential that the underpinning evaluation is itself comprehensive, of high quality and balanced. In this case, the evidence provided certainly does not justify the strength with which such non-specific and unbalanced opinions were allowed to be communicated on this platform.
Competing interests: A number of my published papers were funded by GlaxoSmithKline and/or included their products. I also produced a report subitted to EFSA on behalf of the British Specialist Nutrition Association to evaluate claims regarding oral rehydration solutions.