Re: Forty years of sports performance research and little insight gained
Author comment in reply to responses and direct communications to analysis piece: Forty years of sports performance research and little insight gained
(BMJ 2012; 345 doi: 10.1136/bmj.e4797 (Published 19 July 2012) Cite this as: BMJ 2012;345:e4797
In our analysis of forty years of sports performance research and little insight gained we state in the methods ‘We did not review posters, supplements, theses, or unavailable articles.’ We have corrected instances in the data supplement, supplied by GSK, where author or email communication has amended papers inclusion, required clarification of methods or the quality assessment.
Changes to the data supplement (accompanying this comment) are highlighted in bold in the accompanying pdf which include:
• Removal of one paper from the supplement [point 26]
• Changed seven studies from high risk of bias to moderate risk of bias [points 1, 2, 3, 4, 5, 17, 27]
• Marked one paper as duplicate [point 16]
• work never published to supplement [point 28]
Of this list of 176 studies, with the removal of the one not relevant paper, the number of studies should now read as 175 and of the studies we were able to critically review 106 studies (101 clinical trials) dating from 1971 through to 2012, should read as 105 studies (100 clinical trials).
In terms of poorly designed research we stated: ‘Most studies (76%) were low in quality because of a lack of allocation concealment and blinding, and often the findings contrasted with each other.’ With the change of seven studies from low to moderate, and the removal of one moderate study, the proportion of low quality studies is 70%. However, in terms of our assessment we have been generous, particularly with regard to the lack of detail in the published papers. Readers should remain sceptical of work where methods such as randomized or blinded are reported without accompanying details. Where a lack of detailed reporting occurs the Cochrane risk of bias tool judges the risk of bias to be uncertain, particularly when information is inadequate.
Therefore, no studies were judged to be of
high quality.
The lack of allocation concealment remains a considerable issue: methods of randomization were not routinely reported across the 100 clinical studies. Proper randomisation relies upon adequate allocation concealment. Such a sequence keeps the investigator and the participants unaware of the intervention assignments: ‘Inadequate allocation concealment leads to exaggerated estimates of treatment effect, on average, but with scope for bias in either direction.’ [Lancet. 2002 Feb 16;359(9306):614-8.]
Methods such as systematic rotation reported in points 8, 9, 10 and 11 are not deemed randomized.
In terms of blinding, methods of who were exactly blinded were not routinely reported: Schulz and Grimes state the importance of blinding and its detailed reporting in papers:
‘Many investigators and readers naïvely consider a randomised trial as high quality simply because it is double blind, as if double-blinding is the sine qua non of a randomised controlled trial. Although double blinding (blinding investigators, participants, and outcome assessors) indicates a strong design, trials that are not double blinded should not automatically be deemed inferior. Rather than solely relying on terminology like double blinding, researchers should explicitly state who was blinded, and how.’ [Lancet. 2002 Feb 23;359(9307):696-700]
Given, many articles in the supplement report blinding without details readers should remain vigilant about the quality. Where papers only reported single blinded participants [point 12] this was judged as lack of adequate blinding.
The sequence for such studies is to report whether the investigator delivering the intervention are blinded to the allocation (including the method of randomization), whether the participants were blinded (where possible) and crucially whether the outcome assessors were blinded when performing the data analysis.
Where thesis, posters, supplements, reviews were included in the list of studies [points 18,19,20,21,24 and 25] we report in the paper and the supplement we did not review their quality.
Changes and responses to the data supplement and comments are outlined in Box 1. We continue to welcome dialogue over the supplement, the inclusion of papers and clarification of methods for these papers, particularly for those we were unable to review due to access or lack of publication.
see attached pdf for full comments and updated supplement
Competing interests:
No competing interests
04 September 2012
Carl Heneghan
Clinical Reader
Peter GIll, David Nunan, Kamal Mahtani, Rafael Perera
Rapid Response:
Re: Forty years of sports performance research and little insight gained
Author comment in reply to responses and direct communications to analysis piece: Forty years of sports performance research and little insight gained
(BMJ 2012; 345 doi: 10.1136/bmj.e4797 (Published 19 July 2012) Cite this as: BMJ 2012;345:e4797
In our analysis of forty years of sports performance research and little insight gained we state in the methods ‘We did not review posters, supplements, theses, or unavailable articles.’ We have corrected instances in the data supplement, supplied by GSK, where author or email communication has amended papers inclusion, required clarification of methods or the quality assessment.
Changes to the data supplement (accompanying this comment) are highlighted in bold in the accompanying pdf which include:
• Removal of one paper from the supplement [point 26]
• Changed seven studies from high risk of bias to moderate risk of bias [points 1, 2, 3, 4, 5, 17, 27]
• Marked one paper as duplicate [point 16]
• work never published to supplement [point 28]
Of this list of 176 studies, with the removal of the one not relevant paper, the number of studies should now read as 175 and of the studies we were able to critically review 106 studies (101 clinical trials) dating from 1971 through to 2012, should read as 105 studies (100 clinical trials).
In terms of poorly designed research we stated: ‘Most studies (76%) were low in quality because of a lack of allocation concealment and blinding, and often the findings contrasted with each other.’ With the change of seven studies from low to moderate, and the removal of one moderate study, the proportion of low quality studies is 70%. However, in terms of our assessment we have been generous, particularly with regard to the lack of detail in the published papers. Readers should remain sceptical of work where methods such as randomized or blinded are reported without accompanying details. Where a lack of detailed reporting occurs the Cochrane risk of bias tool judges the risk of bias to be uncertain, particularly when information is inadequate.
Therefore, no studies were judged to be of
high quality.
The lack of allocation concealment remains a considerable issue: methods of randomization were not routinely reported across the 100 clinical studies. Proper randomisation relies upon adequate allocation concealment. Such a sequence keeps the investigator and the participants unaware of the intervention assignments: ‘Inadequate allocation concealment leads to exaggerated estimates of treatment effect, on average, but with scope for bias in either direction.’ [Lancet. 2002 Feb 16;359(9306):614-8.]
Methods such as systematic rotation reported in points 8, 9, 10 and 11 are not deemed randomized.
In terms of blinding, methods of who were exactly blinded were not routinely reported: Schulz and Grimes state the importance of blinding and its detailed reporting in papers:
‘Many investigators and readers naïvely consider a randomised trial as high quality simply because it is double blind, as if double-blinding is the sine qua non of a randomised controlled trial. Although double blinding (blinding investigators, participants, and outcome assessors) indicates a strong design, trials that are not double blinded should not automatically be deemed inferior. Rather than solely relying on terminology like double blinding, researchers should explicitly state who was blinded, and how.’ [Lancet. 2002 Feb 23;359(9307):696-700]
Given, many articles in the supplement report blinding without details readers should remain vigilant about the quality. Where papers only reported single blinded participants [point 12] this was judged as lack of adequate blinding.
The sequence for such studies is to report whether the investigator delivering the intervention are blinded to the allocation (including the method of randomization), whether the participants were blinded (where possible) and crucially whether the outcome assessors were blinded when performing the data analysis.
Where thesis, posters, supplements, reviews were included in the list of studies [points 18,19,20,21,24 and 25] we report in the paper and the supplement we did not review their quality.
Changes and responses to the data supplement and comments are outlined in Box 1. We continue to welcome dialogue over the supplement, the inclusion of papers and clarification of methods for these papers, particularly for those we were unable to review due to access or lack of publication.
see attached pdf for full comments and updated supplement
Competing interests: No competing interests