Evolution of poor reporting and inadequate methods over time in 20 920 randomised controlled trials included in Cochrane reviews: research on research study

We would like to respond to the issues raised by Prof Puljak:

1) Cochrane reviewers should be trained in risk of bias assessment, and we acknowledge that we cannot check that they actually are trained. However, they are more likely than non-Cochrane reviewers to be trained. Although lack of training can yield inconsistent ratings of risk of bias, this variability is decreased by assessment in duplicate. In addition, variability should be at random and there is no reason to think that this variability could result in a bias when assessing the evolution over time.

2) We only considered individual items and not overall assessment. We indicated in the methods that we extracted risk of bias for each item and that an item judged at unclear risk of bias can be considered poorly reported. We did not report “for each item” in each sentence of the Methods section to avoid repetition. However, our Results section (both text and figures) clearly show that results are presented for each item and not overall.

3) Thank you very much for highlighting this point. Yes, the Y axes should be “Proportion at high risk (%)” in Figure 5. An error was introduced during production and is being corrected.

4) In the second column of the graphs in Figure 5, we report results for each item of the evolution of the proportion of trials at high risk of bias after excluding trials at unclear risk of bias. Therefore, for this column, we only considered trials with an adequate reporting of the item. We did this because a trial can only be assessed at high risk of bias when reporting is adequate. The number of trials at unclear risk of bias is high and decreases over time for some items, which may induce confounding when assessing the evolution of the proportion of trials at high risk of bias over time.

5) As indicated in our methods, we selected only Cochrane reviews with an evaluation of the following items: sequence generation, allocation concealment, blinding (whatever the blinding domain) and incomplete outcome data. For blinding, because not all trials evaluated the same blinding items, the actual number of trials used slightly varied, which was not the case for sequence generation, allocation concealment and incomplete outcome data. Therefore, the actual number of trials used (as indicated in Figure 3 and Figure 5) is 20,920 for sequence generation, allocation concealment and incomplete outcome data; 19,794 for blinding of participants and personnel; and 20,712 for blinding of outcome assessors.

6) We acknowledge that there could be variability when assessing risk of bias because of different definitions used by Cochrane reviewers, particularly for incomplete outcome data because the guidance provided in the Cochrane handbook is rather vague as compared to other items such as sequence generation and allocation concealment. Nevertheless, such variability should be at random and there is no reason to think that this variability could bias our results especially those concerning the evolution over time. We discuss this point in the limitations section of our article in which we present some preliminary results regarding differences in risk of bias assessment for trials included in more than one systematic review. We are also conducting a study to explore sources of discordance in risk of bias assessment.