Re: Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study
Dear Editor
Moustgard et al (1) made a very valuable scientific contribution by explaining in great detail what was investigated and how. We would like to explain hereafter, why we consider the results of the study to be untenable but the conclusion, in contrary, to be valid. We find the results of the study to be untenable, because using existing data from highly selected meta-analyses is neither a) helpful, nor b) transparent or c) valid.
a) It would be helpful to quantify the effect of blinding under Experimental Study Conditions (ESC), if the delivery of care to those groups of patients was blinded under Real World Conditions (RWC), too. As this is not possible, the gain of knowledge by estimating the effect of blinding in experimental studies is limited. Rather, one should strive to test the specificity of interventions under controlled everyday conditions, i.e. in Pragmatic Controlled Trials (PCTs) (2-4).
b) We believe the study (1) to be intransparent, because every author decides on the basis of her or his beliefs who should be blinded and whether at all. This individual author’s choice depends on various factors and is therefore seen as reasonable by the Cochrane Bias Methods Group / Cochrane Statistical Methods Group (5). However, reasons should be given for the decision. The selection of so-called 'informative meta-analyses‘ is not explained in the study presented (1). It remains unclear, why Moustgaard et al. exclusively chose meta-analyses, which included blinded as well as non-blinded studies. Every author of a meta-analysis will decide following her or his convictions, whether blinding is to be considered a confounder. Consequently, authors who believe blinding to be a relevant confounder will include either only blinded or non-blinded studies in their meta-analysis. If a meta-analysis therefore includes blinded as well as non-blinded studies - as the ones discussed in the paper - it can be presumed that the authors did not consider blinding as a confounding factor. This means that the meta-analysis of Moustgaard et al., which was undertaken to quantify the effect of blinding, includes only selected meta-analyses in which blinding had not been considered a confounding factor. This selection substantially distorts the results.
c) The precision of the study presented is insufficient, as no reasons are given for (or against) the blinding of individual partners (patients, doctors, data analysts). The methods section (1) states only that studies were compared, if the same partners had been blinded. The decision to blind or not to blind depended on further characteristics of the individual studies (5) and these were evidently not accounted for by Moustgaard er al. (1). This undifferentiated selection further distorts the results.
In summary, the authors of the discussed study conclude blinding to have no noteworthy effect on the outcome of studies. This conclusion is untenable for the above detailed reasons. However, this statement may be disregarded because, as Altman and Bland remarked "Absence of evidence is not evidence of absence" (6). The interesting and important aspects of this paper require a more detailed discussion.
This study confirms the need to strictly differentiate between ESCs and RWCs. Blinding can be carried out in an experimental study, e.g. in an RCT. It is required there to exclude non-specific effects, e.g. placebo effects. In the care of patients under RWC, both the patients and the study conditions (randomization and blinding) are different from those under ESCs. Therefore, it cannot be assumed that the unspecific effects of treatment, which can be measured under ESCs by blinding, are consistent with the non-specific effects under RWC. According to current understanding, the non-specific effects under RWC can only be detected in a PCT (2 -4). A PCT compares “innovative” treatments with pooled “control treatments”. The presumed superiority of an "innovative" therapy over a pool of already available therapies (all with similar mostly "non-specific" effects) can be confirmed in a PCT that compares the effectiveness of therapies in target groups with corresponding risk profiles. The risk profiles of all subgroups are specific for each of the investigated endpoints (e.g. death or side effect or cost).
The risk classification shall be carried out separately for each of the assessed endpoints to ensure that only patients with corresponding risk profiles related to the assessed endpoint are compared in all groups. A formal distinction between specific and non-specific effects is not possible under RWC; but specific effects can be assumed if one or a few therapies in which superiority over pooled control group was suspected can be confirmed by comparison in a controlled observational study, a PCT.
References
1. Moustgaard H, Clayton GL, Jones HE, Boutron I, Jørgensen L, Laursen DLT, Olsen MF, Paludan- Müller A, Ravaud P, Savović J, Sterne JAC, Higgins JPT, Hróbjartsson A. Impact of blinding on es-timated treatment effects in randomised clinical trials: meta-epidemiological study. BMJ. 2020;368:l6802. doi:10.1136/bmj.l6802.
2. Porzsolt F, Eisemann M, Habs M, Wyer P. Form Follows Function: Pragmatic Controlled Trials (PCTs) have to answer different questions and require different designs than Randomized Controlled Trials (RCTs). J Publ Health 2013;21:307-313. DOI 10.1007/s10389. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3655212/
3. Porzsolt F, Rocha NG, Toledo-Arruda AC, Thomaz TG, Moraes C, Bessa-Guerra TR, Leão M, Migowski A, Araujo de Silva AR, Weiss C. Efficacy and Effectiveness Trials Have Different Goals, Use Different Tools, and Generate Different Messages. Pragmatic and Observational Research 2015;6:47-54. DOI http://dx.doi.org/10.2147/POR.S89946
4. Porzsolt F, Weiss Ch, Weiss M, Müller AG, Becker SI, Eisemann M, Kaplan RM. Versorgungsforschung braucht dreidimensionale Standards zur Beschreibung von Gesundheitsleistungen – Teil 2. Monitor Versorgungsforschung 2019;4:53-60. http://doi.org/10.24945/MVF.04.19.1866-0533.2163
5. Higgins JPT, Altman DG, Gøtzsche PC, et al, Cochrane Bias Methods Group, Cochrane Statistical Methods Group. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. doi:10.1136/bmj.d5928
6. Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995;311:485. PubMed PMID: 7647644; PubMed Central PMCID: PMC2550545.
Rapid Response:
Re: Impact of blinding on estimated treatment effects in randomised clinical trials: meta-epidemiological study
Dear Editor
Moustgard et al (1) made a very valuable scientific contribution by explaining in great detail what was investigated and how. We would like to explain hereafter, why we consider the results of the study to be untenable but the conclusion, in contrary, to be valid. We find the results of the study to be untenable, because using existing data from highly selected meta-analyses is neither a) helpful, nor b) transparent or c) valid.
a) It would be helpful to quantify the effect of blinding under Experimental Study Conditions (ESC), if the delivery of care to those groups of patients was blinded under Real World Conditions (RWC), too. As this is not possible, the gain of knowledge by estimating the effect of blinding in experimental studies is limited. Rather, one should strive to test the specificity of interventions under controlled everyday conditions, i.e. in Pragmatic Controlled Trials (PCTs) (2-4).
b) We believe the study (1) to be intransparent, because every author decides on the basis of her or his beliefs who should be blinded and whether at all. This individual author’s choice depends on various factors and is therefore seen as reasonable by the Cochrane Bias Methods Group / Cochrane Statistical Methods Group (5). However, reasons should be given for the decision. The selection of so-called 'informative meta-analyses‘ is not explained in the study presented (1). It remains unclear, why Moustgaard et al. exclusively chose meta-analyses, which included blinded as well as non-blinded studies. Every author of a meta-analysis will decide following her or his convictions, whether blinding is to be considered a confounder. Consequently, authors who believe blinding to be a relevant confounder will include either only blinded or non-blinded studies in their meta-analysis. If a meta-analysis therefore includes blinded as well as non-blinded studies - as the ones discussed in the paper - it can be presumed that the authors did not consider blinding as a confounding factor. This means that the meta-analysis of Moustgaard et al., which was undertaken to quantify the effect of blinding, includes only selected meta-analyses in which blinding had not been considered a confounding factor. This selection substantially distorts the results.
c) The precision of the study presented is insufficient, as no reasons are given for (or against) the blinding of individual partners (patients, doctors, data analysts). The methods section (1) states only that studies were compared, if the same partners had been blinded. The decision to blind or not to blind depended on further characteristics of the individual studies (5) and these were evidently not accounted for by Moustgaard er al. (1). This undifferentiated selection further distorts the results.
In summary, the authors of the discussed study conclude blinding to have no noteworthy effect on the outcome of studies. This conclusion is untenable for the above detailed reasons. However, this statement may be disregarded because, as Altman and Bland remarked "Absence of evidence is not evidence of absence" (6). The interesting and important aspects of this paper require a more detailed discussion.
This study confirms the need to strictly differentiate between ESCs and RWCs. Blinding can be carried out in an experimental study, e.g. in an RCT. It is required there to exclude non-specific effects, e.g. placebo effects. In the care of patients under RWC, both the patients and the study conditions (randomization and blinding) are different from those under ESCs. Therefore, it cannot be assumed that the unspecific effects of treatment, which can be measured under ESCs by blinding, are consistent with the non-specific effects under RWC. According to current understanding, the non-specific effects under RWC can only be detected in a PCT (2 -4). A PCT compares “innovative” treatments with pooled “control treatments”. The presumed superiority of an "innovative" therapy over a pool of already available therapies (all with similar mostly "non-specific" effects) can be confirmed in a PCT that compares the effectiveness of therapies in target groups with corresponding risk profiles. The risk profiles of all subgroups are specific for each of the investigated endpoints (e.g. death or side effect or cost).
The risk classification shall be carried out separately for each of the assessed endpoints to ensure that only patients with corresponding risk profiles related to the assessed endpoint are compared in all groups. A formal distinction between specific and non-specific effects is not possible under RWC; but specific effects can be assumed if one or a few therapies in which superiority over pooled control group was suspected can be confirmed by comparison in a controlled observational study, a PCT.
References
1. Moustgaard H, Clayton GL, Jones HE, Boutron I, Jørgensen L, Laursen DLT, Olsen MF, Paludan- Müller A, Ravaud P, Savović J, Sterne JAC, Higgins JPT, Hróbjartsson A. Impact of blinding on es-timated treatment effects in randomised clinical trials: meta-epidemiological study. BMJ. 2020;368:l6802. doi:10.1136/bmj.l6802.
2. Porzsolt F, Eisemann M, Habs M, Wyer P. Form Follows Function: Pragmatic Controlled Trials (PCTs) have to answer different questions and require different designs than Randomized Controlled Trials (RCTs). J Publ Health 2013;21:307-313. DOI 10.1007/s10389. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3655212/
3. Porzsolt F, Rocha NG, Toledo-Arruda AC, Thomaz TG, Moraes C, Bessa-Guerra TR, Leão M, Migowski A, Araujo de Silva AR, Weiss C. Efficacy and Effectiveness Trials Have Different Goals, Use Different Tools, and Generate Different Messages. Pragmatic and Observational Research 2015;6:47-54. DOI http://dx.doi.org/10.2147/POR.S89946
4. Porzsolt F, Weiss Ch, Weiss M, Müller AG, Becker SI, Eisemann M, Kaplan RM. Versorgungsforschung braucht dreidimensionale Standards zur Beschreibung von Gesundheitsleistungen – Teil 2. Monitor Versorgungsforschung 2019;4:53-60. http://doi.org/10.24945/MVF.04.19.1866-0533.2163
5. Higgins JPT, Altman DG, Gøtzsche PC, et al, Cochrane Bias Methods Group, Cochrane Statistical Methods Group. The Cochrane Collaboration’s tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. doi:10.1136/bmj.d5928
6. Altman DG, Bland JM. Absence of evidence is not evidence of absence. BMJ. 1995;311:485. PubMed PMID: 7647644; PubMed Central PMCID: PMC2550545.
Competing interests: No competing interests