Letters

Evaluating guidelines across primary care-secondary care interface

BMJ 2001; 323 doi: https://doi.org/10.1136/bmj.323.7316.808 (Published 06 October 2001) Cite this as: BMJ 2001;323:808

Methodological weaknesses and poor reporting undermine authors' conclusions

  1. Nick Freemantle, professor (N.Freemantle{at}bham.ac.uk),
  2. John Wood, principal statistician
  1. Department of Primary Care and General Practice, University of Birmingham, Birmingham B15 2TT
  2. GlaxoSmithKline, Harlow CM19 5AW
  3. Department of General Practice, University of Glasgow, Glasgow G12 0RR
  4. MRC Social and Public Health Sciences Unit, University of Glasgow, Glasgow G12 8RZ
  5. Health Services Research Unit, University of Aberdeen, Aberdeen AB25 2ZD

    EDITOR—Readers' ability to extract information from Morrison et al's study is impeded by errors in analysis and reporting.1 The trial evaluated clinical guidelines on the management of subfertility for doctors in general practice. The hope was that consequent improvements to procedure in primary care would affect processes in secondary care after referral, with more efficient management overall.

    The subjects of the study were healthcare professionals, and the focus (the object of the intervention and the outcomes measured) was on their behaviour. All the results, however, are given as aggregated patient referrals, which does not respect the structure of the experiment.

    Although one can compare case mix through aggregated patient characteristics (table 1), it is wrong to assess and report the effects of the intervention on this basis.2 The authors adjust for within practice correlation in their analysis, but at a level that is inappropriate. Two important consequences are the absence of information on variability between clinicians and inappropriate inflation of the sample size. Indeed, at a recent Medical Research Council meeting on cluster randomised trials it was acknowledged that using patients as the unit of analysis for trials in which the healthcare provider formed the natural subject is always incorrect.

    Other concerns centre on the significant difference between the guideline and non-guideline groups in the number of appropriate investigations carried out in general practice, flagged as a “positive result” for the trial. Firstly, some confusion surrounds this statement. The paper refers to a “significantly higher mean number,” with means given in table 2. Alongside the means, however, is an odds ratio with confidence interval and significance (P=0.05). It is these latter figures that are given in the abstract, described as the binary outcome “all relevant investigations carried out,” though now with P=0.025 (presumably one sided).

    Secondly, and fundamentally, this significant result stands in isolation among the outcomes measured. Naturally, it is efficient to measure several outcomes in a trial, summarising these with confidence intervals. However, if we ask essentially the same yes/no question several times in significance tests (here, whether guidelines modified doctors' behaviour) and get a single “yes” in reply, then multiplicity matters. The direct way through this is to pre-specify a single primary outcome to identify whether the trial is positive.3 As no such outcome is defined in the paper, the correct interpretation is that the trial is neutral. No post-hoc action can turn around this conclusion.

    We frequently have just one opportunity to assess important questions of healthcare delivery in randomised controlled trials. These opportunities are wasted when studies are poorly designed and reported.

    References

    1. 1.
    2. 2.
    3. 3.

    Authors' reply

    1. Jill Morrison, professor (jmm4y{at}clinmed.gla.ac.uk),
    2. Alastair Leyland, research scientist,
    3. Jeremy Grimshaw, professor
    1. Department of Primary Care and General Practice, University of Birmingham, Birmingham B15 2TT
    2. GlaxoSmithKline, Harlow CM19 5AW
    3. Department of General Practice, University of Glasgow, Glasgow G12 0RR
    4. MRC Social and Public Health Sciences Unit, University of Glasgow, Glasgow G12 8RZ
    5. Health Services Research Unit, University of Aberdeen, Aberdeen AB25 2ZD

      EDITOR—These authors believe that there are errors in the analysis and reporting of our study. We agree that patients should not be used as the unit of analysis for trials in which the healthcare provider forms the natural subject. For this reason our data were analysed as multilevel models with the multilevel modelling software MlwiN.1 As Freemantle and Wood have themselves pointed out, “multilevel models using appropriate statistical techniques can have some advantages over simple analyses at the appropriate unit of analysis.”2 Information about our use of multilevel modelling was given in the fuller version of our paper on the BMJ website but was not included in the shorter version in the paper journal.

      In our view, it is inappropriate of these authors to cite anecdotal evidence from a meeting if that evidence is not currently available in any published form to other researchers. Readers are unable to assess the strength or weight of the assertion. The authors are correct to point out the error in the abstract. The P value of 0.05 reported in table 1 is the correct one, and we apologise for missing the error in the abstract when proofreading.

      The message as to whether the guidelines modify doctors' behaviour is summed up precisely in table 1 and the text of the paper. The average number of investigations was significantly higher in intervention than control practices. Each of the five investigations was carried out more often in the intervention practices than the control practices, although the differences were not significant at the 5% level. It would take some sleight of hand before such results could be considered to point to the conclusion that the trial was neutral. Although we showed some modest change in doctors' behaviour, however, we were cautious about claiming that the overall intervention was worthwhile because we did not see the anticipated benefits in secondary care.

      References

      1. 1.
      2. 2.
      View Abstract

      Sign in

      Log in through your institution

      Free trial

      Register for a free trial to thebmj.com to receive unlimited access to all content on thebmj.com for 14 days.
      Sign up for a free trial

      Subscribe