Beware of look-alikes to randomisation. There is no evidence of
proper randomisation and no evidence of real time follow-up. If there
were not too many fatal flaws in this paper one would be tempted to
conclude that it shows the lack of superior effectiveness of specialised
care compared to standard care that is properly funded. As it werethe
jury is still out and standard care may turn out to be superior if we pose
a two-way null-hypothesis and keep an open mind.
1, The objective did not appear clear ab initio. It does not specify
whether the study is examining the effectiveness in reducing relapse rate,
readmission rate or maintaining contact with patients comopared with
standard care. This is not pedantic because this focus is required for
power calculation. Is this a case of casting your net wide and see what
turns up?
2, Is this a truly randomised controlled trial? Even after reading
the full text on the internet (after a lively debate in our journal club)
I remain confused as to how randomisation was performed. How were the
random numbers generated? Were the subjects randomised as they were
indentified or at what stage were they randomised? What was the accrual
rate over the 18months and did this necessitate block randomisation? The
study was reported as though 140 subjects were available for randomisation
on day 1. The reporting of the randomisation does not meet the CONSORT
standard (altman 1996). DerSimonian et al, (1982) asserted that an
author's assurance that randomisation was performed is not convincing
unless the method used is fully discussed.
3, Was the follow up period real time and if so why was it necessary
to search the case notes for important data on relapse? In a proper
randomised controlled trial, there is no need to rely "on record systems
for data on relapse which are susceptible to errors and biases", the
errors and biases that RCT are designed to avert.
4, In a real time experiment of this kind, how can subjects be seen
"as soon as feasible after randomisation and at follow up 18months later"?
The report suggests that data on relapse and readmission were collected
retrospectively (after the have occured).
5, Readers are confused about the nature of the specialised care of
which we know nothing except the professionals involved and the label of
assertive outreach model. What is standard care and how is this different
from specialised care apart from the fact that the latter makes more
contact with patients? It is therefore difficult to replicate the study.
6, In a two-phase study of this kind involving screening, the
parameters of the screening instrument are important. What is the
sensitivity/specificty/positive predictive value of the screening
instrument.
7, One of the most glaring shortcomings of the study is the assertion
that 120 patients will be required (suggesting power calculation); 140
were recruited and data were collected from 136 and yet the study is
"underpowered"! How can a study that recruits more subjects than required
for the calculated power be underpowered. This begs the question of
whether power calculation was performed. Assuming it was what primary
objective was used? what difference was sought and what false positive
level was chosen?
8, In an important study of this kind it is not necessary to bemoan
the lack of power. The question should have been asked before embarking
on the study as to whether more than 140 subjects can be recruited in
Lambeth over 18months. If not a longer period of recruitment should have
been considered. Using insufficient sample size in an experiment of this
kind is an ethical issue.
9, For a study of this kind, the lack of operational definition of
relapse is worrying. Would a patient who complained of hearing voices on
two occasions during the follow up period be classified as relapsed? So
what is relapse in the context of this study?
10, The reporting of the study demontrates Rosenthal effect, the
phenomenon that investigators almost always find what they expect to find.
How else can we justify the conclusion that those in specialised care (as
expected) "were less likely to relapse...." although the odds of doing so
of 0.46 to 1 is not statistically significant after adjusting for possible
confounders and despite the admitted susceptibility of the study to
"errors and biases".
11 Finally on the issue of health economcs, I wonder why the cost
effectiveness was not discussed. Does that suggest that standard care
could be more cost effective?
Rapid Response:
Is this a randomised controlled trial?
Beware of look-alikes to randomisation. There is no evidence of
proper randomisation and no evidence of real time follow-up. If there
were not too many fatal flaws in this paper one would be tempted to
conclude that it shows the lack of superior effectiveness of specialised
care compared to standard care that is properly funded. As it werethe
jury is still out and standard care may turn out to be superior if we pose
a two-way null-hypothesis and keep an open mind.
1, The objective did not appear clear ab initio. It does not specify
whether the study is examining the effectiveness in reducing relapse rate,
readmission rate or maintaining contact with patients comopared with
standard care. This is not pedantic because this focus is required for
power calculation. Is this a case of casting your net wide and see what
turns up?
2, Is this a truly randomised controlled trial? Even after reading
the full text on the internet (after a lively debate in our journal club)
I remain confused as to how randomisation was performed. How were the
random numbers generated? Were the subjects randomised as they were
indentified or at what stage were they randomised? What was the accrual
rate over the 18months and did this necessitate block randomisation? The
study was reported as though 140 subjects were available for randomisation
on day 1. The reporting of the randomisation does not meet the CONSORT
standard (altman 1996). DerSimonian et al, (1982) asserted that an
author's assurance that randomisation was performed is not convincing
unless the method used is fully discussed.
3, Was the follow up period real time and if so why was it necessary
to search the case notes for important data on relapse? In a proper
randomised controlled trial, there is no need to rely "on record systems
for data on relapse which are susceptible to errors and biases", the
errors and biases that RCT are designed to avert.
4, In a real time experiment of this kind, how can subjects be seen
"as soon as feasible after randomisation and at follow up 18months later"?
The report suggests that data on relapse and readmission were collected
retrospectively (after the have occured).
5, Readers are confused about the nature of the specialised care of
which we know nothing except the professionals involved and the label of
assertive outreach model. What is standard care and how is this different
from specialised care apart from the fact that the latter makes more
contact with patients? It is therefore difficult to replicate the study.
6, In a two-phase study of this kind involving screening, the
parameters of the screening instrument are important. What is the
sensitivity/specificty/positive predictive value of the screening
instrument.
7, One of the most glaring shortcomings of the study is the assertion
that 120 patients will be required (suggesting power calculation); 140
were recruited and data were collected from 136 and yet the study is
"underpowered"! How can a study that recruits more subjects than required
for the calculated power be underpowered. This begs the question of
whether power calculation was performed. Assuming it was what primary
objective was used? what difference was sought and what false positive
level was chosen?
8, In an important study of this kind it is not necessary to bemoan
the lack of power. The question should have been asked before embarking
on the study as to whether more than 140 subjects can be recruited in
Lambeth over 18months. If not a longer period of recruitment should have
been considered. Using insufficient sample size in an experiment of this
kind is an ethical issue.
9, For a study of this kind, the lack of operational definition of
relapse is worrying. Would a patient who complained of hearing voices on
two occasions during the follow up period be classified as relapsed? So
what is relapse in the context of this study?
10, The reporting of the study demontrates Rosenthal effect, the
phenomenon that investigators almost always find what they expect to find.
How else can we justify the conclusion that those in specialised care (as
expected) "were less likely to relapse...." although the odds of doing so
of 0.46 to 1 is not statistically significant after adjusting for possible
confounders and despite the admitted susceptibility of the study to
"errors and biases".
11 Finally on the issue of health economcs, I wonder why the cost
effectiveness was not discussed. Does that suggest that standard care
could be more cost effective?
Competing interests:
None declared
Competing interests: No competing interests