Statistics Notes: Diagnostic tests 3: receiver operating characteristic plotsBMJ 1994; 309 doi: https://doi.org/10.1136/bmj.309.6948.188 (Published 16 July 1994) Cite this as: BMJ 1994;309:188
All rapid responses
We (Maciej Gorkiewicz, Kam Cheong Wong, Paul CY Chen, Lai Kah Lee,
Jagmohni Kaur) have been collaborating to clarify the application of ROC
methodology to select target group for suicidal intervention in a
situation of limited resources. This 'response to BMJ' facility has
enabled the authors to collaborate and enhance the application of ROC
methodology. The constructive feedback has been considered and the
'avoidable misunderstanding' between authors is clarified.
Suicidal intervention may include interviewing all students in the
target group with the relevant Youth Risk Behavior Surveillance
questionnaire and provides psychological counselling. The following
content has been accepted for presentation at the 30th Annual Conference
of the German Classification Society at Berlin Germany on 8th -10th March
With reference to the Youth Risk Behavior Surveillance conducted in
the United States of America (USA), a cross sectional survey was carried
out on 4500 adolescent students from 14 schools in a state in Malaysia.
The details of the research were published in the Journal of Adolescent
Health (USA). Chen et al reported that the suicidal behaviour is
significant among Forms (or Grades as known in other country) in a school.
Within one Form in a school there could be more than one class. A Form in
a school can be regarded as a cluster because when intervention is to be
implemented, it is practical to intervene all the classes within the Form
in the school. There were a total of 73 clusters in the 14 schools.
Individual who has positive suicidal behavior is defined as 'positive
individual' (POS) who likely needs psychological support. The definition
of Positives (POS) is as follows:
POS: (Suicdone = 1) or [((thinksui = 1) or (suicplan = 1)) and (sad = 1)];
where Suicdone = 1 if suicide was carried out, thinksui = 1 if seriously
considered attempting suicide, suicplan = 1 if made a suicide plan, sad =
1 if felt sad or hopeless. Participants who did not meet the above
definition were assumed did not have suicidal behavior. Other factors
include gender; ethnic group (Malay, Chinese, Indian, others); district
(urban, semi urban, rural); alcohol (no, yes); and absenteeism (number of
days absent from school in the past 30 days).
All the 73 clusters were ranked according to their percentages of
positive suicidal behaviour. The cluster with the lowest %POS was ranked
1, and the cluster with the highest %POS was ranked 73. We can use the
rank of the cluster as a cut-off decision point to decide which clusters
are to be intervened. Every rank gives rise to different true positives
(TP) and false positive (FP). Thus, a Receiver Operating Characteristic
(ROC) curve can be plotted. This ROC is termed ‘ideal ROC’ because it is
established based on the % POS rank that is known after implementing the
USA Youth Risk Behaviour Surveillance Questionnaire.
On the other hand, a linear regression was performed on the dependent
variable (i.e. the rank) and independent variables (i.e. gender,
absenteeism, ethnicity, alcohol). The rank generated from the linear
regression is called ‘predicted rank’. Each ‘predicted rank’ gives rise
to different TP and FP with reference to the USA Youth Risk Behaviour
Surveillance Questionnaire. Thus, these ‘predicted ranks’ can be plotted
as another ROC curve which is called ‘practical ROC’. The ‘ideal ROC’ can
be established after intervention, but the ‘practical ROC’ can be
established based on the regression of the factors that are known before
In a situation where there is limited resource to carry out suicidal
interventions, the optimal cut-point was defined by the cross of the
‘practical ROC’ curve with the ‘resource line’ which was modelled with an
equation: TP = C - FP*(1 - Pr(POS)) / Pr(POS); where: C is a constant,
depends on percentage of the total clusters that can be intervened based
on the limited resource, and Pr(POS) is the percentage of positive
suicidal behaviour in the population. For an instance, if the cut-point
is ‘rank X’, any cluster that rank X and above will be intervened with the
USA Youth Risk Behaviour Surveillance Questionnaire and psychological
The percentage of positives suicidal behaviour in the surveyed
population was estimated as Pr(POS) = 8.8%. For an instance, with a
limited resource that can intervene only 20.5% of all the clusters, the
true positive (TP) without using a ‘practical ROC’ was estimated as 0.22
(based on the cross of the ‘resource line’ with the diagonal of a ROC).
With a ‘practical ROC’, the TP increased to 0.25 (based on the cross of
the ‘resource line’ with the ‘practical ROC’), which led to a TP-increment
of about 14%. However, with an ‘ideal ROC’, the TP was estimated as 0.40
(based on the cross of the ‘resource line’ with the ‘ideal ROC’); which
led to a TP-increment of about 82%. Thus, it is justified to seek future
research to establish a better predicting model.
Suicide in adolescence and young adulthood ranks among the 5 leading
causes of death in many countries. Ability to recognise suicidal behavior
at early stage so that intervention can be put in place to prevent loss of
life is critical. This study provides a methodology to implement suicidal
intervention in an optimum way in a situation of limited resources for
1. Paul CY Chen, Lai Kah Lee, Kam Cheong Wong, Jagmohni Kaur. Factors
relating to adolescent suicidal behavior: a cross-sectional Malaysian
school survey. J of Adolescent Health 2005; 37(4): 337.e11-337.e16.
Competing interests: No competing interests
Sorry, but I’m afraid that the rapid response Application of ROC Plot
in Predicting Adolescents’ Suicidal Behavior by Kam Cheong WONG, Lai Kah
Lee (MD, MPH), Paul C.Y. Chen (MD, MPH, FFCM), Jagmohni Kaur (MBBS, MSc,
MHSM) (13 January 2004) bases on common and terrible misunderstanding:
“…that ROC plot could be deployed to obtain optimum cut-off point … “
without any additional information.
Authors say nothing about the used criterion of optimality.
It is easy to see that there several reasonable criterions can be used
depending on the users interests and on the information available at the
moment: criterion of minimal expected cost of wrong classifications (a
Bayesian criterion); four criterions of the best rule under limited TP or
FP or odds ratio TP/FP or odds ratio (1-TP)/(1-FP); criterion of the best
rule under limited share of objects recognized as positive (and then
treated). All above criterions are represented on the ROC plane with their
families of the straight lines. The slopes of lines of equal expected cost
are equal to TP/FP = (PN/PP)*(costN/costP), were PN/PP is the odds ratio
of probability PN that a negative object needs to be recognized to
probability PP that a positive object needs it, costN/costP is a ratio of
misclassification costs, costN it is a cost of wrong classification of a
single negative object as positive, costP otherwise, positive object as
negative. The lines nearer to ideal point (TP = 1, FP = 0) represent the
lower expected costs. For the next criterion, the lines of the equal share
of the treated objects are: TP*PP + FP*PN = const.
Besides, there are sometimes proposed two heuristic criterions without
clear justifications and without needed any additional information about
practical diagnostic processes. First one maximizes area under curve AUC =
(1 + TP – FP)/2 under ROC for a fixed single rule, what can be interpreted
as assumption that there slope TP/FP = (PN/PP)*(costN/costP) = 1 is valid.
The second one minimizes the Euclidean distance from chosen ROC-point to
ideal one TP = 1, FP = 0.
Obviously, there the huge of distinctly various “optimal” cut-off points
on the same ROC curve can be determined.
What is a resume? The ROC curve characterizes all family of possible
and reasonable cut-offs, not any single cut-off. The main advantage of the
ROC analysis consists in this, that use of the ROC analysis creates
opportunity to delay a decision which cut-off point from considered ROC
curve recommend to apply it in practice. There the common misunderstanding
was, that the impatient user recognises this great advantage as the main
inconvenience, and he/she attempts to make final conclusion using some
heuristic criterion without clear justifications.
Maciej Górkiewicz, E-mail: email@example.com
Competing interests: No competing interests
The contribution of Altman and Bland on Receiver Operating
Characteristic (ROC) Plots (BMJ 1994; 309:188) is much appreciated. We
wish to substantiate a statement in their article which says “Having
determined that a test does provide good discrimination the choice can be
made of the best cut off point for clinical use.” We applied ROC plot in
determining the optimum cut-off point for predicting adolescents’ suicidal
behavior in a cross-sectional questionnaire survey on 4500 adolescent
students in Malaysia from June 2001 to August 2001. The questionnaire was
established with reference to Youth Risk Behavior Surveillance conducted
in the United States of America, modified to suit our local context.
Sampling of schools was done using stratified random sampling. Schools
were divided based on their districts and locality i.e. either urban or
rural. Students were assured that the information gathered would be
treated confidentially to ensure maximal response. The “suicidal
behavior” presented in this letter was “seriously consider attempting
Based on multiple logistic regressions, factors significantly
relating to the urban adolescents’ suicidal behavior were ‘felt sad or
hopeless’, ‘number of days felt unsafe to go to school’, ‘riding with a
driver who had been drinking alcohol’ (risk taking behavior), and ‘number
of days absent from school’. On the other hand, factors relating to the
rural adolescent suicidal behavior were ‘felt sad or hopeless’, ‘physical
fight’, and ‘physical fight resulting in injury’.
The predictive probabilities on the adolescents’ suicidal behavior
obtained from the above multiple logistic regression equation were used to
generate a ROC plot. Based on the data for the 2325 urban adolescents,
the area under the ROC plot was 0.829 which was significance with
p<0.000 and a 95% confidence interval that did not contain 0.5 i.e.
0.789 to 0.869
With reference to the ROC plot for urban adolescents, a cut-off point
‘0.02’ was selected as it gave a true positive rate (sensitivity) of 85.3%
and false positive rate (1-specificity) of 36.1% in predicting the urban
adolescents’ responses to the question “During the past 12 months, did you
seriously consider ending your life?”
Then, the multiple logistic regression equation and the ROC cut-off
value of ‘0.02’ were applied to predict the suicidal behavior of
adolescents from a semi-urban area known as Port Dickson (496
respondents). When the calculated probability by the multiple logistic
regression equation reached and exceeded 0.02; the participant was
predicted to have seriously considered attempting suicide. The prediction
was then compared with the actual respondent’s answer to the question:
“During the past 12 months, did you seriously consider ending your life?”
The true positive rate was 69.8% (30 of 43), and the false positive rate
was 36.5% (165 of 452).
The same approach was applied to study the data on rural adolescents’
suicidal behavior, which was split into modeling dataset (891 respondents)
and validation dataset (788 respondents). The area of the ROC plot for
the 891 rural adolescents’ suicidal behavior was 0.754 (p<0.000; 95%
confidence interval 0.685 to 0.822). The cut-off point “0.03” was
selected as it gave a true positive rate (sensitivity) of 77.0% and false
positive rate (1-specificity) of 32.5% in predicting the rural adolescents
in the modeling dataset. Then, the multiple logistic regression equation
and ROC cut-off point of 0.03 were applied on the validation dataset. The
true positive rate was 72.6% (45 of 62), and the false positive rate was
33.0% (238 of 721).
The multiple logistic regression and ROC optimum cut-off point
provided a reasonable good predictive ability to initialize appropriate
intervention on adolescent students who had seriously consider attempting
suicide. The true positive rates for both of the above models were chosen
to be as high as possible because of the serious consequence of a false
negative prediction. A false negative prediction might result in an
adolescent with serious suicidal ideation went away unattended.
In conclusion, we found that ROC plot could be deployed to obtain
optimum cut-off point for our study.
Competing interests: No competing interests