Statistics Notes: Diagnostic tests 3: receiver operating characteristic plots

Dear Sir,

We (Maciej Gorkiewicz, Kam Cheong Wong, Paul CY Chen, Lai Kah Lee,
Jagmohni Kaur) have been collaborating to clarify the application of ROC
methodology to select target group for suicidal intervention in a
situation of limited resources. This 'response to BMJ' facility has
enabled the authors to collaborate and enhance the application of ROC
methodology. The constructive feedback has been considered and the
'avoidable misunderstanding' between authors is clarified.

Suicidal intervention may include interviewing all students in the
target group with the relevant Youth Risk Behavior Surveillance
questionnaire and provides psychological counselling. The following
content has been accepted for presentation at the 30th Annual Conference
of the German Classification Society at Berlin Germany on 8th -10th March
2006.

Methodology:

With reference to the Youth Risk Behavior Surveillance conducted in
the United States of America (USA), a cross sectional survey was carried
out on 4500 adolescent students from 14 schools in a state in Malaysia.
The details of the research were published in the Journal of Adolescent
Health (USA)[1]. Chen et al[1] reported that the suicidal behaviour is
significant among Forms (or Grades as known in other country) in a school.
Within one Form in a school there could be more than one class. A Form in
a school can be regarded as a cluster because when intervention is to be
implemented, it is practical to intervene all the classes within the Form
in the school. There were a total of 73 clusters in the 14 schools.

Individual who has positive suicidal behavior is defined as 'positive
individual' (POS) who likely needs psychological support. The definition
of Positives (POS) is as follows:
POS: (Suicdone = 1) or [((thinksui = 1) or (suicplan = 1)) and (sad = 1)];
where Suicdone = 1 if suicide was carried out, thinksui = 1 if seriously
considered attempting suicide, suicplan = 1 if made a suicide plan, sad =
1 if felt sad or hopeless. Participants who did not meet the above
definition were assumed did not have suicidal behavior. Other factors
include gender; ethnic group (Malay, Chinese, Indian, others); district
(urban, semi urban, rural); alcohol (no, yes); and absenteeism (number of
days absent from school in the past 30 days).

All the 73 clusters were ranked according to their percentages of
positive suicidal behaviour. The cluster with the lowest %POS was ranked
1, and the cluster with the highest %POS was ranked 73. We can use the
rank of the cluster as a cut-off decision point to decide which clusters
are to be intervened. Every rank gives rise to different true positives
(TP) and false positive (FP). Thus, a Receiver Operating Characteristic
(ROC) curve can be plotted. This ROC is termed ‘ideal ROC’ because it is
established based on the % POS rank that is known after implementing the
USA Youth Risk Behaviour Surveillance Questionnaire.

On the other hand, a linear regression was performed on the dependent
variable (i.e. the rank) and independent variables (i.e. gender,
absenteeism, ethnicity, alcohol). The rank generated from the linear
regression is called ‘predicted rank’. Each ‘predicted rank’ gives rise
to different TP and FP with reference to the USA Youth Risk Behaviour
Surveillance Questionnaire. Thus, these ‘predicted ranks’ can be plotted
as another ROC curve which is called ‘practical ROC’. The ‘ideal ROC’ can
be established after intervention, but the ‘practical ROC’ can be
established based on the regression of the factors that are known before
intervention.

In a situation where there is limited resource to carry out suicidal
interventions, the optimal cut-point was defined by the cross of the
‘practical ROC’ curve with the ‘resource line’ which was modelled with an
equation: TP = C - FP*(1 - Pr(POS)) / Pr(POS); where: C is a constant,
depends on percentage of the total clusters that can be intervened based
on the limited resource, and Pr(POS) is the percentage of positive
suicidal behaviour in the population. For an instance, if the cut-point
is ‘rank X’, any cluster that rank X and above will be intervened with the
USA Youth Risk Behaviour Surveillance Questionnaire and psychological
counselling.

Results:

The percentage of positives suicidal behaviour in the surveyed
population was estimated as Pr(POS) = 8.8%. For an instance, with a
limited resource that can intervene only 20.5% of all the clusters, the
true positive (TP) without using a ‘practical ROC’ was estimated as 0.22
(based on the cross of the ‘resource line’ with the diagonal of a ROC).
With a ‘practical ROC’, the TP increased to 0.25 (based on the cross of
the ‘resource line’ with the ‘practical ROC’), which led to a TP-increment
of about 14%. However, with an ‘ideal ROC’, the TP was estimated as 0.40
(based on the cross of the ‘resource line’ with the ‘ideal ROC’); which
led to a TP-increment of about 82%. Thus, it is justified to seek future
research to establish a better predicting model.

Conclusions:

Suicide in adolescence and young adulthood ranks among the 5 leading
causes of death in many countries. Ability to recognise suicidal behavior
at early stage so that intervention can be put in place to prevent loss of
life is critical. This study provides a methodology to implement suicidal
intervention in an optimum way in a situation of limited resources for
implementing intervention.

Reference:

1. Paul CY Chen, Lai Kah Lee, Kam Cheong Wong, Jagmohni Kaur. Factors
relating to adolescent suicidal behavior: a cross-sectional Malaysian
school survey. J of Adolescent Health 2005; 37(4): 337.e11-337.e16.

Competing interests:
None declared