- Ann Oakley, professora,
- Deirdre Fullerton, research officera,
- Janet Holland, senior research lecturera,
- Sean Arnold, research officera,
- Merry France Dawson, research officera,
- Peter Kelley, research officera,
- Sheena McGrellis, research officera
- Correspondence to: Professor Oakley.
- Accepted 23 November 1994
Objectives: To locate reports of sexual health education interventions for young people, assess the methodological quality of evaluations, identify the subgroup with a methodologically sound design, and assess the evidence with respect to the effectiveness of different approaches to promoting young people's sexual health.
Design: Survey of reports in English by means of electronic databases and hand searches for relevant studies conducted in the developed world since 1982. Papers were reviewed for eight methodological qualities. The evidence on effectiveness generated by studies meeting four core criteria was assessed. Judgments on effectiveness by reviewers and authors were compared.
Papers: 270 papers reporting sexual health interventions.
Main outcome measure: The methodological quality of evaluations.
Results: 73 reports of evaluations of sexual health interventions examining the effectiveness of these interventions in changing knowledge, attitudes, or behavioural outcomes were identified, of which 65 were separate outcome evaluations. Of these studies, 45 (69%) lacked random control groups, 44 (68%) failed to present preintervention and 38 (59%) postintervention data, and 26 (40%) omitted to discuss the relevance of loss of data caused by drop outs. Only 12 (18%) of the 65 outcome evaluations were judged to be methodologically sound. Academic reviewers were more likely than authors to judge studies as unclear because of design faults. Only two of the sound evaluations recorded interventions which were effective in showing an impact on young people's sexual behaviour.
Conclusions: The design of evaluations in sexual health intervention needs to be improved so that reliable evidence of the effectiveness of different approaches to promoting young people's sexual health may be generated.
There should be greater awareness among health education researchers, practitioners, and policy makers of the need to base sexual health education strategies for young people on sound evidence of effectiveness
Journals should refuse to accept methodo- logically flawed papers
Funders should refuse to support studies with methodologically flawed designs
By using the best evidence available a randomised controlled trial should be under- taken of a sexual health education intervention for young people with 5–10 years of follow up
Reducing the risks some young people take with their health is an aim incorporated into current “health of the nation” targets in the United Kingdom. These include reducing by at least half the rate of conceptions among under 16s by the year 2000 and reducing by at least one fifth by this year (1995) the rate of gonorrhoea among men and women aged over 15.1 Health education interventions are widely seen as the most appropriate strategy for promoting young people's sexual health, particularly in view of the fact that many studies show low levels of information among young people about sexuality, reproduction, contraception, and sexually transmitted diseases2 3 4 5 and in view of the evidence that age at first intercourse is showing a steady decline with cohort age.6
School based health education has been poorly described and inadequately evaluated.7 8 9 10 11 Health education in general has been criticised for a low level of evaluation and poor evaluation design.12 13 Many evaluations describe the process of implementing interventions rather than the impact of these on health related outcomes. Though there is an important role for process and other observational studies, these cannot answer questions about effectiveness.
Electronic and hand searches were conducted for reports in English on sexual health interventions with young people aged 0–19. The following databases were used: the Social Science Citation Index (BIDS), Medline, Psyclit, Eric, the Health Education Authority's Unicorn database, and the National HIV/AIDS Prevention Information Service database. Hand searches were carried out of the journals Health Education Research, Health Education Journal, Health Education Quarterly, AIDS Education and Prevention, The Journal of School Health, and Family Planning Perspectives from 1982 to 1994. Contacts with other researchers generated additional studies, as did trawling through the bibliographies of located ones (which resulted in adding some pre-1982 studies). Unpublished studies were included when possible. Publication details were keyworded and stored on a computer reference manager.
Written reports of studies were reviewed by using a set of detailed guidelines for assessing methodological quality, following the principles in the Cochrane Collaboration14 and used in a previous review.15 There were eight methodological quality criteria: (a) clear definition of aims; (b) a description of the intervention package and design sufficiently detailed to allow replication; (c) inclusion of a randomly allocated control group; (d) provision of data on numbers of participants recruited to the study and control groups; (e) provision of preintervention data for the study and control groups; (f) provision of postintervention data for the study and control groups; (g) attrition rates reported for the study and control groups; (h) findings reported for each outcome measure as described in the aims of the study. These criteria were considered essential to a well designed evaluation. Adequacy of sample size was noted, but as so many studies had major design flaws, this information was usually redundant. Other important criteria such as adequacy of period of follow up and the appropriateness of the interventions studied and outcome measures selected were not included in the “essential” list because of the large element of subjectivity in assessing whether they had been met.
A study meeting all eight criteria was regarded as a “gold standard” study. Following on from other work,13 16 a smaller sample of core criteria from the list was selected in order to divide the studies into the two broad groups: “sound” and “flawed.” Sound studies were those which met the four criteria of employing randomly allocated control groups or control groups shown to be equivalent to the study groups before intervention on sociodemographic characteristics and measures used as outcome variables; providing preintervention and postintervention data; and reporting on all outcomes. If a study presented data based only on participants who provided information after intervention, there must have been evidence of equivalence between “stayers” and “drop outs,” and “intention to treat” analysis must have been used. Two reviewers with backgrounds in quantitative social science independently assessed each study. Any disagreements were discussed and resolved with a third reviewer and by discussion with the members of the study's steering group, which included statisticians familiar with the aims of generating evidence based reviews of effectiveness. The results of the reviewing process plus descriptive information on the studies were entered into a specialist computer database.
A total of 619 studies were located, of which 304 had a specific focus on sexual health. Hard copies were acquired for 270 studies. There were 87 reports of evaluations. These subdivided into 73 outcome evaluations and 14 process evaluations. The 73 outcome evaluations included six linked pairs of studies and one group of three linked studies. Separate outcome evaluations therefore totalled 65. The proportion of outcome evaluations located by electronic searches ranged from 12% (8; Unicorn) to 20% (13; Eric and Psyclit) to 40% (26; BIDS). Twenty three (36%) of the evaluations were located by hand searches and personal contacts. All except one of the outcome evaluations were published. Fifty nine (91%) were carried out in North America, three (5%) in the United Kingdom, two (3%) in other European countries, and one elsewhere. There were no studies targeting 0–11 year olds; 42 (65%) targeted 12–16 year olds and 23 (35%) 17–19 year olds. Forty eight (74%) of the studies described interventions in school settings. In 30 (46%) studies the focus of the intervention was on HIV and AIDS and the remaining 35 (54%) were concerned with pregnancy prevention, sexually transmitted diseases, or sexual health more generally. Of the 65 outcome evaluations, only 15 (23%) had a follow up interval of 12 months or more, and in 25 (38%) the follow up interval was three months or less.
Tables I and II give data from the methodological review of the outcome evaluations. Evaluations meeting the eight methodological criteria varied from all 65 which stated their aims clearly to 20 (31%) which used the design of a randomised controlled trial (table I). Only four studies gave information about the method of “random” allocation used. Only 34 reports discussed the issue of consent (of which only 20 discussed consent of participants). Of the 65 studies, only 16 gave intervention participation and refusal rates, and none provided information on the numbers of subjects who declined to take part in the study as a whole (not shown in table I). Table II shows the numbers of methodological criteria met by the 65 studies. Only four (6%) studies met all eight criteria; 43 (66%) met five or fewer. Twelve (18%) studies met the four core criteria.
Eleven of the methodologically sound studies were carried out in North America and one in Finland. Eight were school based, three comprised college students, and the 12th worked with runaway youths. Six of the 12 studies focused on HIV and AIDS within the broader context of sexual risk taking behaviour. Six included non-randomised control groups but provided information confirming the equivalence of the intervention and control populations on sociodemographic characteristics before intervention.17 18 19 20 21 22 A further three studies had used random allocation but failed to include information on the method used23 24 25; one used sealed envelopes26 and two used random numbers.27 28
Tables III and IV compare the claims to effectiveness made by authors of studies with those derived from the review process as a result of assessments made by members of the research team. (Reviewers' judgments about effectiveness in the flawed studies looked at the evidence presented in the paper irrespective of design features.) Overall there was 40% agreement between authors and reviewers on effectiveness (seven (11%) studies effective, 12 (18%) partially effective, two (3%) ineffective, four (6%) unclear, one (2%) harmful; and in four cases there was some agreement on effect (positive effect in three (5%), negative or unclear effect in one (2%)). In 35 (54%) cases authors said the intervention was effective and the reviewers disagreed, usually because the evidence was unclear.
Of the 12 sound studies, seven were considered by reviewers to be reports of effective (three) or partially effective (four) interventions (table IV).
All three effective interventions were carried out in North America. In the study by Barth et al a sex education programme delivered to high school students was aimed at increasing knowledge and student-parent communication.18 Outcome data were collected immediately after the programme and at six and 18 months. The experimental curriculum increased students' knowledge and improved communication with parents on sexual health matters. DiClemente et al looked at the impact of a course of AIDS and other information on sexually transmitted diseases in middle and high school students.19 Students exposed to the course had more correct answers to knowledge questions and were more accepting of classmates with HIV and AIDS. The third intervention judged to be effective was carried out with a group of runaway adolescents by Rotheram-Borus et al.20 Seventy eight runaways at one residential shelter given up to 30 HIV and AIDS education sessions directed at general knowledge, coping skills, access to health care, and individual barriers to safer sex were compared with 67 runaways at a non-intervention shelter. Sexual behaviours were assessed before the intervention and three and six months later. After the sessions young people reported more consistent condom use and less high risk sexual behaviour.
PARTIALLY EFFECTIVE INTERVENTIONS
Four interventions were judged partially effective by reviewers. Abramson et al evaluated an AIDS education course offered by a university biology department.17 Students exposed to the course were less likely than students taking another, unrelated course to believe that strategies such as quarantining people with AIDS were an effective public health measure, were more informed about modes of HIV transmission, and were more likely to report both carrying and using condoms. Ashworth et al tested the effectiveness of a single lesson incorporating video material and a discussion led by AIDS educators in changing school students' knowledge.27 Increased knowledge was displayed two weeks later among students given the lesson; students were less worried about current exposure to HIV but more worried about acquiring it as adults.
Wenger et al examined the effect of AIDS education and testing for HIV on communication about the risk of HIV infection and sexual behaviour in American college students.26 Students attending the health clinic who were interested in HIV education and testing were randomly assigned to three groups—education alone, education plus testing, and a control group. Those in the education plus testing group reported increased communication with sexual partners about HIV risk six months later, but there were no differences between the three groups in knowledge about AIDS, number of partners, or condom use. Herz et al evaluated the impact of a family life education programme for inner city, minority group elementary school students with a mean age of 13 years.24 The aim of the programme was to reduce teenage pregnancies, and the intervention consisted of 15 sessions. Curriculum topics included personality, physical, and emotional development; nutrition and hygiene; reproduction; relationships; and developing educational and career goals. In comparison with controls, programme participants displayed improved knowledge, increased awareness of birth control methods, and a greater tendency (boys only) to acknowledge mutual responsibility for contraception.
Two interventions were judged ineffective. Hamalainen and Keinanen-Kiukaanniemi conducted a controlled study of the effect of HIV and AIDS knowledge and attitudes of one 45 minute lesson on safer sex and sexually transmitted diseases, including HIV and AIDS, with 15 year old schoolchildren in Finland.23 Thomas et al reported an evaluation of the McMaster “teen” programme with 13 year olds and a four year follow up assessment based on questionnaires and psychological tests.28 The programme aimed at providing information about development, sexuality, and relationships; improving communication; and developing problem solving and decision making skills. It did not include information on contraceptive methods and their use, as this lay outside the Ontario Ministry of Education guidelines. Outcome data are being collected over four years. Evaluation so far indicates no greater use of contraception or of abstinence among the students.
INTERVENTIONS WITH UNCLEAR EFFECT
The impact of two interventions was judged to be unclear. In the study by Thomas et al intervention consisted of an eight month, 12 session sex education programme aimed at increasing young adolescents' knowledge of human sexuality by means of a multimedia approach with ninth grade students in a rural South Carolina school.21 Wanlass et al aimed at examining the impact among university undergraduates of different instructional formats on students' attitudes to sexuality and sexual behaviour.25 Four experimental conditions compared the effect of a lecture alone, a small group discussion alone, both together, and a lecture plus review on attitudinal and knowledge outcomes with that in a control group.
One intervention, reported on by Christopher and Roosa, was judged to be harmful.22 The study evaluated an abstinence education programme (the “success express” programme) targeted at a group of low income minority group youths in Arizona with a mean age of 13 years. The programme consisted of six lessons. It was designed to reduce premarital sexual activity by promoting premarital sexual abstinence through information about reproduction and the implications and risks of sexual behaviour and the development of decision making skills. The programme had a high rate of attrition (over one third of subjects in the intervention group and almost a quarter of the controls); none of the desired changes in attitudes or behaviour occurred in the sample as a whole or in the subgroup who were virgins before the intervention. More young men in the intervention group than in the control group claimed to have initiated intercourse by the end of the programme.
These results point to the need to improve on evaluation design in sexual health promotion. Most sexual health interventions with young people are not evaluated. Of those that are, fewer than one in five meet the minimum criteria for a methodologically sound evaluation. Major design problems identified in this review, which support the findings of others,9 10 29 are the use of non-equivalent control groups or failure to use control groups, relying on a pretest and post-test design, high attrition rates (often in the region of 50%), and failure to discuss the implications of attrition.
The NHS research and development programme emphasises the need to base clinical practice on the scientific evidence derived from systematic reviews of effectiveness.30 There is an equivalent need to base social interventions in health care, including health education and health promotion, on sound evidence about which strategies are effective and which are not. In sexual health, as in other topics, observational studies of effectiveness “run a poor second” to randomised controlled trials.31 Descriptive studies reporting on the processes in implementing different types of intervention, or analysing interview and other data relating to the experiences of study participants, are clearly important but cannot answer questions about the effectiveness of different approaches in changing health related outcomes. Randomised controlled trials provide a remedy to the inferential uncertainties of non-experimental designs. Among health education researchers the notion seems to be widespread that random allocation to experimental groups is “ethically” more dubious than the uncontrolled experimentation resulting from less robust designs or from the implementation of unevaluated programmes. If spending large sums on research with seriously incapacitating design faults is a “scandal” in clinical medicine,32 it is equally so in health education.
Many of the lessons of systematic reviews in clinical medicine are applicable to health education. For example, the conclusion that non-randomised studies tend to yield larger estimates of treatment effects than studies using random allocation33 has substantial implications for many commonly accepted claims to the effectiveness of health education. However, many of the flaws identified in this review seem to be sustained in ongoing studies in Britain (Sex Education Forum practice database, 1994).
While observational evidence suggests that sexual health education may increase young people's knowledge but not change their behaviour,34 35 many studies do not even examine behavioural outcomes.36 Of the seven methodologically sound studies identified in this review and judged to present evidence of intervention effectiveness, only two showed short term effects on young people's reported sexual behaviour. There is no evidence that providing practical information and contraception leads to sexual risk taking behaviour, but there is evidence that chastity education may encourage sexual experimentation. Many young people want practical information and help in avoiding unwanted pregnancy and sexually transmitted diseases rather than didactive approaches emphasising anatomical or moral aspects of sexual behaviour,37 38 and they want this within a context which is sensitive to the real material and other constraints of young people's lives.29
Recommendations for future work
Use the results of evidence based reviews to design future interventions
Base interventions on what young people say they want in sexual health information and resources
Focus on changing behaviours rather than simply on knowledge or attitudes
Evaluate intervention effectiveness by using the design of randomised controlled trials
Include an adequate follow up period to look at both short term and long term effectiveness
Further well designed studies are needed with a long enough follow up to justify conclusions about the effectiveness of sexual health education in reaching “health of the nation” goals. In the absence of such evidence much of the present endeavour in sexual health promotion for young people can only be described as “knitting without a pattern.”39
Not all relevant studies are included in this review. Some hard copies of papers remained to be located, and data entry stopped on 23 March 1994; hence studies located after that date were not entered. We should be pleased to hear of further studies that ought to be included when we update this review.
This work was supported by the Medical Research Council, the Health Education Authority, and the University of London Institute of Education.