Rapid Responses to:

PRIMARY CARE: A Scott Lennox, Liesl M Osman, Ehud Reiter, Roma Robertson, James Friend, Ian McCann, Diane Skatun, and Peter T Donnan Cost effectiveness of computer tailored and non-tailored smoking cessation letters in general practice: randomised controlled trial BMJ 2001; 322: 1396 [Abstract] [Full text]

Rapid Responses: Submit a response to this article

Rapid Responses published:

Putting evidence into practice

Ruth Livingstone   (10 June 2001)

Non-tailored smoking cessation letter is available to download

Scott Lennox   (12 June 2001)

Is simple written advice on stopping smoking really so effective and cheap?

Robert Bunney   (25 June 2001)

Yes, it really is

Scott Lennox   (6 July 2001)

Comments on Lennox et al.

Victor J Strecher, Wayne F Velicer, Rodrick Little   (9 August 2001)

Presentation of results is inappropriate

Guido Wendland   (14 August 2001)

Reply to Strecher et al.

Scott Lennox, Ehud Reiter, Liesl Osman   (31 August 2001)

General Comment on Tailoring

Ehud Reiter   (31 August 2001)

Re: Presentation of results is inappropriate

Scott Lennox, Peter Donnan, Liesl Osman   (7 September 2001)

New correspondence email

Ehud R Reiter   (15 October 2002)

Putting evidence into practice 10 June 2001
Ruth Livingstone, General Practitioner Little Surgery, Stamford Send response to journal: Re: Putting evidence into practice	Congratulations on a useful piece of research. Mass mailing of non- tailored smoking cessation advice letters appears to be a cost effective intervention. Any practice with a good electronic database could use this method with minimum administrative costs. However, in order to do so, we need details of the information provided in the non-tailored letter. Could this information be provided on the BMJ Website?
Non-tailored smoking cessation letter is available to download 12 June 2001
Scott Lennox, Clinical senior lecturer Department of General Practice and Primary Care, University of Aberdeen Send response to journal: Re: Non-tailored smoking cessation letter is available to download	The non-tailored smoking cessation letter evaluated in our paper can be downloaded in Word format from http://www.csd.abdn.ac.uk/research/stop/letter.html Copyright rests with the Chief Scientist Office of the Scottish Executive Health Department. The letter may be used free of charge for non-profitmaking purposes.
Is simple written advice on stopping smoking really so effective and cheap? 25 June 2001
Robert Bunney, General Practioner Brannam Medical Centre,Barnstaple,N.Devon,EX329EW Send response to journal: Re: Is simple written advice on stopping smoking really so effective and cheap?	Surely any summary of the effectiveness or cost of sending non tailored smoking cessation letters in primary care should use as a denominator the whole 7427 smokers originally identified from practice records not the 2610 who were left after responding to a covering letter,consent form,a questionnaire to collect information and two reminders at three week intervals? Of course it cannot be assumed that the smokers left to enter into the trial were more likely to respond to advice but it seems highly probable and the different resource implications are considerable.
Yes, it really is 6 July 2001
Scott Lennox, Clinical senior lecturer Department of General Practice and Primary Care, University of Aberdeen Send response to journal: Re: Yes, it really is	Dr Bunney is correct that a cost effectiveness analysis should take into account all targeted individuals and not just those who chose to participate. Our study reported analyses using both methods. The cost of £464 for the 15 additional quitters in the non-tailored group compared with the control group (£31 per additional quitter) is based only on the costs for the 846 smokers who agreed to participate. We then presented data based on all individuals originally targeted in the non-tailored group: the 7427 individuals targeted at the start of the study were divided into the three arms of the trial, giving a nominal 2476 in each group. The costs of subsequent analyses were based on this figure. Since we had no information on how non-respondents would have reacted had they received a non-tailored letter, we calculated cost effectiveness for best case and worst case scenarios. Our worst case scenario assumed that the 1630 non-respondents (2476-846) would have been entirely unmoved by the letter and so would have provided no additional quitters. This assumption produced a cost per quitter of £89, which remains extremely low in comparison with other smoking cessation interventions. Diane Skatun, Research fellow, Health Economics Research Unit, University of Aberdeen Scott Lennox, Clinical senior lecturer, Department of General Practice and Primary Care, University of Aberdeen
Comments on Lennox et al. 9 August 2001
Victor J Strecher, Professor; Professor; Professor Univ. of Michigan; Univ. of Rhode Island; Univ. of Michigan, Wayne F Velicer, Rodrick Little Send response to journal: Re: Comments on Lennox et al.	In their recent BMJ article, Lennox et al.(1) state that: “Two studies in North America investigated computer generated personalized letters as a method of encouraging smoking cessation” referring to a set of two studies Strecher and colleagues published in 1994(2) and a study published by Velicer and colleagues in 1993.(3) (The cited Velicer et al.(3) paper was only a general description of computer-based or tailored interventions. Readers interested in the actual outcome study should see Prochaska et al.(4)). Lennox et al. fail to reference a number of published studies examining the impact of tailored print materials, the majority of which demonstrate positive results.(5-10) In a recent meta-analysis of tailored and untailored self-help smoking cessation materials, Lancaster et al.(5) found an odds ratio of 1.41 (CI: 1.14-1.75) for computer-tailored materials compared with untailored or stage-matched materials (8 studies), and a non-significant odds ratio of 1.08 (CI: 0.97-1.21) for untailored materials compared with brief personal advice (28 studies). The odds ratio found in their meta-analysis of tailored print materials compares favorably with pro-active telecounseling for smoking cessation: a high-reach but far more expensive intervention. Like other studies, Lennox et al. found a low cessation rate from an untailored smoking cessation letter when compared with a no-intervention control group. The authors, however, also found a low cessation rate from their tailored smoking cessation letter, less than half that found in other studies. Lennox et al. conclude that their low cessation rate was a function of methodological strengths in their study: that it “was carried out on a randomly chosen population who had not actively volunteered to take part in the intervention and had no special motivation to quit.” While this was the case for their study, it was also the case in many other tailoring studies that found much higher cessation rates. Both the Strecher et al.(2) studies and three of the University of Rhode Island (URI) studies(8-10) devoted a great deal of attention and resources to proactively obtaining representative samples of cigarette smokers. Strecher et al. called patients at home or approached patients in waiting rooms, achieving 90% and 78% participation rates, respectively. The URI group used random digit dialing in one large study(10) and contacted all members receiving services from a large Northeastern USA Health Maintenance Organization in the other two,(7,8,10) achieving 80-85% participation rates. The Lennox et al. study sent mail questionnaires to a sample of smokers obtained from medical records, which is a reasonable methodological approach. However, given their comparatively low response rate (42%), it is misleading to imply that their lower cessation rates were a result of selecting a broader spectrum of smokers. The authors also state that: “In contrast, the high rates of cessation in the other studies were based on self reported cessation, and subjects who dropped out were omitted from the calculations of rates of continuing smoking.” The two studies presented in Strecher et al.(2) examined cessation rates both by treating subjects lost to follow-up as treatment failures and by omitting them from the analysis, yielding similar conclusions. The Prochaska et al.(9) study actually reported five alternative ways of handling missing data, including several of the recently developed statistical methods for addressing the problem. A recent consensus conference on research methods for smoking cessation research11 strongly recommended the use of modern statistical procedures to address missing data problems instead of the type of ad hoc procedure employed by Lennox et al. Lennox et al. discuss the Strecher et al.(2) results as if there was one study when in fact there were actually two. They also state that the Strecher et al. study: “based its success rate on a sub-group of light smokers who had intended to quit.” In fact, neither of the two studies reported by Strecher et al.(2) used only smokers intending to quit. The first study enrolled smokers in precontemplation, contemplation, and preparation stages. The second study enrolled smokers in contemplation and preparation stages. The contemplation stage does not necessarily include those who intend to quit. A contemplation question asked about “interest” in quitting over the next six months – certainly not to be equated with “intent.” The two studies reported by Strecher et al.(2) hypothesized that light-moderate smokers would be more responsive to a behaviorally-tailored intervention, given that heavier smokers would be more likely to be addicted to nicotine – a question that had not been examined or even carefully considered a decade ago. Strecher et al. reported what was regarded by the authors to be a set of interesting and fairly consistent findings. The findings were considered promising and further research was suggested to incorporate nicotine replacement therapy with behaviorally-tailored cessation materials. Addition of tailored behavioral materials to NRT has now been found to significantly improve cessation rates over NRT with untailored behavioral materials, and has been demonstrated in both research and practice to be a viable behavioral adjunct to NRT.12 The Lennox et al. study differs from other tailored message studies in its use of biochemical validation. Lennox et al. claim that their low cessation rates were in part due to the validation of self-reported cessation behavior. Their validated success rates (3.5% overall) were, indeed, far lower than their self-reported cessation rates (6.0% overall). Had the authors actually obtained biochemical samples from all but a negligible group of participants, we would feel more comfortable with the use of biochemically-validated cessation as the primary outcome variable. Unfortunately, biochemical samples were not obtained for a substantial group of participants: 15 subjects could not be followed and 7 refused to provide a sample. One sample was lost in the laboratory. Thirty-three subjects “had started smoking again before a sample could be obtained” from a research nurse. No information is provided regarding the time lag between the self-report at six months and the time a sample could be obtained. Apparently, cessation was validated among 89 of 98 (91%) smokers who provided a sample that was actually tested. Classifying as treatment failures all of the respondents who claimed to have quit smoking at six months but, for many reasons, did not receive a biochemical test, is an extremely conservative approach to computing success rates, likely in our view to considerably understate the true rate. In fact, large population trials of smoking cessation typically do not collect biochemical assays for this very reason: logistic follow-up of all reported quitters is nearly impossible, and the classification of all of these respondents introduces its own error. In smaller, more tightly-controlled studies, biochemical validation is generally easier and warranted. The Lennox et al. study was not tightly-controlled – in fact, the research staff were apparently never in direct contact (physically or even orally) with the subjects. The issue of whether biochemical validation of self-reported abstinence is necessary has been studied extensively. The decision to rely on self-reported abstinence is usually made in light of the findings of two reviews of over 20 studies each with tens of thousands of smokers,(13,14) data from four large-scale randomized smoking cessation trials (N>5,000),(15) and the large ancillary study performed by the COMMIT Research Group.(16) These publications unanimously question the need for biochemical validation in large-scale population-based studies, demonstrate several sources of measurement error in biochemical assessments reducing their validity as the gold standard of accuracy of smoking behavior, and report only minimal differences between self-reported and biochemically validated quit rates. Misreporting does occur, and biochemical validation appears to be most appropriately used, in high-demand interventions, such as in a cessation clinic or group.(14,17) A recent consensus conference on research methods for smoking cessation research(18) does not recommend the use of biochemical validation in large scale population-based studies. Given that biochemical validation was employed, a more informative analysis plan would have included results presented under a variety of different assumptions. For example, in addition to the conservative approach of the authors, analyses could be done using reported cessation rates, and imputed validation quit rates that applied the 91% validation rate amongst those biochemically tested to those who reported quitting but were not biochemically validated for reasons unrelated to the outcome. Drop-outs could be excluded, or counted as treatment failures. If consistent results are obtained under these alternative approaches, then confidence in the conclusions is bolstered; otherwise sensitivity of the outcome to this methodological issue can be assessed. One potential, but likely only partial, explanation for the much lower cessation rates in the Lennox study is the abbreviated follow-up period (6 months). Other studies have employed much longer follow-up periods such as 18 months(4,8,10) or 24 months.(9) These studies have also presented some evidence that tailored communications result in a delayed treatment effect. This is particularly the case in studies that include all smokers since early stage smokers, i.e., precontemplators, will need some time to progress to cessation. In the four Rhode Island studies, the point prevalence cessation rate at 6 months was 13.9% compared to 22.9% at 18 months. It is important to remember that failure to reject the null hypothesis means that the evidence from the study in question was consistent with the hypothesis of no treatment effect. It does not mean that the null hypothesis is correct. There are many explanations for the failure of study to reject the null hypothesis, including a poor design, an inadequate sample size, poor implementation of the intervention, etc. It certainly does not mean that studies reporting a positive result are incorrect. Another explanation for the low cessation rates found in the tailored letter group in Lennox et al. study is that the tailored materials developed by Lennox et al were simply not effective, for reasons that include an inappropriate conceptual model, a lack of an empirical basis for the decision rules, and lack of tailoring depth. Having taken the their program online (http://www.csd.abdn.ac.uk/research/stop/onlineQ.htm), we found the amount of information extracted from the data to be very similar to their untailored letter. The information was not very instructive and there was very little sense that connections between psychosocial constructs were being made. Stage-based messages seemed very crude, despite a great deal of research over the past decade finding significant variance within stages of change. The fact that participants in Lennox et al. did not find the tailored letters any more useful or relevant than the untailored letter supports the idea that the tailored messages in their intervention do not have had sufficient fidelity. Brug et al.(19) found their tailored dietary fat reduction letters to be both more useful and more relevant than untailored letters. Their tailored letters, in turn, resulted in significantly higher dietary fat reduction. Early versions of message tailoring, including those developed for the Strecher et al.2 studies, “fired” blocks of text, with substitutions made within the text for specific variables, on the basis of individual variable classifications. More recent embodiments of tailoring tend to be longer, use many connections between variables, are more psychologically detailed, and include tailored follow-up materials. Thus, the lack of evidence of efficacy of the Lennox. et al. intervention cannot be generalized to other, high-depth, interventions. The Lennox et al. system is broadly based on the Transtheoretical Model developed by Prochaska and colleagues.(20) However, it departs in several significant ways from that model, with little apparent empirical or theoretical justification. The first three stages are transformed into seven stages that combine aspects of the temporal dimension, i. e., the stages of change, and the Decisional Balance from the intermediate outcome dimension of the model. An empirically-based set of subtypes within stages is available but was not employed here.(21,22) The self-efficacy variable, also from the intermediate outcome dimension, was employed in a second level of tailoring along with several other variables from other theoretical models. The processes of change were apparently omitted. Decision rules determine what intervention materials are matched to each smoker. When the decision rules are not based on either a strong theoretical basis or empirical results, the basic assumption of tailored interventions, that appropriate materials are presented to each individuals, may be seriously violated. To summarize: a previous review,(6) found that a majority of tailoring studies in the smoking area demonstrated positive results. This conclusion was strongly supported in a more recent meta-analysis of tailored print materials(5) and an increasing number of recent publications. There are also a number of studies supporting the efficacy of tailoring in other areas. The large majority of tailoring studies in the dietary fat reduction area have demonstrated positive results.(24) Positive results of tailored materials have also been demonstrated in the areas of breast cancer screening (e.g., Skinner et al.(25) and Rakowski, et al.;(26) and physical activity (e.g., Marcus et al.(27) and Bock et al.(28)). In each of these areas, there are also negative results. Simply asking whether tailoring works is like asking whether movies entertain. Clearly some entertain and others do not. Different movies also entertain different audiences. A recent request for applications from the U.S. National Cancer Institute (reference) states that: “The NCI now funds many studies that compare tailored print interventions to usual care interventions ... However, when these studies have not been successful in achieving significant impact, it usually has not been possible to identify the reasons”. We believe that the Lennox et al. study falls into this category. Accordingly, the next generation of tailoring research should begin to examine the “active ingredients” of tailoring, for whom these active ingredients work, and why they work.(6,29,30) Victor J. Strecher, PhD, MPH Professor and Associate Director, Cancer Prevention and Control University of Michigan Comprehensive Cancer Center University of Michigan Wayne F. Velicer, PhD Professor and Co-Director Cancer Prevention Research Center University of Rhode Island Rodrick Little, PhD Professor and Chairman, Department of Biostatistics School of Public Health University of Michigan References 1. Lennox AS, Osman LM, Reiter E, Robertson R, Friend J, McCann I, Skatun D, Donnan PT. Cost effectiveness of computer tailored and non-tailored smoking cessation letters in general practice: randomised controlled trial. BMJ. 2001; 322:1396. 2. Strecher VJ, Kreuter M, Den Boer DJ, Kobrin S, Hospers HJ, Skinner CS. The effects of computer-tailored smoking cessation messages in family practice settings. J Fam Practice. 1994; 39: 262-270. 3. Velicer WF, Prochaska JO, Bellis JM, DiClemente CC, Rossi JS, Fava, JL, Steiger JH. An expert system intervention for smoking cessation. Addict Beh. 1993; 18: 269-290. 4. Prochaska JO, DiClemente CC, Velicer WF, Rossi JS. Standardized, individualized, interactive and personalized self-help programs for smoking cessation. Health Psychol. 1993; 12: 399-405. 5. Lancaster T, Stead L, Silagy C, Sowden A. Effectiveness of interventions to help people stop smoking: findings from the Cochrane Library. BMJ 2000; 321: 355-358. 6. Strecher VJ. Computer-tailored smoking cessation materials: A review and discussion. Pat Educ and Counseling 1999; 36: 107-117. 7. Velicer WF, Prochaska JO. An expert system intervention for smoking cessation. Pat Educ and Counseling 1999; 36:119-129. 8. Velicer WF, Prochaska JO, Fava JL, Laforge RG, Rossi JS. Interactive versus non-interactive interventions and dose-response relationships for stage matched smoking cessation programs in a managed care setting. Health Psychol 1999; 18: 21-28. 9. Prochaska JO, Velicer WF, Fava JL., Rossi JS, Tsoh JY. Evaluating a population-based recruitment approach and a stage-based expert system intervention for smoking cessation. Addict Beh 2001; 26: 583-602. 10. Prochaska JO, Velicer WF, Fava JL, Ruggiero L, Laforge RG, Rossi JS, Johnson, SS, Lee PA.. Counselor and stimulus control enhancements of a stage-matched expert system intervention for smokers in a managed care setting. Prev Med 2001; 32: 23-32. 11. Hall SM, Delucchi K, Velicer W F, Kahler C, Ranger-Moore J, Hedeker D, Tsoh J, Niaura R. Statistical analysis of randomized trials in tobacco treatment. Nic and Tob Res. (in press) 12. Shiffman S, Paty JA, Rohay JM, DiMarino ME, Gitchell J. The efficacy of computer-tialored smoking cessation material as a supplement to nicotine polacrilex gum therapy. Arch of Intern Med. 2000; 160: 1675-81. 13. Patrick DL, Cheadle A, Thompson DC, Diehr P, Koepsill T, Kinne S. The validity of self-reported smoking: A review and meta-analysis. Am J Public Health. 1994; 84: 1086-1093. 14. Velicer WF, Prochaska JO, Rossi JS, Snow MG. 1992. Assessing outcome in smoking cessation studies. Psychol Bull 111:23-41. 15. Glasgow RE, Mullooly JP, Vogt TM, Stevens VJ, Lichtenstein E, Hollis JF, Lando HA, Severson HH, Pearson KA, Vogt MR. 1992. Biochemical validation of smoking status in public health settings: pros, cons, and data from four low intensity intervention trials. Addict Behav 18:504-527. 16. COMMIT Research Group. Community Intervention Trial for Smoking Cessation (COMMIT): I. Cohort results from a four-year community intervention. Am J Public Health. 1995; 85:183-192.. 17. Strecher VJ, Becker MH, Clark NM, Prasada-Rao, P. Using patients' descriptions of alcohol consumption, diet, medication compliance, and cigarette smoking: the validity of self-reports in research and practice. J Gen Intern Med 4:160-166, 1989. 18. Benowitz, NL, Jacob III P, Ahijevych K, Jarvis MJ, Hall S, LeHouezec J, Hansson,A, Lichtenstein E., Henningfield J, Tsoh J., Hurt RD, Velicer WF. Biochemical Verification of Tobacco Use and Cessation. Nic and Tob Res (in press). 19. Brug J, Steenhuis I, Van Assema P, De Vries H. The impact of a computer tailored nutrition intervention. Prev Med, 1996 20. Prochaska JO, Velicer WF. The Transtheoretical Model of health behavior change. (Invited paper). Am J Hlth Prom, 1997 12, 38-48. 21. Velicer WF, Hughes SL, Fava JL, Prochaska,JO, DiClemente CC. An empirical typology of subjects within stage of change. Addict Beh. 1995; 20: 299-320. 22. Norman GJ, Velicer WF, Fava JL, Prochaska JO. Cluster subtypes within stage of change for a representative sample of smokers. Addict Beh. 2000; 25: 183-204. 23. Patrick DL, Cheadle A, Thompson DC, Diehr P, Koepsell T, Klinne S. The validity of self-reported smoking: A review and meta-analysis. Am J Public Health. 1994; 84:1086-1093. 24. Brug J, Campbell M, van Assema P. The application and impact of computer-generated personalized nutrition education: a review of the literature. Patient Educ Couns. 1999; 36:145-56. 25. Skinner CS, Strecher VJ, Hospers H. Physician's recommendations for mammography: do tailored messages make a difference? Am J Public Health. 1994, 84:43-9. 26. Rakowski W, Ehrich B, Goldstein MG, Rimer BK, Pearlman DN, Clark MA, Velicer WF, Woolverton H. Increasing mammography among women aged 40-74 by use of a stage-matched, tailored intervention. Prev Med. 1998; 27: 748-756. 27. Marcus BH, Emmons KM, Simkin-Silverman LR, Linnan LA, Taylor ER, Bock BC, Roberts B, Rossi JS, Abrams DB. (1998). Evaluation of motivationally tailored vs. standard self-help physical activity interventions at the workplace. Amer J Health Promo. 1998; 12: 246-253. 28. Bock BC, Marcus BH, Pinto BM, Forsyth LH. Maintenance of physical activity following an individualized motivationally tailored intervention. Ann Behav Med. 2001 Spring; 23:79-87. 29. Abrams D, Mills S, Bulger D.Related Articles. Challenges and future directions for tailored communication research. Ann Behav Med 1999 21: 299-306. 30. Rimer B, Glassman B. Is there use for tailored print communications in cancer risk communications? J Natl Cancer Inst Monogr. 1999; 25:140-8.
Presentation of results is inappropriate 14 August 2001
Guido Wendland, Statistician Institute of Health Economics and Clinical Epidemiology Send response to journal: Re: Presentation of results is inappropriate	I see no reason to believe, why non-tailored intervention should be more effective than tailored intervention. On the contrary, I bet the tailored intervetion was thought to improve the smoking cessation benefit. The presentation of the results - picking for the largest difference - is thereore data-driven. A pre-planned analysis of this situation must account for the multipicity of possible comparisons. Possible approaches could have been to reduce the significance levels simultaneously or to order hypertheses beforehand. Obviously this was not done in this study. A sensible approach would for example first compare letter (whether tailored or non-tailored) versus no letter. That would yield an 52%-increase in self-reported smoking cessation (3.9% vs 2.6%) which is not statistically significant (p=0.08). Furthermore interactions between treatment group and reporting bias seems possible for me, because people might be more inhibited to "lie" under certain circumferences. Again, it is questionable why the cost-effectiveness analysis only compared non-tailored letter vs no letter. Apart from that, reporting of cost-effectiveness should come with some sort of sensitivity analysis (probabilistic or non-probabilistic). Here this sensitivity analysis is based on some additional assumptions but not on the observed data variability. If the sensitity analysis would have been based e.g. on the reported confidence limits the results would not have been very inspiring, but would have better reflected the uncertainty of the presented results. To conclude, the success of the "intervention" was too small ,if any, to draw conclusions concerning the cost-effectiveness. Analyses, where target events are so small either need much larger differences between groups or much larger sample sizes. Competing interests: None
Reply to Strecher et al. 31 August 2001
Scott Lennox, Clinical senior lecturer, lecturer, senior research fellow University of Aberdeen, Ehud Reiter, Liesl Osman Send response to journal: Re: Reply to Strecher et al.	Prof. Strecher remarks on the fact that we cite only two other studies of tailored interventions for smoking cessation. This was in our introduction, in the context of explaining why we chose to conduct our study, and cites the only two studies that were published at the time (1996). We cite and comment on Prof. Strecher's own review of the tailoring literature(1)in our discussion section. While our interpretations of the existing literature may vary, we share Prof. Strecher's view that some movies entertain and some don't. (In that respect, one wonders about the usefulness of reviews that calculate odds ratios for the entertainment values of groups of disparate movies). He is certainly correct that the lack of effectiveness of our tailoring software does not mean that other tailoring software cannot work; indeed we see the main message of our paper being "non-tailored letters are effective", not "tailored letters are not effective". Perhaps the most important methodological lesson from our work is that studies of the effectiveness of tailored patient information material should include a control group that receives non-tailored material that is as similar as possible to the tailored material. Some of the experiments cited by Prof. Strecher compared cessation rates in groups that received tailored material to control groups that received no material, or that received non-tailored material that was very different from the tailored material. Since our study has shown that non-tailored material can be surprisingly effective (a finding mirrored in a recent study of tailored and non-tailored dietary intervention(2)), a good non-tailored control is essential to support claims of the effectiveness of tailoring. As Prof. Strecher mentions, there is a fundamental difference between our tailoring software, STOP, and previous systems, and that is the source of tailoring rules. The systems Prof. Strecher describes base their tailoring rules on psychological models such as the Transtheoretical Model. Although STOP was partially based on this model, it was primarily based on tailoring rules acquired from health professionals during "knowledge acquisition" sessions, using techniques developed by the expert -systems community. For example, we asked health professionals to write letters based on smoker questionnaires, analysed the strategies they used to do this, and attempted to model the computer software on these strategies. The "knowledge acquisition" process, as well as some of the problems we encountered, are described elsewhere.(3) Our overall conclusion was that this was a good way of acquiring hypotheses about appropriate tailoring rules, but that these hypotheses should have been empirically tested before being incorporated into the tailoring software. As a point of clarification, Prof. Strecher mentions the Web version of STOP. This in fact is a simplified version of the main STOP system used in the clinical trial, and in particular omits one type of tailoring (importance-based size constraint optimisation(4)) which is computationally time-consuming. The Web version is intended to give people who are new to the area an idea of what STOP does, and certainly in concept is similar to STOP, but it is not identical. The issue of biochemical validation in low-intensity studies is complex, and remains controversial. There are essentially two issues. The first is whether self-report is likely to be equally accurate in control and intervention subjects. The evidence for this is not as clear- cut as Prof. Strecher suggests. The meta-anlaysis by Patrick et al.(5) included only three intervention studies: self-reports were less accurate in these studies, and the authors concluded that biochemical validation should be considered in intervention studies. The COMMIT study(6) used self-report in its main analysis. It mentioned an ancillary study of biochemical validation but gave no details of study size, methodology or data. The review by Velicer et al.(7) rested its case for self-report in minimal intervention studies on an assumption: namely that subjects in minimal intervention studies would have little incentive to lie about their smoking status. While this may seem a reasonable assumption, it is not one for which there is any convincing evidence: the fact remains that it is not possible to verify smoking status in subjects who do not present for validation. The same applies to the work of Glasgow et al.(8): it could say nothing about the true smoking status of the 27% of self- reported quitters who did not undergo validation, nor about the original smoking status of the 12% who claimed to have restarted smoking in the interval between survey and biochemical validation. Since low-intensity interventions tend to produce only modest increases in cessation rates, even a small difference in deception rate between control and intervention groups could affect outcomes. As Prof. Strecher points out, there are potential inaccuracies in biochemical validation, and in treating as continuing smokers all subjects lost to follow up and all respondents not undergoing validation. The major potential source of bias in our approach is that subjects in the different arms may not have been equally likely to take part in follow up. One could argue that control subjects, having had no cessation letter as such, might feel more apathetic towards the study and be less likely to respond, even if they had stopped smoking. This would produce a falsely low cessation rate in the control group. In fact our data shows that slightly more control group subjects completed follow-up than did subjects in either of the letter groups. The second issue regarding validation is more straightforward. If one assumes that self-report is equally accurate in all arms of an intervention study, one can make a variety of assumptions regarding the cessation rates in non-respondents, those lost to follow-up, and those not undergoing validation. This then gives a range of outcomes, much as sensitivity analysis does in health economics. Our methodology gives a 'worst case' cessation rate, which in all probability is an underestimate of the true rate, since it is likely that there will be some quitters amongst those lost to follow-up. However, even this analysis produces a cost per quitter for the non-tailored letter that compares very favourably with other low intensity interventions. There is no doubt that validation is time consuming, expensive and has its own potential inaccuracies. It would be nice to dispense with it, but only if we can be reasonably confident in the validity of self-report. Perhaps the forthcoming publication from the recent consensus conference(9) which Prof. Strecher alludes to will include some further evidence or argument to clarify matters. We look forward to reading it. References 1. Strecher VJ. Computer-tailored smoking cessation materials: A review and discussion. Pat Educ and Counseling 1999;36:107-117. 2. Kreuter MW, Oswald DL, Bull FC, Clark EM. Are tailored health education materials always more effective than non- tailored materials? Health Educ Res 2000;15:305-315. 3. Reiter E, Robertson R, Osman L. Knowledge acquisition for natural language generation. In Proceedings of the First International Conference on Natural LanguageGeneration (INLG-2000), 217-224. Downloadable from http://www.csd.abdn.ac.uk/research/stop 4. Reiter E. Pipelines and size constraints. Computational Linguistics 2000;26:251-259. Downloadable from http://www.csd.abdn.ac.uk/research/stop 5. Patrick DL, Cheadle A, Thompson DC, Diehr P, Koepsill T et al. The validity of self-reported smoking: A review and meta-analysis. Am J Public Health 1994;84:1086-1093. 6. COMMIT Research Group. Community Intervention Trial for Smoking Cessation (COMMIT): I. Cohort results from a four-year community intervention. Am J Public Health 1995;85:183-192. 7. Velicer WF, Prochaska JO, Rossi JS, Snow MG. Assessing outcome in smoking cessation studies. Psychol Bull 1992;111:23-41. 8. Glasgow RE, Mullooly JP, Vogt TM, Stevens VJ, Lichtenstein E et al. Biochemical validation of smoking status in public health settings: pros, cons, and data from four low intensity intervention trials. Addict Behav 1992;18:504-527. 9. Benowitz NL, Jacob III P, Ahijevych K, Jarvis MJ, Hall S et al. Biochemical verification of tobacco use and cessation. Nic and Tob Res (in press).
General Comment on Tailoring 31 August 2001
Ehud Reiter, Lecturer Dept of Computing Science, University of Aberdeen Send response to journal: Re: General Comment on Tailoring	I was in charge of developing the tailoring software in the Aberdeen project, and I would like to make some general comments about tailoring. These are personal comments which supplement the reply sent by Dr. Lennox on behalf of our group to Prof. Strecher's remarks. The idea of computer-tailoring information based on some kind of "recipient model" has been tested in many other applications in addition to patient-information ones. The initial work in this field was done for computer-aided-learning systems, where the idea was to build up a model of what the student knew and did not know, and use this to tailor the information presented to him or her. Indeed some of the software used in early patient-information tailoring systems, such as Authorware, was developed for computer-aided-learning applications. Many other applications have been investigated; an interested reader may wish to browse through the User Modelling and User Adapted Interaction journal (http://umuai.informatik.uni-essen.de/). Probably the highest profile such system yet built (although very different from patient-information systems) is the "paper clip" help in Microsoft Office, which essentially attempts to build a model of what the user is attempting to do and what the user does and does not know, and suggest appropriate help topics (see http://research.microsoft.com/research/dtg/horvitz/lum.htm) Unfortunately, although there have been some successes, generally tailoring has been less effective than hoped, especially when its effectiveness has been carefully measured against a good non-tailored control. Perhaps one reason for this is the incredible complexity of human beings. If a computer system had complete knowledge about the recipient's knowledge, beliefs, intentions, preferences, and concerns, then perhaps tailoring could be very effective; but the sort of recipient models used in current tailoring systems fall far short of this. If we jump from the general to the specific and look at STOP, our computer system that produced tailored smoking-cessation letters, we do not know why STOP was ineffective. Indeed, it is possible that it was effective in some subgroups, such as heavy smokers, but the clinical trial lacked the power to show this. But my personal belief, which is largely based on qualitative discussions with smokers about the project, is that we didn't know enough about the smokers. Our questionnaire focused on general smoking habits, beliefs, and concerns. We knew nothing about specific experiences, and little about people's general interests, abilities, and preferences. In other words, we didn't know that W's mother had just died, that X had a difficult 2-year old, that Y had a degree in Chemistry, or that Z was French. A number of smokers have basically told me that STOP would have been more effective if it had had such knowledge, and I suspect (although I have no proof) that there is a lot of truth in this. A very similar point was made by Kreuter et al [1]. I believe more research needs to be done on such "broad spectrum" tailoring. For example, I currently have a PhD student investigating tailoring texts according to the recipients reading ability (http://www.csd.abdn.ac.uk/~swilliam/phd.html). It would be nice to see more such research on tailoring according to general recipient characteristics, and perhaps greater awareness in the tailored patient information community that "behavioural construct tailoring" ([1], page 313) is not the only possible kind of tailoring. [1] Kreuter MW, Oswald DL, Bull FC, Clark EM. Are tailored health education materials always more effective than non-tailored materials? Health Education Research 2000;15:305-315.
Re: Presentation of results is inappropriate 7 September 2001
Scott Lennox, Clinical senior lecturer, Senior lecturer in medical statistics, Senior research fellow University of Aberdeen, University of Dundee, University of Aberdeen, Peter Donnan, Liesl Osman Send response to journal: Re: Re: Presentation of results is inappropriate	The main hypothesis was that the tailored letter would have a higher cessation rate than no letter, as stated in the discussion, but the non- tailored arm was included because we believed from the start that a good non-tailored letter could have a greater effect than no letter. We refute the assertion that only the largest differences were presented: all comparisons were clearly stated in the results. We agree that multiple testing is important in statistical inference. However, we clearly stated that the differences between the arms just failed to reach statistical significance and so taking account of multiple testing would only make these more non-significant and is therefore a pointless exercise. The comparison of all letter interventions vs. non- letter is already presented in the results section. Whether or not people receiving different letters or no letter should differentially 'lie' when reporting cessation is not relevant to the results which are based only on reports of cessation supported by biochemical validation, and which count every non-validated self-report within the continuing smoker group. There was not enough space to report all the cost-effectiveness analyses. We agree that further work needs to be done in looking at sensitivity of the results to various assumptions. Once all the results are presented it is sensible to look at the cost-effectiveness of the intervention that had the largest impact. It is obvious that the tailored intervention would not be as cost-effective. The size of the cessation results is fully considered in the discussion. The differences may appear to be small but the implications for health and costs over a large population are not.
New correspondence email 15 October 2002
Ehud R Reiter, Lecturer in Computing Science University of Aberdeen, Aberdeen AB24 3UE, UK Send response to journal: Re: New correspondence email	Scott Lennox has left research, so inquiries about this paper should be sent to Liesl Osman (l.osman@abdn.ac.uk) or Ehud Reiter (ereiter@csd.abdn.ac.uk).