BMJ  2007;335:1290-1292 (22 December), doi:10.1136/bmj.39419.662998.BE

Mixed messages

Screening programme evaluation applied to airport security

¹ Department of Epidemiology, Harvard School of Public Health, Boston, MA 02115, USA, ² Department of Economics, Harvard University, Littauer Center, Cambridge, MA, USA, ³ J-Poverty Action Laboratory, Massachusetts Institute of Technology, Cambridge, MA 02139-4307, USA, ⁴ Prevention and Control, Siteman Cancer Center, Washington University School of Medicine, Campus Box 8109, St Louis, MO 63110, USA

Eleni Linos, Elizabeth Linos, and Graham Colditz investigate whether airport security screening would pass the National Screening Committee’s criteria for an effective screening test

Safety is paramount to travellers. Governments agree, and the airport operator BAA has spent £20m (28m; $41m) on airport security in the past year alone.¹ Add the $15bn that the government of the United States spent between 2001 and 2005 on aviation screening,² or the estimated $5.6bn that worldwide airport protection costs each year,³ and we reach one conclusion—airport screening is extremely costly. Yet on 30 July 2007, the head of the International Air Transport Association, Giovanni Bisignani, launched a scathing attack on airport security in the United Kingdom: he claimed that the UK’s "unique screening policies inconvenience passengers with no improvement in security."⁴

Complaints about the cost of airport security have flooded the news in recent months, but the problem is not new. The UK has seen a 150% increase in airport security costs since the terrorist attacks on 11 September 2001 and even steeper rises since the London bombings on 5 July 2005.⁵ With such high value attached to airport security, the details of efficacy, precision, and cost effectiveness of screening methods are easy to ignore. Protection at any cost is a reassuring maxim for us jetsetters. But preventing any death—whether from haemorrhagic stroke, malignant melanoma, or diabetic ketoacidosis—is surely an equally noble cause. In most such cases, screening programmes worldwide are closely evaluated and heavily regulated before implementation. Is airport security screening an exception?

Screening evaluated

The UK National Screening Committee’s remit is to assess screening technologies on the basis of sound scientific evidence and advise on whether they should be implemented, continued, or withdrawn.⁶ The table outlines the criteria used to evaluate screening programmes. These criteria include an important and treatable condition, an accurate and acceptable test, and sufficient evidence of benefit of the proposed screening project from randomised trials. To be considered for a screening programme, the condition must be common and of considerable burden to society. Furthermore, a "preclinical" phase must exist, during which the condition can be detected and treated. Cervical cancer is a classic example—although morbidity and mortality are high worldwide, if detected early, premalignant lesions can be cured. The criteria also mandate that a suitable screening test should be simple, safe, and validated. For example, cholesterol monitoring—used to screen for hyperlipidaemia and prevent its complications—fits these requirements. It is acceptable to the population, it has well defined cut-off values, and the benefit of treatment is established, making it an excellent screening test. Yet things are rarely this straightforward, and for most screening programmes we rely on scientific evidence to show efficacy and effectiveness, cost-benefit balance, and acceptability.

View this table: [in this window] [in a new window]   National Screening Committee criteria for implementation of screening programmes

 
Discussion on whether screening programmes should be implemented inevitably centres on at least one of these key criteria. For example, recent debates on cervical screening have focused on the test—namely, the sensitivity and predictive value of testing for human papillomavirus⁷ or liquid based cytology⁸ compared with conventional cervical smears. For lung cancer screening the sticking point has been the quality of the evidence showing that computed tomography screening improves overall mortality.⁹¹⁰¹¹ A similar debate for prostate specific antigen testing remains unresolved.

We examine whether airport security screening is an acceptable screening programme—is the evidence sufficient to meet the National Screening Committee’s criteria? We then identify points of future research that could encourage a more rigorous evaluation of airline security measures.

Airport security

The "disease" and its treatment
Presumably, one of the negative outcomes or "diseases" we are trying to prevent is injury to passengers or crew as a result of in-flight terrorist attacks. The time between arriving at the airport and boarding the plane is the latent period during which dangerous objects can be detected and attacks prevented by confiscation, explosive disarmament, or arrest. These are analogous to the condition, preclinical phase, and treatment—so, far so good. But although any potential threat to the safety of passengers is a noteworthy cause worth fighting against, such events are extremely rare.

Since 1969, only 2000 people have died as a result of explosives on planes, yet the US department of homeland security spends more than $500m annually on research and development of programmes to detect explosives at airports.¹² Even the devastating 11 September 2001 attacks caused around 3000 deaths, which is similar to the number of deaths attributed to high blood glucose each day¹³ or the number of children dying of the human immunodeficiency virus every three days worldwide.¹⁴ The publicity awarded to such terrorist attacks is so high that the perceived threat is far higher than the numbers suggest. Furthermore, the cost of airport security ($9 per passenger) is 1000 times higher than for railway security ($0.01 per passenger), even though the number of attacks on trains is similar to that in planes.¹⁵ This is analogous to committing mammography resources to screening only the left breast, and ignoring the right side, even though cancer can affect both breasts.

The tests and evidence of benefit
We systematically reviewed the literature on airport security screening tools. A systematic search of PubMed, Embase, ISI Web of Science, Lexis, Nexis, JSTOR, and Academic Search Premier (EBSCOhost) found no comprehensive studies that evaluated the effectiveness of x ray screening of passengers or hand luggage, screening with metal detectors, or screening to detect explosives. When research teams requested such information from the US Transportation Security Administration they were told that evaluating new screening programmes might be useful, but it was overshadowed by "time pressures to implement needed security measures quickly."¹⁶ In addition, we noticed that new airport screening protocols were implemented immediately after news reports of terror threats (fig 1).

View larger version (26K): [in this window] [in a new window] [PowerPoint Slide for Teaching]   Fig 1 Timeline of changes to airport screening protocols, costs, and news events related to terrorist threats

 
The little we do know about airport security screening comes from investigations of the factors that influence the sensitivity of visual screening of x ray images. These studies conclude that sensitivity depends on the screener’s experience, rather than the precision of the machine. Practice improves the screener’s performance, but unfamiliar or rare objects are hard to identify regardless of experience.¹⁷¹⁸¹⁹ Mammography radiologists realise this and undergo years of specialised training after medical school.²⁰

Even without clear evidence of the accuracy of testing, the Transportation Security Administration defended its measures by reporting that more than 13 million prohibited items were intercepted in one year.²¹ Most of these illegal items were lighters. The screening literature shows that length time and lead time bias produce misleading interpretations of screening studies because of earlier detection of more benign cases that would not necessarily become clinically apparent (overdiagnosis). A similar problem arises with the above reasoning—although more than a million knives were seized in 2006, we do not know how many would have led to serious harm.

The questions

The absence of scientific evaluations of the screening tools currently in place and the vast amount of money spent by governments worldwide on airport security have led us to muse over current airport security protocols and wonder about their optimal implementation. What is the sensitivity of the screening question, "Did you pack all your bags yourself?" and has anyone ever said no? Can you hide anything in your shoes that you cannot hide in your underwear? What are the ethical implications of preselecting high risk groups? Are new technologies that "see" through clothes acceptable? What hazards should we screen for? Guns and explosives certainly, but what about radioactive materials or infectious pathogens? Concerns about cost effectiveness—including the indirect costs of passengers’ time spent in long queues—will be central to future decisions, but first we need solid evidence of benefit.

An experiment

If we were to evaluate the effectiveness of airport screening, we would start by assessing the accuracy of current tests for illegal objects in passengers’ luggage. This would yield only preliminary information on screening test performance; we would need to reapply for funding to evaluate the overall benefit of security screening on mortality and calculate the number needed to screen to prevent the death of one traveller.²² After informing the airport managers, gaining approval from research ethics committees and police, and registering our trial with one of the acceptable International Committee of Medical Journal Editors trial registries, we would select passengers at random at the check-in desks and give each traveller a small wrapped package to put in their carry-on bags. (We would do this after they have answered the question about anyone interfering with their luggage.) A total of 600 passengers would be randomised to receive a package, containing a 200 ml bottle of a non-explosive liquid, a knife, or a bag of sand of similar weight (control package) in a 1:1:1 ratio. Investigators and passengers would be blinded to the contents of the package. Our undercover investigators would measure how long it takes to get through security queues and record how many of the tagged customers are stopped and how many get through. A passenger who is stopped and asked to open the wrapped box would be classed as a positive test result, and any unopened boxes would be considered a negative test result. We would use the number of true and false positives and true and false negatives to estimate the sensitivity and specificity of the current screening process and pool the waiting times to estimate an average waiting time for each passenger (fig 2).

View larger version (63K): [in this window] [in a new window] [PowerPoint Slide for Teaching]   Fig 2 Study design flow chart for evaluation of current screening test for hand luggage

 
We have heard rumours that this sort of thing actually goes on—that agents occasionally carry illicit items through airport screening units to "test" them and identify gaps in security. Perhaps the evidence we are searching for is strong, but secret. And of course rigorous airport screening may have other benefits. It certainly deters the transport of any illicit object, such as less dangerous but equally unwanted plants, animals, or drugs. In addition, in the midst of mounting reports of thwarted terrorist attacks on airports, the process is comforting to frequent flyers and their families. Nevertheless, the absence of publicly available evidence to satisfy even the most basic criteria of a good screening programme concerns us.

Conclusion

Of course, we are not proposing that money spent on unconfirmed but politically comforting efforts to identify and seize water bottles and skin moisturisers should be diverted to research on cancer or malaria vaccines. But what would the National Screening Committee recommend on airport screening? Like mammography in the 1980s, or prostate specific antigen testing and computer tomography for detecting lung cancer more recently, we would like to open airport security screening to public and academic debate. Rigorously evaluating the current system is just the first step to building a future airport security programme that is more user friendly and cost effective, and that ultimately protects passengers from realistic threats.

---

Thanks to Lorelei Mucci, Monica McGrath, Mike Stoto, and Pat Cox for useful discussions.

Contributors and sources: Eleni L and GC conceived and designed the study. All authors helped collect data and write and edit the manuscript. Eleni L is guarantor. GC has worked extensively on breast and colorectal cancer screening and advises the American Cancer Society on implementation of screening programmes.

Funding: NIH grant R25 CA098566 provided salary support for Eleni L. The funder had no role in study design; in the collection, analysis, and interpretation of data; in the writing of the report; and in the decision to submit the article for publication.

Competing interests: None declared.

Provenance and peer review: Not commissioned; externally peer reviewed.

References

1. BBC News . Airline body attacks UK security. 2007 July 30. http://news.bbc.co.uk/1/hi/uk/6922992.stm.
2. Lipton W. US to spend billions more to alter security systems. New York Times 2005 May 8.
3. International Air Transport Association. The air transport industry since 11 September 2001. www.iata.org/NR/rdonlyres/92FC0755-1D63-4931-A983-847CC1EA897A/0/airtransportsince911.pdf.
4. International Air Transport Association Press Release . Half year traffic results: passenger growth strong, cargo sluggish. 2007. www.iata.org/pressroom/pr/2007-07-30-01.
5. BBC news. Airport security costs "too high." 2007 Jul 14. http://news.bbc.co.uk/1/hi/uk/6898576.stm.
6. Health Departments of the United Kingdom. First report of the national screening committee. 1998. www.nsc.nhs.uk/pdfs/nsc_firstreport.pdf.
7. Runowicz CD. Molecular screening for cervical cancer—time to give up Pap tests? N Engl J Med 2007;357:1650-3.[Free Full Text]
8. Denton KJ. Liquid based cytology in cervical cancer screening. BMJ 2007;335(7609):1-2.[Free Full Text]
9. Bach PB, Jett JR, Pastorino U, Tockman MS, Swensen SJ, Begg CB. Computed tomography screening and lung cancer outcomes. JAMA 2007;297:953-61.[Abstract/Free Full Text]
10. Henschke CI, Yankelevitz DF, Libby DM, Pasmantier MW, Smith JP, Miettinen OS. Survival of patients with stage I lung cancer detected on CT screening. N Engl J Med 2006;355:1763-71.[Abstract/Free Full Text]
11. Black WC, Baron JA. CT screening for lung cancer: spiraling into confusion? JAMA 2007;297:995-7.[Free Full Text]
12. Ripley A. How much risk are we willing to take? Time 2006 Aug 21. www.time.com/time/magazine/article/0,9171,1226170,00.html.
13. Danaei G, Lawes CM, Vander Hoorn S, Murray CJ, Ezzati M. Global and regional mortality from ischaemic heart disease and stroke attributable to higher-than-optimum blood glucose concentration: comparative risk assessment. Lancet 2006;368:1651-9.[CrossRef][Medline]
14. WHO. Report on the global AIDS epidemic. 2006. www.unaids.org/en/HIV_data/2006GlobalReport/default.asp.
15. Tolman J. Hearing on transit security training procedures. WA: United States House of Representatives Committee on Homeland Security, 2006.
16. US Government Accountability Office. aviation security: risk, experience, and customer concerns drive changes to airline passenger screening procedures, but evaluation and documentation of proposed changes could be improved. 2007. www.gao.gov/docsearch/abstract.php?rptno=GAO-07-634.
17. Smith JD, Redford JS, Washburn DA, Taglialatela LA. Specific-token effects in screening tasks: possible implications for aviation security. J Exp Psychol Learn Mem Cogn 2005;31:1171-85.[CrossRef][Web of Science][Medline]
18. McCarley JS, Kramer AF, Wickens CD, Vidoni ED, Boot WR. Visual skills in airport-security screening. Psychol Sci 2004;15:302-6.[CrossRef][Web of Science][Medline]
19. Wolfe JM, Horowitz TS, Kenner NM. Cognitive psychology: rare items often missed in visual searches. Nature 2005;435:439-40.[CrossRef][Medline]
20. Esserman L, Cowley H, Eberle C, Kirkpatrick A, Chang S, Berbaum K, et al. Improving the accuracy of mammography: volume and outcome relationships. J Natl Cancer Inst 2002;94:369-75.[Abstract/Free Full Text]
21. Transportation Security Administration Screening Statistics. Facts and figures for 2006. www.tsa.gov/research/screening_statistics.shtm.
22. Rembold CM. Number needed to screen: development of a statistic for disease screening. BMJ 1998;317:307-12.[Abstract/Free Full Text]

CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    StumbleUpon    Technorati    What's this?

Rapid Responses:

Read all Rapid Responses

A Serious Note

Peter A West
bmj.com, 23 Dec 2007 [Full text]

The cost of a "negative test"

Ganesan Karthikeyan
bmj.com, 26 Dec 2007 [Full text]

Dates

Robert J Shaw
bmj.com, 2 Jan 2008 [Full text]

Online poll

Find out more >>