Click here to access the corrected complete pdf version.

Opening the black box of clinical judgment

Part III: Medical Science and Education

Lawrence L Weed, Lincoln Weed

Thus far we have focused on reforms of patient care itself. Next we turn to the effect of those reforms on the development of new medical knowledge, and the related issue of how knowledge is perceived by its users. That discussion leads to an analysis of medical education.

A. Harvesting new medical knowledge from routine patient care

The new tools and approaches needed to better apply existing medical knowledge are also key to harvesting new medical knowledge from routine patient care. For example, known disease conditions that now go unrecognized when their expected "classic" manifestations do not appear may become much more recognizable, based on new patterns revealed in detailed medical records of detailed findings on thousands of patients.

Such analysis of disease and clinical outcomes, however, is unreliable unless inputs are controlled. For example, poor execution of a physical examination or other diagnostic procedures can distort prevalence data for diagnoses that the procedures are intended to identify. Reliable findings can be assured only if the medical personnel who make them are subject to rigorous, ongoing credentialing. Moreover, the findings must be comprehensive and carefully selected, which can be achieved only if medical personnel are guided by information tools rather than ad hoc judgments in determining which findings to check. In addition, findings must be reliably documented in a problem-oriented manner so that that their logical context is preserved. Controlling inputs in these ways to prevent failures of quality would permit scientifically rigorous study and feedback on existing knowledge as an automatic byproduct of routine care to a much greater degree than is now possible. (1)

At present, expensive, time-consuming clinical trials are undertaken in part because uncontrolled inputs and poor record keeping practices limit the utility of routine care as a source of data for research. Yet, clinical trials and other outcome studies often fail to account for and control all of the relevant variables and alternatives. For example, judging the effectiveness of an invasive or costly diagnostic procedure depends on the alternative for comparison. If the alternative is doctors' diagnostic judgments based on making a series of simple clinical observations, the effectiveness and efficiency of that alternative turns on what observations are selected, the doctors' skill and consistency in making them (recall the study we cited in part II showing a low skill levels in listening to heart sounds (2)), and the doctors' reliability in recording the observations, linking them and recognizing their significance. All of those inputs are subject to radical improvement with use of proper information tools and quality controls.

Similarly, comparisons of treatment outcomes require controlling quality of execution of the alternative treatments being compared, clear definition and consistency in determining the indications for treatment, and accounting for individual differences in the severity of disease (3) and other relevant patient characteristics. These requirements are often not satisfied. A recent study of randomized, "controlled" clinical trials of interventions for shoulder pain, found "little evidence to support the use of any of the common interventions," "no uniformity in the way shoulder disorders are labeled or defined," wide variation in outcome measures, and insufficient basis to determine the reliability, validity and responsiveness of these outcome measures. (4) Here again, proper information tools, medical records and other quality controls can bring about radical improvements. Knowledge coupling software, for example, automatically brings about consistency in definition and selection of relevant findings.

New tools and approaches facilitating the development of medical knowledge can also alter perceptions of existing medical knowledge. The microscope, for example, by extending the power of the eye, not only expanded scientific knowledge but revealed error in existing "knowledge." Similarly, knowledge coupling tools, by extending the power of the mind, can make everyone in medicine more sensitive to patient uniqueness and the fallibility of textbook disease classifications. Medical knowledge tells us of the limited elements that individual patients have in common — recurrent clusters of findings (to which a disease label may be assigned), causation (which may permit redefining the disease or identifying effective treatment), clinically significant variables (which may be useful in assessing disease or treatment). At the same time, because of patient uniqueness and complexity, a degree of arbitrariness is inherent in any disease classification and in any attribution of a disease to a single cause. Indeed, the very concepts of the "same disease" or a single cause or a "standard" treatment for different individuals, although necessary to organize our knowledge, are at the same time artificial and limiting. Grouping phenomena on the basis of a few characteristics is inevitably misleading when additional characteristics have relevance. The decision making context and the purposes of the parties involved determine the significance of commonalities or differences. (5)

Capturing those differences requires not only knowledge coupling tools but also sound medical records. Conventional medical records highlight a "chief complaint" and organize data according to its source. Yet, focusing on a chief complaint may lead to overlooking other, equally or more important problems, and source-oriented data are difficult to locate and interpret because their logical context is lost. In contrast, sound medical records highlight a complete problem list and organize data around it. Such records facilitate grasp of complex data and provide an escape from the tyranny of a simple diagnostic label. Seeing the total patient in this way is vital for both patient care and medical science. As Ian Lawson wrote 25 years ago concerning patients with multiple problems:

The interrelations of problems are as important as the individual problems themselves. Symptoms and problem profiles, rather than summary diagnostic labels, often prove more sensitive in therapeutic management and may eventually lead to a different kind of care organization and epidemiology. [Recognizing this point will] create immediate conflicts with 'third party' agents and their prototype definitions of 'eligible' illness. Indeed, the sooner their computer experience gets wise to (or gets "blown" by) the realities of multiproblem interrelational analysis and management, the better for all. (6)

These concepts are vital in assessing quality of care and clinical outcomes. Patient uniqueness means that the inputs required from caregivers may vary immensely for patients with the "same disease." Moreover, high quality care cannot be defined in terms of favorable outcomes, because caregivers do not control many of the inputs (patient behaviors, environmental conditions) that determine outcomes. Thus, high quality care is the total sequence of competently performed and logically linked steps within a system that clearly defines the inputs within caregivers' control. The steps required will vary for every patient. Variation among caregivers is entirely appropriate when it reflects patient uniqueness. What is not appropriate is variation arising from the idiosyncracies of caregivers themselves (a phenomenon all too apparent to patients who go to different doctors for the same problem and find wide variation in how those doctors proceed). When such variation is brought under control, other dimensions of quality of care and clinical outcomes can be analyzed and improved.

To reiterate, doing so depends on employing adequate software tools and medical records. Recall our discussion in part I of Francis Bacon's admonition: "The sole cause and root of almost every defect in the sciences is this, that while we falsely admire and extol the powers of the human mind, we do not search for its real helps." (7) Bacon was writing in reaction against academic and ecclesiastical adherence to Aristotelian dogma, which he saw as a barrier to inductive scientific inquiry. He was acutely aware of the pitfalls of abstract thought divorced from observation. The learning from observation by those engaged in commercial and practical activities enormously impressed Bacon. He also witnessed a flowering of intellectual life outside of the universities. What he saw led him to reject the sterile, scholastic disputation taught by ecclesiastical and university authorities. Science and practical learning he viewed as cumulative, collaborative activities that enable escape from the limits of received authority and the individual mind. (8)

In the present scientific age, we think of ourselves as having achieved Bacon’s ideal. But in medical science we have fallen short. Researchers and practitioners have an unwarranted confidence in established knowledge and their own perceptions. They need new tools used routinely in order to capture the reality of patient uniqueness — the constant divergence from the generalizations of medical knowledge.

Inattention to such divergence reflects perhaps a larger phenomenon that Stephen Jay Gould has observed. "Our culture," he argues, "encodes a strong bias either to neglect or ignore variation. We tend to focus instead on measures of central tendency, and as a result we make some terrible mistakes, often of considerable practical import." As an example he cites his own medical history. At age 40 he was "diagnosed with abdominal mesothelioma, a rare and 'invariably fatal' form of cancer (to cite all official judgments at the time)." But his scientific background "had taught me to treat variation as a basic reality, and to be wary of averages — which are, after all, abstract measures applicable to no single person, and often largely irrelevant to single cases." He thus recognized the "need to place myself in the most probable region of the variation based on particulars of my own case," and he realized that the statistics (eight months median survival time after diagnosis) represented "an abstraction with no special relevance to my own case." He then surmised (and the data confirmed) that the distribution of deaths was "right skewed," with a few people achieving long-term survival. He concluded that his own characteristics and circumstances gave him a reasonable chance of more than short-term survival. Believing that he need not prepare for imminent death, he turned instead to fighting the disease. Ultimately, an experimental treatment turned out for him to be successful. (9)

Such cases of terminal illness give patients and doctors the greatest possible motive to look behind statistical information, and also give them the greatest opportunity to do so, because detailed underlying data, and stories about individual variation, are more likely to be available. Yet, even in such cases the generalizations of medical knowledge may distort decisions. And in more routine cases such distortion is endemic. Doctors are "educated" with fallible generalizations, and when they encounter individual variation they naturally resort to unaided judgment. Aware that such experience over time informs their judgment, doctors naturally lose sight of the fact that their experience may distort judgment as well, because the range of variation that any doctor personally encounters is limited. For these reasons, doctors are slow to see the need for habitual use of comprehensive data gathering and information tools to process that data. They regard those measures as fallbacks for the occasional puzzling case, trusting their judgment and limited data to lead them to correct decisions in most cases. Such a belief reflects insufficient awareness of their own cognitive limitations. Using sound information tools on a discretionary, judgmental basis is analogous to an astronomer's using a telescope only when he sees something unusual in the sky. In contrast, using the necessary tools routinely can change perceptions.

A crucial advance becomes possible when ordinary caregivers are sensitive to patient uniqueness and employ the necessary tools. They will then naturally compare their patients with the medical literature. Noticing variations, they will become a constant source of feedback, through communication with software builders and researchers, on existing medical knowledge. Equally important, ordinary caregivers using knowledge coupling software and rigorous medical records will habitually collect structured clinical data and record it in maximally usable form. Refined outcome studies can thus be conducted as a byproduct of patient care. In these ways, ordinary caregivers and ordinary patient care, far more than is now the case, can become part of a cumulative, collaborative enterprise to advance medical knowledge.

In short, lack of controlled inputs harms the use, understanding and development of medical knowledge for patient care. These realities call into question the current model of graduate medical education.

B. Implications for medical education and credentialing

Medical education began to assume its current form at the turn of the century, when the Flexner report advocated the Johns Hopkins post-graduate model, founded in basic science. Paul Starr has described the reform perspective at that time (10): "As Flexner saw it, a great discrepancy had opened up between medical science and medical education. While science had progressed, education had lagged behind. 'Society reaps at this moment but a small fraction of the advantage which current knowledge has the power to confer.'" (11)

Ever since Flexner's report, the enormous time and expense consumed by graduate medical education have been viewed as necessary for bringing scientific standards to medical practice. The reality, however, is that medicine falls short of scientific standards in both education and practice. Moreover, medicine is not only a scientific endeavor but also a commercial enterprise subject to time constraints and productivity demands very different from academic science. Yet medical education and practice fall short of commercial standards of both productivity and quality. Finally, medicine is a humanitarian activity with high standards of compassionate and ethical behavior, yet by those standards too it falls short. These problems are systemic, and they call into question the core premises of graduate medical education.

Consider first the humanitarian perspective. If physicians' first ethical obligation is to "do no harm," then medical errors resulting in harm constitute breaches of ethical duty. Given the levels of error that have been documented in medicine, quality is a major ethical issue. Equally important, the underlying systemic flaws that lead to error also undermine the humanitarian impulses of doctors and other caregivers. We observed in Part I that financial gain is more readily achievable than high quality because quality is unmanageable in the present system. The result is that physicians tend to act on incentives for financial gain by default. The extent to which the economic interests of doctors influence medical decision making became apparent in recent decades with studies showing an epidemic of unnecessary care by fee-for-service physicians in the U.S. The advent of managed care and capitated compensation has further exposed the susceptibility of medical decision making to questionable economic influences. In light of this evidence, graduate medical education can only be regarded as unsuccessful in its central duty — equipping practitioners to live up to the medical profession's humanitarian ideals. The point is not that physicians have lower ethical standards than other professionals. On the contrary, the public legitimately regards physicians as more trustworthy than most professionals. The point is that this trust is largely an individual achievement, resulting from personal standards of behavior and personal relationships with patients. This system within which physicians are educated to function is not worthy of patients' trust.

Turning from humanitarian to scientific standards, here again we find that graduate medical education falls short of what medicine requires. Consider the beginning medical school curriculum. Faculties overload students with abstract knowledge — textbook answers to questions they have not asked about observations they have not made. Learning of this kind is the antithesis of scientific inquiry. Students who undergo this process can easily become doctors who "quote what is in the book and deny what is in the bed." A number of studies, for example, have documented the phenomenon of students who unconsciously "fabricate" findings in patient examinations, perhaps because the findings "are consistent with their understanding of the disease believed to exist or because they are consistent with the 'classic presentation' of the disease felt to be most likely." Moreover, this and other cognitive errors resist corrective measures; a recent study of medical education "demonstrates that even an active-learning intervention designed to enhance clinical skills, including diagnosis, did not reduce the likelihood of errors in reasoning." (12)

After the beginning curriculum, medical students are thrust into clinical settings, in the hope that they will somehow learn to apply their abstract knowledge to real patients effectively while mastering a broad range of manual skills. This hope is unfounded because graduate medical education does not assure learning conditions — manageable scope, the opportunity for single-minded attention, careful progression from simple to complex tasks, close feedback, an individualized program  that are fundamental for students to mobilize enthusiasm and develop mastery. Learning tends to happen on a "sink or swim" basis; that is, students learn to a great extent on their own, with less structure and less organized feedback than in their formal education.

The immense scope of what students are expected to learn is especially relevant to the domain of knowledge and decision making. Medical schools operate on the premise that a necessary "core of knowledge" can be identified and taught as an indispensable basis for later specialization. It is believed that the core of knowledge must be broad in scope and that gifted, well-schooled medical students have sufficient cognitive ability to learn it. Yet, that core of knowledge varies from one institution to another, medical students do not learn all they are taught at each institution, they retain only part of what they learn, that residue varies with each individual, and some of that residue quickly becomes obsolete. Extensions of graduate medical education — board certification, continuing education courses, individual study of the medical literature — are futile attempts to paper over the mind’s inherent limitations as a tool for storage and retrieval of medical knowledge. (13) Formal education is therefore ineffective and inefficient even for the limited function of transmitting general knowledge. Moreover, even if that function could be carried out effectively; coupling general knowledge with patient data in complex cases involves too much detail for the unaided mind to handle reliably under the time constraints of medical practice. Not surprisingly, therefore, studies have "questioned the correlation of superior knowledge retention to professional performance, suggesting that an individual’s ability to ‘bring order to the informational chaos that characterizes one’s everyday environment’ (14) determines whether that professional continues to perform competently." (15)

Operating on the belief that students must be indoctrinated with received knowledge as preparation for real patient care, medical schools trap student minds in what Tolstoy called "the snare of preparation." Like a drug, such education has toxicity as well as benefit. By teaching students to assume the Sisyphean burden of acquiring a core of knowledge, medical education inhibits students from attaining that which practitioners truly need — mastery of skills and recognition that their acquired knowledge falls far short of what patient care demands. The effect is to reinforce a basic human need to deny uncertainty. Dr. Jay Katz has described "how readily any awareness of uncertainty succumbs to venerable authority and orthodoxy. These powerful defenses against awareness of uncertainty continue to rule professional practices." (16) Sociologist Robert Weaver has further described findings in the literature on this phenomenon:

A major task undertaken during medical training is learning to manage the uncertainty associated with medicine and medical education. For instance, medical students learn the disadvantages of "doubting too much" and displaying these doubts to peers, superiors, and patients. Instead, they often develop a misleading sense of certitude or come to don a "cloak of competence" to help them manage the impressions of others and, ultimately, the image they have of themselves. Confidence and belief in what one is doing is a central component of the "clinical mentality" as Friedson describes it. Doubts about the ambiguities of "unusual" cases, even when acknowledged by the practitioner, are often "silenced" or otherwise not shared with the patient. (17)

Doctors emerge from this process with insufficient sensitivity to patient uniqueness and the fallibility of medical knowledge. We must educate a new kind of caregiver, resistant to the generalizations and misconceptions of his teachers, equipped with scientific habits of rigor and independent inquiry. The goal of medical education must be to impart skills and behaviors rather than knowledge. To do so, the only workable mode of education is careful engagement of students in patient care itself, using knowledge rather than learning it in the abstract, employing information tools rather than erudition. If the worlds of action and knowledge do not connect easily and securely in this way, then good students become cynical and distrustful rather than fully engaged. (18)

The premises of medical education, the legal authority it confers to act upon unaided judgment, and the financial and social rewards for doing so, tend to reinforce basic traits of human nature — faith in one's own cognitions and insensitivity to one’s own ignorance — that undermine scientific rigor in medical practice. Francis Bacon long ago observed the tension between science and the mind's normal mode of operation: "The human understanding, when any propostion has been once laid down, . . . forces everything else to add fresh support and confirmation . . . rather than sacrifice the authority of its first conclusions." (19)

Although medical school faculties, students and practitioners try to overcome this basic human trait, their attempts inevitably fall short of what properly designed software tools and medical records can achieve. With the rigorous combinatorial analysis that those devices facilitate, the realities of individual patients generate rapid, organized, cumulative feedback on the hypotheses of practitioners and the generalizations of medical knowledge. Such feedback represents a superior medical education for all caregivers. (18) "By contrast, our present educational premises and overuse of statistical thinking tend to confirm and buttress past notions, right or wrong. Above all they stifle progress toward expecting and dealing honestly with the ultimate uniqueness of each patient." (20)

Turning from scientific standards of intellectual behavior to commercial standards of quality and productivity, we still find that graduate medical education falls short. Quality failures are pervasive. The Addison’s disease case study discussed in part I states that when doctors fail to recognize the correct diagnosis for seriously ill patients, "vastly excessive testing and numerous attempts to treat putative diagnoses are the rule" (emphasis added). According to the 1997 meta-analysis prepared for the National Coalition on Health Care that we cited in part I, "most care that has been studied" is deficient, regardless of the type of care, its location, its funding, or the nature of the patient population. (21) The Harvard Medical Practice Study, using an extremely conservative definition of medical error, found that 3.7 percent of the medical records studied reflected error, a far higher rate than what is tolerated in other industries. (22) Moreover, different studies using different methodologies and showing error rates up to ten times as high suggest that the Harvard study may seriously understate the extent of error. (23) And medical advances, in the form of more information to comprehend and more complex interventions to risk, paradoxically increase the potential for error, absent organized systems to prevent it. (24)

Equally significant are failures of productivity. These failures arise from a disconnect between the sheltered, academic world of medical science and the conditions of medical practice. A scientist chooses one or a few problems to research at a time, carefully designs experiments, controls the variables and executes experiments with whatever degree of time and care are necessary to achieve verifiable results. Those conditions are alien to medical practice, where patient needs determine what the doctor must consider, understanding a single problem in a single patient may require monitoring and analysis of multiple variables over time, patients usually have multiple problems, the doctor is responsible for dozens of patients in the hospital and hundreds of patients in the ambulatory setting. Patients must be cared for as they appear, and emergencies must be handled as they arise. Yet, clinicians are not educated or equipped to function effectively in these conditions. This contrasts sharply with many commercial workers who function as parts of carefully controlled, organized, coordinated systems where high productivity and quality are economic necessities. Medical education must better account for the fundamentally different needs presented by research and development functions on the one hand, and production or delivery functions on the other.

Prevailing views and conduct in the health care system ignore these realities. The rituals of lectures and examinations still occur in medical schools, board certification is still taken seriously as a credential, and practicing physicians still rely heavily on their own mental capacities for information retrieval and processing. Moreover, it is not clear that medical schools on their own will ever transform their approaches in the domains of either skill or knowledge, because to do so would diminish the exclusivity conferred by graduate medical education. As Atul Gawande has written, for example, developing high skill levels involves "superspecialization [that] raises the question of whether the best medical care requires fully trained doctors." Dr. Gawande observed Canadian surgical teams specializing exclusively in hernia repairs. "None of the three surgeons I watched . . . would even have been in a position to conduct their own procedures in a typical American hospital, for none had completed general surgery training. . . . Yet after apprenticing for a year or so they were the best hernia surgeons in the world." (25)

In the domain of knowledge, other fields are ahead of medicine in taking advantage of modern information tools to bypass traditional forms of education. (26) (27) Medicine is bound to a university community that has never faced up to the problem of distributing the fruits of intellectual wealth. Universities concern themselves with "the advancement of knowledge through research, the transmission of knowledge through teaching, the preservation of knowledge in scholarly collections and the diffusion of knowledge through publishing." (28) Teaching and publishing ill-equip caregivers to integrate medical knowledge with individual patient needs. Universities thus poorly serve the masses of people who depend on skillful and efficient use of knowledge. It can still be said, as Flexner wrote in 1910 of the gap between medical education and medical science: "Society reaps at this moment but a small fraction of the advantage which current knowledge has the power to confer." If progress is ever to be achieved, new devices for accessing knowledge and new modes of education must be put to use. (5) (18) (29) (30) (31)

Key messages

• The new tools and approaches needed to better apply existing medical knowledge are also key to harvesting new medical knowledge from routine patient care.

• Analysis of disease and clinical outcomes is unreliable unless inputs are well-defined, appropriate and consistently used.

• New tools and approaches to assure the quality of cognitive inputs can alter how we perceive and apply existing medical knowledge, by making us more sensitive to patient uniqueness and the fallibility of textbook descriptions of disease and treatment.

• Medicine is at once a scientific, a commercial and a humanitarian endeavor, and in each of those dimensions, high standards must be enforced.

• Systemic failures to achieve these high standards expose fundamental flaws in medical education and credentialing.

• By teaching students to assume the Sisyphean burden of mastering knowledge, medical education interferes with mastery of skill and instills false ideals of individual knowledge and professional autonomy.

 

  1. Kozak A. Local Clinicians Need Knowledge Tools. Am Psychol 1996.51:1335-1336.
  2. Mangione S, Nieman L. Cardiac auscultatory skills of internal medicine and family practice trainees: a comparison of diagnostic proficiency. JAMA 1997;278:717-722.
  3. Davenport R, Dennis M, Warlow R. Effect of correcting outcome data for case mix; an example from stroke medicine. BMJ 1996;312:1503-1505.
  4. Green S, Buchbinder R, Glazier R, Forbes A. Systematic review of randomised controlled trials of interventions for painful shoulder: selection criteria, outcome assessment, and efficacy. BMJ 1998;316:354-360.
  5. Weed LL. Knowledge coupling: new premises and new tools for medical care and education. New York: Springer-Verlag, 1991. p. 184-89.
  6. Lawson I. Comments on the POMR. In Driggs MF ed. Problem-directed and medical information systems. New York: Inter-Continental Medical Book Corporation, 1974.
  7. Bacon F. Novum Organum: Aphorisms Concerning the Interpretation of Nature and the Kingdom of Man, First Book, para. 9 (1620) (Montague trans.).
  8. Kors A. The origin of the modern mind. Springfield (VA): The Teaching Company, 1996 [recorded lectures].
  9. Gould S. Full house: the spread of excellence from plato to darwin. New York: Harmony Books (Crown Publishers), 1996. p. 44-51.
  10. Starr P., The social transformation of American medicine. New York, Basic Books, 1982. p. 120.
  11. Quoting Abraham Flexner, Medical education in the United States and Canada, Bulletin No. 4, New York, Carnegie Foundation for the Advancement of Teaching, 1910.
  12. Friedman M, Connell K, Olthoff A, Sinacore J, Bordage G. Thinking about student thinking: medical student errors in making a diagnosis. Acad Med. 1998;73(No. 10/Oct. Supp):S19-S21.
  13. Davis D, Thomson M, Oxman A, Haynes B. Changing physician performance: a systematic review of the effect of continuing medical education strategies. JAMA 1995;274:705-705.
  14. Pottinger.Competence testing as a basis for licensing: problems and prospects, Washington, D.C.: National Center for the Study of Professions, 1977.
  15. Pew Health Professions Commission, Taskforce on Workforce Regulation. Reforming Health Care Workforce Regulation. San Francisco: The Commission, 1995. p. 26
  16. Katz J. The silent world of doctor and patient. New York: Free Press - Macmillan, 1984. p. 179.
  17. Weaver R. Clinical uncertainty, responsibility and change in medical practice: a sociological response to Weed's knowledge coupling innovation. 1997. [Unpublished; citations omitted]..
  18. Weed LL. A touchstone for medical education. Harvard Medical Alumni Bulletin. 1974 (Nov./Dec.):13-18.
  19. Bacon F. ibid. note 7, para. 46.
  20. Weed CC. Philosophy, interpretation and use of problem-knowledge couplers [unpublished monograph]. Burlington, Vermont, PKC Corporation, 1982-98. Available from: URL: http://www.pkc.com.
  21. Shuster M, McGlynn E, Brook R. Why the quality of U.S. health care must be improved. Washington, D.C : National Coalition on Health Care, 1997. p. 13-14.
  22. Leape L. Error in Medicine. JAMA 1994; 272:1851-1857.
  23. Andrews A, Stocking C, Krizek T, Gottlieb L., Krizek C, Vargish T, Siegler M. An alternative strategy for studying adverse events in medical care. Lancet 1994; 309-313.
  24. Robin E. Medical care can be dangerous to your health New York: Harper & Row, 1984. p. 100.
  25. Gawande A. No Mistake. The New Yorker 1998 March 30; p. 111-116.
  26. University of Georgia, College of Education. EPSS Resources Home Page. http://itech1.coe.uga.edu/EPSS/EPSS.html.
  27. Longman P. The janitor stole my job: new software is expanding competition for white-collar jobs, U.S. News and World Report 1997 Dec. 1; p. 50.
  28. Pelikan J. The Idea of the University. New Haven: Yale U. Press, 1992. p. 17, 156, 165.
  29. Weed LL. Reengineering medicine: questions and answers. Federation Bulletin: The Journal of Medical Licensure and Discipline 1995;82:24-36.
  30. Weed LL. Physicians of the future, N Eng J Med 1981:304:903-907.
  31. Weed, LL. Medical records, medical education and patient care. Cleveland: Case Western Reserve University Press, 1969. p. v-vii, 101-107.