Obstacles to answering doctors' questions about patient care with evidence: qualitative studyBMJ 2002; 324 doi: https://doi.org/10.1136/bmj.324.7339.710 (Published 23 March 2002) Cite this as: BMJ 2002;324:710
- John W Ely, associate professor ()a,
- Jerome A Osheroff, director of informaticsb,
- Mark H Ebell, associate professorc,
- M Lee Chambliss, assistant clinical professord,
- Daniel C Vinson, associate professore,
- James J Stevermer, assistant professore,
- Eric A Pifer, assistant professorf
- a Department of Family Medicine, University of Iowa College of Medicine, Iowa City, IA 52242, USA
- b Praxis Press, New York, NY 10010, USA
- c Department of Family Practice, Michigan State University, East Lansing, MI 48824, USA
- d Moses Cone Hospital Family Medicine Residency, Greensboro, NC 27401, USA
- e University of Missouri-Columbia School of Medicine, Columbia, MO 65212, USA
- f Division of General Internal Medicine, University of Pennsylvania, Philadelphia, PA 19104, USA
- Correspondence to: J W Ely
- Accepted 7 December 2001
Objective: To describe the obstacles encountered when attempting to answer doctors' questions with evidence.
Design: Qualitative study.
Setting: General practices in Iowa.
Participants: 9 academic generalist doctors, 14 family doctors, and 2 medical librarians.
Main outcome measure: A taxonomy of obstacles encountered while searching for evidence based answers to doctors' questions.
Results: 59 obstacles were encountered and organised according to the five steps in asking and answering questions: recognise a gap in knowledge, formulate a question, search for relevant information, formulate an answer, and use the answer to direct patient care. Six obstacles were considered particularly salient by the investigators and practising doctors: the excessive time required to find information; difficulty modifying the original question, which was often vague and open to interpretation; difficulty selecting an optimal strategy to search for information; failure of a seemingly appropriate resource to cover the topic; uncertainty about how to know when all the relevant evidence has been found so that the search can stop; and inadequate synthesis of multiple bits of evidence into a clinically useful statement.
Conclusions: Many obstacles are encountered when asking and answering questions about how to care for patients. Addressing these obstacles could lead to better patient care by improving clinically oriented information resources.
What is already known on this topic
What is already known on this topic Doctors are encouraged to search for evidence based answers to their questions about patient care but most go unanswered
Studies have not defined the obstacles to answering questions in a systematic manner
A comprehensive description of such obstacles has not been presented
What this study adds
What this study adds Fifty nine obstacles were found while attempting to answer clinical questions with evidence; six were particularly salient
The obstacles were comprehensively described and organised
Doctors are urged to practise evidence based medicine when faced with questions about how to care for their patients.1 They are advised to ask questions that can be answered with evidence and to evaluate the results of original research.1, 2, 3 But this advice may be difficult to follow in the pressurised atmosphere of a busy practice, where doctors are more likely to seek answers from readily available sources.4, 5, 6 Doctors are overwhelmed by the amount of information available, yet they often cannot answer their questions about specific clinical problems. 5 7, 8, 9
Much has been written about the qualities of a good question but little about the qualities of a good answer.1, 2, 3 Traditionally, the burden has been placed on the practitioner, who is told to ask “well built clinical questions” and to find the “best available evidence” to answer them. 1 3 We decided to shift the burden from the practitioner to the researcher and the author, who should address questions that occur in practice and synthesise original research so that it can be directly applied to patients.
We aimed to describe the range of obstacles that occur when trying to obtain evidence based answers to real clinical questions. We sought to build a taxonomy that characterises the problems that arise when searchers attempt to answer doctors' questions. This taxonomy could serve as a basis for better resources of knowledge and more accessible information within these resources. It could also guide strategies for finding relevant information in current resources. Doctors need up to date, high quality answers at the point of care within minutes.5 Before these objectives can be met with new information systems, the problems with current resources and search strategies need to be described.
Selection of questions
We collected 1101 questions from 103 family doctors in Iowa by using observations. The participants and procedures for data collection for this aspect of the study are described elsewhere.5 Briefly, after each consultation an observer asked the doctor to report any questions that occurred about how to care for the patient. We collected straightforward questions (“What is the dose of metformin?”) as well as vague uncertainties that would normally be kept to oneself (“I'm not sure what this rash is, but I'm going to call it a contact dermatitis for now.”). Using computer generated random numbers from a uniform distribution, we selected a random sample of 200 questions. Some of these questions were not amenable to evidence based answers (for example, “What is causing her abdominal pain?” “Is it ethical for me to take care of my own file clerk, who has back pain and wants a work excuse?”). Through an iterative process of reviewing questions, creating a classification scheme, coding questions, and revising the classification scheme, we developed a method of identifying questions that were potentially answerable with evidence. This iterative process, which has been described elsewhere, led to the development of an “evidence taxonomy” (box). 10 11 Using this taxonomy, we found that 106 questions (53% of the original 200) could potentially be answered with evidence.
Evidence taxonomy used to classify 200 questions from family doctors
“What is the drug of choice for epididymitis?”
No intervention (n=35)
“How common is depression after infectious mononucleosis?”
No evidence (n=35)
“What is the name of that rash that diabetics get on their legs?”
“What is causing her anaemia?”
“How do you stop somebody with five problems, when their appointment is only long enough for one?”
After listing these 106 questions in random order, one of us (JE) answered the first 10, the investigators answered two, and JE answered eight, totalling 20. We agreed on three criteria for selecting the two questions to be answered by all investigators: the question should be clearly stated, there should be a high likelihood of finding good quality evidence to answer it, and the answer should potentially have an impact on patient care. By using these criteria we selected “What is the proper treatment of gastro-oesophageal reflux disease (GERD)?” and “What should I use for atopic dermatitis?”
We did not follow a standardised search strategy because we wanted to study obstacles related to the strategy. We searched textbooks, journal articles, and various computer applications, but we did not seek individual consultations with humans. Working independently, the investigators completed searches that they thought were sufficient to avoid missing important evidence. While searching, the investigators used a modified “think aloud” method to write field notes that documented the obstacles they encountered. 12 13
Development of the taxonomy
We used three data sources to develop the initial taxonomy. The primary source consisted of obstacles documented in field notes written by the investigators as they attempted to answer the questions. The second source comprised frustrations that the investigators had encountered while answering other clinical questions. The third source consisted of problems reported in the literature. 1 14, 15, 16 The obstacles were described and organised into a taxonomy by using qualitative text analysis. The taxonomy was developed with an approach in which initial “codes” (obstacles described in the “think aloud” field notes) were augmented with obstacles described in the literature and previously encountered by the investigators.17
Validation of the taxonomy
To help validate the taxonomy, we first asked four volunteers (two medical librarians and two university family doctors) to answer four additional questions from the same dataset. Each volunteer coded their own field notes and identified obstacles that were not optimally characterised in the existing taxonomy. Secondly, we asked 21 practising doctors (purposively selected from a list of former trainees from practices in Iowa) to describe on paper the problems they encountered when attempting to answer one of their own questions. Thirdly, we completed 16 half day observation periods involving four randomly selected practising doctors in Iowa (four observation periods per doctor). We asked these doctors to “think aloud” as they attempted to answer their own questions. Based on these three additional sources of data, we added four obstacles to the taxonomy. The final version of the taxonomy was approved by all investigators.
The box shows the taxonomy of obstacles, with descriptions of each obstacle. The taxonomy was organised according to the steps in asking and answering questions18, 19, 20: recognise a gap in knowledge, formulate a question, search for relevant information, formulate an answer, and use the answer to direct patient care. Most of the obstacles were supported by the data we obtained, but a few were primarily generated from the previous experiences of the investigators or from the literature. These distinctions are noted in the box.
Obstacles to answering clinical questions
1. Obstacles related to recognising an information need
1.1. Doctor's lack of awareness of an information need. The doctor makes decisions about patient care, completely unaware of a gap in knowledge. (P)
1.2 Doctor's suppression of a recognised information need. On some level the doctor is aware of a gap in knowledge but suppresses it due to time pressures, embarrassment, personal characteristics, or characteristics of the clinical setting. (P)
2. Obstacles related to formulating the question
2.1. Inability to answer patient specific questions with general resources. Patient specific questions (“What is this rash?”) and vague cries for help (“I don't know what to do with this patient”) cannot be answered by a general resource. (D)
2.2. Missing patient data requiring unnecessarily broad search for information. Questions that include demographic information, clinical information, and patient preferences may help focus the search and shorten the answer. The type of patient data that would help varies depending on the question and may not be apparent until the search for an answer is under way. (D)
2.3. Uncertainty about the scope of the question and unspoken ancillary questions. It may not be clear whether the original question should be broadened to include potential ancillary questions. The answer to “What is the antibiotic of choice for pneumonia in a 5 year old?” could include ancillary prescribing information (for example, dose, adverse effects) thus avoiding the need to consult a second resource. An intermediary searcher may not anticipate such ancillary questions. (D)
2.4. Obstacles related to modifying the question
2.4.1. Uncertainty about changing specific words in the question. The doctor may ask a question using words that lead to difficulties in the search for information. For example, the word “sciatica” is less conducive to a literature search than “low back pain.” (D)
2.4.2. Unhelpful modifications resulting from flawed communication between the doctor and searcher. The real information need may be lost as it is communicated from the doctor to an intermediary searcher, such as a librarian. (D)
2.4.3. Need for modifications apparent only after the search has begun. Often a helpful modification to the original question becomes apparent only after the search for information is under way and the searcher learns more about the topic. (D)
2.4.4. Difficulty modifying questions to fit the PICO format (patient, intervention, comparison, outcome). Sackett et al suggest four elements for clinical questions: patient or problem, intervention, comparison, and outcome.1 However, many clinical questions do not involve interventions, comparisons, or outcomes.10 (D)
3. Obstacles related to seeking information
3.1. Failure to initiate the search
3.1.1. Doubt about the existence of relevant information. A search may not be initiated because the doctor doubts the existence of relevant information or doubts that any information found would change the plan of care. (D)
3.1.2. Question not important enough to justify a search. For example, the work of searching may not be justified if the question was motivated more by curiosity than by patient care.8 (L)
3.1.3. Lack of time to initiate search. Practising doctors have only a few minutes to answer their questions,5 but extensive time consuming searches are often required to adequately answer clinical questions.21, 22 (D)
3.1.4. Ready availability of consultation which leads to a referral rather than a search. Practising doctors may refer patients to consultants if they believe excessive time and effort would be required to learn enough about the problem to feel comfortable managing it themselves. (D)
3.2. Obstacles related to the search for information.
3.2.1. Uncertainty about where to look for information. It can be difficult to decide which resources will be most helpful and what should determine the selection of resources. Time available? Familiarity with resource? Type of question? (D)
3.2.2. Less than optimal strategy due to lack of searcher skill. When the searcher lacks skills for searching the literature or lacks familiarity with the internet, relevant information may be missed. (D)
3.2.3. Uncertainty about which order to search resources. After selecting potentially helpful resources it may not be obvious in which order to search these resources or what should guide the order (physical accessibility, resource quality, time available, etc). (D)
3.2.4. Uncertainty about narrowing the search without missing relevant information. When faced with an overwhelming body of knowledge about a topic, it may not be clear how to narrow the search (for example, during a Medline search) without losing relevant information. (D)
3.2.5. Uncertainty about which articles to read thoroughly and how thoroughly to read them. It is often not clear how to select resources, such as journal articles, for thorough reading. Should the decision be based on the title, the abstract, the prestige of the journal, other factors? Once an article has been selected for more thorough reading, how thoroughly should it be read and how can the needed information be found without reading every word. (D)
3.2.6. Uncertainty about the adequacy of the search (when to stop searching). Often it is not clear when to stop searching for information. When can the question be answered adequately? How can it be known that all the important evidence has been found? (D)
3.2.7. Uncertainty about the meaning of null search results. If no interaction between two drugs is listed, does that mean no interaction exists? If a relevant article makes no mention of treatment for a disease, without explicitly stating that there is no treatment, does that mean there is no treatment? (D)
3.2.8. Inadequate indexing of databases used for computerised literature searching. For example, a MeSH term for the topic of interest might not exist or a relevant article might not be indexed under the intuitive MeSH term. (D)
3.2.9. Lack of time to search adequately. Once initiated, the search for information may be suboptimal because pressures on time lead to a poor choice of resources. (D)
3.3. Obstacles related to knowledge resources (for example, books, journals, computers)
3.3.1. Resource physically distant. The resource may not be readily accessible to the doctor. (D)
3.3.2. Topic or relevant aspect of topic not included in a resource that should logically include it. Based on the title of the article or book, coverage of the topic would have been expected. (D)
3.3.3. Inadequacy of the resource's index. The topic may be covered in the text of a book or computer resource but not listed in the index. The topic may be listed in the index but not under an intuitive entry. The index may be inadequately cross referenced. (D)
3.3.4. Resource poorly organised. Resources may be poorly organised within a personal library or reprint file. Information within a resource may be poorly organised or have inadequate titles or subtitles. (D)
3.3.5. Resource not clinically oriented. For example, textbooks are often organised by disease rather than by clinical findings, which forces the doctor to “work backwards.” (D)
3.3.6. Obstacles related to computers (hardware and software problems). Bugs, slowness, unintuitive software, internet problems. (D)
3.3.7. Difficulty accessing resources in libraries. The library may not hold the needed journal or book. The needed volume may not be on the shelf. Journal supplements are often difficult to find or missing. (D)
3.3.8. Resource not authoritative or not trusted. The resource may not be authoritative or it may not be trusted by the searcher. (D)
3.3.9. Resource not current. The resource may not be current or it may be difficult to know if it is current (for example, undated internet sites and printed material). (D)
3.3.10. Inability to interact with a general resource as one could with a human resource. Most general resources do not allow real time interaction with the searcher as could happen with a human resource. There can be no follow up questions. (D)
3.4. Obstacles related to information within resources
3.4.1. Incorrect information. The information simply may be wrong. (D)
3.4.2. Information not current. The resource containing the information may or may not be current, but the information itself is not current. (D)
3.4.3. Failure to anticipate ancillary information needs. There is inadequate anticipation of likely ancillary or follow up questions (for example, the name of a recommended drug is provided but not the dose, forcing the searcher to consult another resource). (D)
3.4.4. Failure to address common comorbid conditions. The information refers to patients in general but does not account for common comorbid conditions or the question is answered for patients with a comorbid condition but not for patients in general. (D)
3.4.5. Inadequate differential diagnosis. A differential diagnosis consists of a list of diseases with little information about how to distinguish among the possibilities. (D)
3.4.6. Failure to define important terms. The information includes terms that are not defined. For example, treatment may vary depending on whether the disease is mild, moderate, or severe, but these terms are not defined. (D)
3.4.7. Inadequate description of clinical procedures. A clinical procedure (for example, thoracentesis) is described but there is insufficient detail to allow the doctor to do it. (P)
3.4.8. Vague or tangential information. The information does not allow the question to be answered directly because of a vague, tangential, or overly general format. (D)
3.4.9. Unnecessarily cautious writing style. The information is overly cautious and may contain unnecessary hedge words (“can,” “may,” etc). The caution may be legitimate (inadequate evidence to support a definitive statement), but it may be unnecessary. (P)
3.4.10. Tertiary care approach to primary care problem. Available information may take an urban interventionist tertiary care approach, which may not be useful to a rural primary care doctor with a non-interventionist philosophy. (D)
3.4.11. Biased information due to conflicts of interest. The author or editor may have conflicts of interest. (D)
3.5. Inadequacy of available evidence
3.5.1. Failure to address the clinical question. Available studies have not adequately addressed the question (for example, “Is smoking a risk factor for sinusitis?”). (D)
3.5.2. Failure to study the comparison of interest. Drug companies often sponsor clinical trials comparing drug A with placebo, but the question is whether drug A is better than drug B. (D)
3.5.3. Failure to study the outcome of interest. An intermediate outcome, such as serum cholesterol level, may be studied rather than more clinically important outcomes, such as myocardial infarction or death. (D)
3.5.4. Failure to study the population of interest. It may not be appropriate to apply results from a referral population to the primary care setting. (D)
3.5.5. Evidence based on flawed methods. Multiple flaws (for example, selection bias, misclassification bias, confounding, etc) may invalidate the results. (D)
3.6. Obstacles related to the use of available evidence
3.6.1. Failure to cite or include relevant evidence. Evidence exists but is not cited. It may be difficult to know if evidence exists and, if it exists, to what extent it has been used to write a chapter or review. (D)
3.6.2. Inadequate synthesis of multiple bits of evidence. Relevant evidence is available but consists of numerous bits of information that have not been synthesised or interpreted. Evidence may be summarised but not systematically or rigorously. Conflicting evidence is presented without providing a definitive recommendation for the clinician who must make a decision. (D)
3.6.3. Difficulty applying results of randomised clinical trials to individual patients. Clinical trials are often narrow in scope and may not apply to patients with comorbid conditions.15 (L)
4. Obstacles related to formulating the answer
4.1. Failure to directly or completely answer the question. Once the relevant information has been gathered, the searcher fails to directly or completely answer the doctor's question (for example, owing to the inadequacy of available information or an inadequate synthesis of adequate information). (D)
4.2. Answer too long or too short. The answer is too long to be helpful to a busy doctor or too short to completely address the information need. (D)
4.3. Answer directed at the wrong audience. Answers for patients may not be helpful to doctors. (D)
4.4. Difficulty addressing unrecognised information needs apparent in the question. It may not be clear how to address unrecognised information needs that are evident in the question. For example, the question might ask about the dose of a drug that is contraindicated (“What is the dose of tetracycline for acne in a pregnant woman?”). (D)
4.5. Discomfort of non-clinician searcher (for example, librarian) formulating an answer to be used in patient care. Intermediary searchers who are not doctors (librarians, nurses) may be comfortable providing information on a given topic but not formulating an answer that would direct patient care. (D)
5. Obstacles related to using the answer to direct patient care
5.1. Answer not trusted. A seemingly adequate answer may not be used if the doctor does not trust the source. (D)
5.2. Answer moot or no longer needed. The answer may be moot or irrelevant because it came too late or the patient improved or got worse before the answer could be applied. (D)
5.3. Answer inadequate. If the answer is thought to be inadequate by the doctor, it may not be used to direct patient care. (P)
(P), obstacles primarily supported by previous experiences of authors; (D), obstacles primarily supported by data collected in this study; (L), obstacles primarily supported by previously published medical literature.
Our methods did not allow a formal frequency analysis, but several obstacles seemed particularly salient because they recurred in the various procedures for data collection, and they were characterised as fundamental problems by the investigators and practising doctors. These were the excessive time required to find information, difficulty modifying the original question, which was often vague and open to interpretation, difficulty selecting an optimal search strategy, failure of a seemingly appropriate resource to cover the topic, uncertainty about how to know when all the relevant evidence has been found so that the search can stop, and inadequate synthesis of multiple bits of evidence into a clinically useful statement.
The obstacles related to evidence fell into two main categories. Firstly, the available evidence was inadequate to directly answer the question either because studies had not addressed the question (“Is smoking a risk factor for sinusitis?”) or because the studies that had addressed the question provided incomplete information. For example, when answering the question about treating gastro-oesophageal reflux disease, we found rigorous comparisons between lansoprazole and placebo and between omeprazole and placebo but the comparisons between lansoprazole and omeprazole were less definitive.23, 24, 25 Secondly, even when the evidence was adequate to answer the question, further obstacles hindered its use in the clinical setting. Available evidence often consisted of individual study results, which had not been synthesised or interpreted for clinicians. The following field notes were written by one of the investigators as he attempted to answer the question, “What should I use for atopic dermatitis?”
Therapies include: ciclosporin—possibly effective; borage oil—probably not effective; UVA1—works; primrose oil with water and oil emulsion—probably effective; topical doxepin—probably not effective; mite elimination—likely to be effective; topical cromolyn—likely to be effective; topical tacrolimus—probably effective; SEZ ASM 981—possibly effective.
Each investigator spent a median of 95 minutes (range 13 to 639 minutes) answering the question about gastro-oesophageal reflux and 45 minutes (range 17 to 374 minutes) answering the question about atopic dermatitis. For all 20 questions, JE spent a median of 327 minutes per question (range 90 to 1075 minutes). The question that took the shortest time was “Are there any drug interactions with St John's wort, specifically with SSRIs?” This question was highly specific, and it soon became apparent that there was almost no evidence to answer it. The question that took the longest time was “Why is there an increasing incidence of asthma, just in general?” The literature on this topic contained extensive speculation and many theories based on observational studies. It was difficult to know when to stop looking for a definitive study, but eventually it became clear that a definitive study did not exist. The highest level of evidence available was a randomised clinical trial for seven of the 20 questions, an observational study for eight questions, and an opinion for the remaining five questions.
The final version of the taxonomy comprised 59 obstacles. The four volunteer coders used 35 problems to code their field notes and made four suggestions to improve the taxonomy. For example, both librarians noted their lack of medical training as an obstacle to formulating an answer. Ten of the 21 practising doctors responded to our request to describe obstacles that arose as they answered one of their own questions. All of the obstacles reported by these doctors had been described in the existing taxonomy. Also, we collected 96 questions during 16 office observations from four additional practising doctors. These data led to the addition of four obstacles to the taxonomy: failure to initiate a search due to doubt about the existence of relevant information, ready availability of consultation, which leads to a referral rather than a search, uncertainty about the meaning of null search results, and resource not clinically oriented.
Obstacles arise when searching for evidence based answers to doctors' questions: we identified 59. Among the most salient were inadequate time to search for information, failure of the resource to address the topic, and inadequate synthesis of multiple bits of evidence into a clinically useful statement. Practising doctors often decided not to pursue their questions because they doubted the existence of useful information in available resources.
In a study of internists in Los Angeles, many questions were phrased in patient specific terms that would make it difficult to find answers from generally available resources.9 For example, a doctor would ask “Should I test the serum procainamide level in this patient?” rather than “What are the indications for measuring serum procainamide?” We excluded such patient specific questions, although we could have modified them into a more general form. In the study from Los Angeles, doctors reported lack of time as the most common barrier to finding information.9 Other investigators have identified obstacles involving computers and the internet. 14 16 One study cautioned against the uncritical use of clinical trial results for direct patient care.15
Although we sought to build a comprehensive list of problems, the taxonomy we developed was primarily based on only 20 questions, which came from a homogeneous group of doctors. However, we attempted to validate our findings by applying the original taxonomy to the obstacles encountered while answering an additional 106 questions, generated by 14 practising doctors (96 questions from consultations plus 10 mailed questions). Our taxonomy's framework was based on the steps in the process of asking and answering questions, but other frameworks could have been used. We tended to blame the author for difficulties the doctor might encounter when searching for information. While we believe that enhancing search skills could overcome some of these difficulties, we chose to focus on how resource developers could address the problems with retrieval of information we identified.
After quantifying and prioritising the obstacles we found, the taxonomy we developed could be used to write recommendations for authors as they attempt to produce clinically useful material. Authors will be most effective if they anticipate the needs of busy clinicians who often have only a minute or two to find information.5 For example, authors who name the drug of choice for a specific condition could include essential prescribing information (dosage, drug interactions, safety in pregnancy), so that the clinician does not waste time consulting another resource. Clinically oriented resources could be written in a question and answer style rather than a disease and topic style. The ongoing surveillance of doctors' changing questions could help keep resources current. Questions without adequate answers could help guide research and funding priorities. Until such research is completed, such questions may prompt the use of holistic clinical care and other alternatives. We often found it helpful to modify questions from the way they were originally stated by the doctor. Such modifications could be developed into recommendations for doctors, as they formulate their questions, and for intermediary searchers, who may play a larger part in the future, as they help doctors practise the best medicine.26
To meet the needs for clinical information, doctors must be aware of their gaps in knowledge and then formulate questions that can be addressed by available resources or patient specific consultations. When faced with a gap in knowledge, doctors must decide whether to do the best they can with their current knowledge or to expand that knowledge by formulating and answering a question. Practising doctors do not have time to search multiple sites or scroll through long text. Nor do they have time to search multiple textbooks or perform literature searches for most of their questions. They need to pick the right resource the first time, the information in that resource needs to be readily found, and all the information must be there. Although it remains to be shown, we believe that systems designed to overcome the obstacles we identified will improve the asking and answering of questions and potentially patient outcomes.
We thank Marcy E Rosenbaum and Toni Tripp-Reimer for their critical review and advice concerning the qualitative analysis, Susan Meadows, Dedra Diehl, Barcey Levy, and Robert Garrett for their help in verifying the final taxonomy, and the practising doctors who helped validate and refine the taxonomy.
Contributors: JWE collected the Iowa questions, answered 20 of these, coordinated the development of the obstacles' taxonomy, and wrote the first draft of the paper. JAO had the original idea of organising the obstacles into a taxonomy. JAO, MLC, DCV, JJS, and EAP answered two questions and helped with repeated revisions of the taxonomy of question types and the taxonomy of obstacles. All authors contributed to editing the paper. JWE and JAO will act as guarantors for the paper.
Funding This study was supported by grants from the American Academy of Family Physicians Foundation (G9518) and the National Library of Medicine (1R01LM07179-01).
Competing interests JAO is an employee of Praxis Press, a company that produces evidence based clinical information resources for primary care doctors.