Introduction

General practitioners need to keep up to date. Evidence based medicine is a useful concept, and we all aspire to knowledge based practice, but it is not easy to appraise and assimilate all this knowledge.¹ There is a huge volume of medical literature, and medical knowledge is increasing at great speed.² We may aspire to practise evidence based medicine, seeking the answers to clinical questions in the literature and managing patients accordingly.³ But how do we assess the quality of the evidence if we have little training in clinical epidemiology and the skills of critical reading? Journal clubs may be of value and have been shown to work for those in training grades,⁴ but they require protected time. Texts and guidelines on critical reading are available but are increasingly complex and often demand a basic expertise. Moreover, little evidence exists that these methods have been assessed or subjected to clinical trial themselves.

We aimed to evaluate the READER model for critical reading and determine what factors influence the five components of the model (Relevance, Education, Applicability, Discrimination, overall Evaluation) (figure).⁵ Our prior hypothesis was that the READER method improves a general practitioner's ability to appraise the literature critically. We analysed the scores given by general practice principals in a randomised controlled trial of critical appraisal of three scientific papers.

View larger version (36K): [in this window] [in a new window]   Components and scoring system in READER method of critical appraisal

	Method

We invited all (n=1015) general practice principals on the medical list in Northern Ireland to participate in a study of three short papers published in the BMJ. Our only incentive was an offer to include all participants in a draw in which the prize was a voucher for a meal for two. We randomly assigned volunteers to two groups by using computerised random number allocation and sent both groups the same three papers selected from the BMJ in 1995.^6-8 These papers were selected from the general practice section of the journal and related to aspects of clinical care that would be of everyday importance in general practice. Each paper was less than one page long. We also requested some personal and practice details which could be possible confounding factors. For one group we included a copy of the READER scoring method and asked if they would rate the papers by using this method (intervention group). We asked the other group (control group) to give a free appraisal of the papers on the basis of "their importance to me in everyday work." They were asked to give two scores, one for scientific quality (maximum score of 10) and a total score (for overall importance (maximum 25)). There are no other known validated scoring methods for critical appraisal, and this free appraisal was an attempt to quantify the overall impression of the control group's opinion. As an incentive to complete the study we again offered to enter participants' names in a draw with a similar prize. Before we began the study we undertook a pilot study of 16 general practitioners. This revealed some minor problems in the instructions to participants and in the wording of the questionnaire. The results of this pilot study enabled us to estimate the sample size required to establish significant differences in the scores. The general practitioners who took part in the pilot were excluded from the main study. After the study we invited a sample of those in the intervention group to repeat their appraisal as a validation exercise. We also sought an objective expert opinion on the methodology scores: we asked an independent epidemiologist with experience in critical appraisal to assign a score to the methodology used in the three papers.

Differences between the two groups were compared by using the Mann-Whitney U test, and between the three groups (the intervention, control, and expert groups) by using Kruskal-Wallis one way analysis of variance on SPSS for Windows. Logistic regression was used to examine the possible associated factors for each component of the score. The Wilcoxon matched pairs, signed ranks test and McNemar's test were used in the repeatability study.

	Results

Of the 1015 principals invited to join the study, we excluded the 16 who had taken part in the pilot study. Of the 999 remaining general practitioners, 343 agreed to take part and were randomised. In all, 118 (69%) general practitioners in the intervention group and 125 (73%) in the control group completed the study (table 1). The only significant difference between the groups was in their sex, with significantly more women in the intervention group (P0.01), although this was a feature of the sampling and did not reflect a differential response. The groups were similar in composition in respect of educational factors (whether they were general practice trainers, whether the practice was a training practice, whether they taught medical students regularly); length of time in practice; location of the practice; and number of partners. Both groups read a similar number of articles from academic journals each month.

In an average month, 22% of participants read more than five articles, while 11% read no articles (table 2). Significantly more men than women read more than five articles a month, but significantly more men also read no articles (P0.05). Trainers were more likely to read more articles (P0.05), and no trainers admitted to reading none. No relation existed between the number of articles read each month and the location of practice, the number of partners, or whether the practice was a training or teaching practice.

The hierarchical order for the 3 papers was different for the two groups, according to the total scores (tables 3 and 4). Both groups gave the lowest score to paper 1. The highest median score in the intervention group was for paper 2; the control group ranked papers 2 and 3 equally, although it gave paper 3 a higher score for scientific quality. The control group gave a significantly higher total score than did the intervention group for paper 1 (P0.01) and for papers 2 and 3 (P0.001). The control group also gave a significantly higher score (P0.0001) for the scientific quality of all three papers than the intervention group did for the "discrimination" component of the READER method. Our independent expert gave a discrimination score of 4 for paper 3 and a score of 2 for papers 1 and 2. The proportion whose scores agreed with the expert appraisal is shown in table 5.

Of the 40 randomly selected participants in the intervention group who were invited to undertake the repeatability study, 19 replied. When the Wilcoxon matched pairs, signed ranks test was applied to the 15 variables (relevance, education, applicability, discrimination, overall evaluation, and total for each of the three papers), only two showed a significant difference between the two assessments. When we aggregated the scores given to each variable into two groups (low and high) we found no significant difference in the scores (McNemar's test).

Almost all participants (215/232) enjoyed taking part in the exercise. Overall, 135 participants (76/112 (68%) in the intervention group and 59/122 (49%) in the control group; (P0.0l)) believed that taking part in the exercise would encourage them to be more critical of published articles in the future.

	Discussion

This was a very large study of critical reading, with 243 doctors completing the study. The participation rate was remarkable in view of the work required and the increasing difficulties in getting general practitioners to respond to surveys.⁹ Journal clubs usually appraise two papers in about one hour,¹⁰ so participating doctors had made a major commitment to our work. In previous, smaller studies, ^{11 12} it took about 30-40 minutes to apply the READER method in a workshop, so each of the participating general practitioners probably spent about two hours reading and completing the paperwork associated with this study.

Confounding
There are possible confounding factors. In theory, critical reading is an objective skill and should not be influenced by the type of practice or seniority. The skills learned in an academic environment, however, may equip doctors to apply the skills of critical reading more effectively. None the less, the groups were similar in educational variables, type of practice, and seniority. There were significantly more women in the intervention group (the group who were asked to use the READER method), and this might be a confounding factor; there might also have been a selection bias if only the general practitioners who had an interest in critical reading took part.

Reading habits
In all, 89% of participants read at least one medical research article per month. There are no comparative data for other branches of the profession, but clearly general practitioners in this study have an interest in keeping up to date with the literature. One can draw few conclusions about the 11% who read fewer than one article each month: they may read professional journals or attend regular medical educational meetings. Indeed, while doctors report that they get most of their information from the literature, they regularly use other written sources and consult colleagues.¹³ All general practitioners were equally interested in reading, and it was unimportant if one was based in a teaching or training practice. All trainers read at least one research article a month, but this educational commitment does not seem to be shared by their practice partners who are not trainers.

Others factors
Other factors may influence the scores that general practitioners give to the components of the READER model. In examining the relation between all recorded factors (whether the doctors were trainers; whether their practice was a training practice or a teaching practice for medical students; the length of time in practice; the location of the practice; the number of partners; the number of articles read each month) and the four main components of the READER method, we found that in all three papers there were only two factors that approached a significance level of P0.01. These were detected in the assessment of paper 1, which focused on the use of lists by patients; the general practitioners who taught medical students were less likely than other doctors to give a high score for "relevance," whereas general practitioners in a training practice were more likely than other doctors to give a high score to this factor. As we explored such a large number of possible relations, these findings may have occurred by chance. No consistent factors, therefore, relating to the characteristics of the participants influenced the outcome of the results of the appraisal. The repeatability study also strongly supports the reliability of the method. This is important in establishing the validity of the model.

Conclusion
There was a significant difference between the scores given with the READER method and those given with free appraisal: general practitioners using the READER model gave a consistently lower total score and a more appropriate score to the methodology. The READER method was both accurate and repeatable. Overall, the articles scored highly on relevance but moderately on education and applicability.