Commissioning a national programme of research and development on the interface between primary and secondary care
BMJ 1995; 311 doi: https://doi.org/10.1136/bmj.311.7012.1080 (Published 21 October 1995) Cite this as: BMJ 1995;311:1080
- Correspondence to: Professor Haines.
- Accepted 3 August 1995
The first call for applications to the NHS research and development programme on the interface between primary and secondary care was advertised in February 1994. A total of 674 outline proposals were submitted and 54 (8%) secured funding. Projects have been commissioned in 16 of the 21 priority areas and around pounds sterling6m has been committed. Analysis shows that multidisciplinary applications are more likely to be funded and that the odds for a successful application are on average nearly doubled for each discipline represented up to five. A survey of applicants and peer reviewers found satisfaction with much of the commissioning process, but peer review and feedback were subject to criticism, particularly by unsuccessful applicants. The programme shows that it is possible to commission a large number of projects in an innovative area of research and development and has identified refinements that will further increase the efficiency and acceptability of the process.
The priority setting process for the national programme of research on the interface between primary and secondary care is described in the accompanying article (p 1076).1 This paper details the commissioning process. We believe that the process should be as “transparent” as possible and that there should be opportunity for comment and constructive criticism to help refine the process.
Commissioning process
Responsibility for the national programme of research and development on the interface between primary and secondary care was given to the research and development directorate at North East Thames Regional Health Authority (now NHS Executive North Thames). A commissioning group chaired by Professor Michael Clarke was established to advise on the programme. The commissioning process (figure) followed a two stage format established by previous NHS programmes.
Outcome of commissioning process to February 1995 (one funded project subsequently withdrawn—see text)
CALL FOR RESEARCH PROPOSALS
In February 1994 an advertisement was published in the national press, in health journals, and by the regional research and development directorate networks. Four page outline proposals were requested and the desirability of multidisciplinary applications mentioned. A total of 1455 requests for further information were made, and the application pack included a brief description of potential subjects for research within each priority sector.2
SHORTLISTING PROCESS
In total, 674 outline proposals were submitted. Initial triage identified 97 that clearly did not address a priority area, and these were excluded. The remaining 577 were then reviewed by a member of the commissioning group and an external reviewer. Reviewers, who were not allocated proposals from their own institutions, were asked to award each proposal a rating from 0 (“unsupportable”) to 4 (“exceptionally high standard: strongly recommend shortlisting”) (box 1). When awarding ratings the reviewers took into account the priorities of the programme and considered nine criteria (box 2).
Box 1—Rating system used by external reviewers to assess proposals
0—Unsupportable
1—Low priority for shortlisting
2—Good: could be shortlisted
3—Very good: recommend shortlisting
4—Exceptionally high standard: strongly recommend shortlisting
Box 2—Criteria considered by external reviewers when rating proposals
Is the concept behind the research idea sound?
Are the study design methods appropriate to the research question?
Are the objectives of the study sufficiently described?
Is there evidence that the applicant can carry out the work?
As far as you are aware, has the relevant published work on the subject been taken into account?
Does the proposal have a multidisciplinary approach?
Are the cost and the duration of the project justified?
Is the study design likely to deliver findings that will benefit the NHS?
Are the methods described for implementation feasible?
Even weighting was given to each priority in the shortlisting process. The proposals were grouped according to their overall rating—namely, high (>/=6; 68 proposals (12%)); borderline/mixed (5, 4 (4-0), (3-1), and 3 (3-0)); 104 proposals (18%)); and low (4 (2-2), 3 (2-1), 2, 1, and 0; 405 proposals (70%)).
The commissioning group focused on the borderline and mixed proposals, and each was discussed in turn. Members were asked to leave the room before the discussion of proposals from their institution and were informed of the outcome only after the meeting. Members could nominate proposals from either of the other two groups—that is, those rated high or low—for discussion before agreement was reached on shortlisting of the highly rated proposals and rejection of the lower rated proposals. A total of 110 (16%) of the 674 proposals submitted were shortlisted, and these covered all of the 21 priority areas, though this was not a criterion of the shortlisting process. A further three proposals that had obtained high ratings were taken forward by the Medical Research Council. Because of the large number of requests for feedback from unsuccessful applicants (over 300) it was not possible to provide detailed feedback.
EVALUATION OF FULL PROPOSALS
Altogether 103 full applications were submitted. A withdrawn application secured a cash limited award from the MRC, and the commissioning group subsequently agreed joint funding. Each proposal received at least two peer reviews. One hundred and twenty five reviewers took part in this stage of commissioning, many of whom had also supported the programme at the shortlisting stage.
Each proposal was assigned a lead member and two readers from the commissioning group. The meeting followed a similar format to that of the shortlisting. The lead member made a funding recommendation and the group focused on the borderline proposals. Decisions on all but 25 of the projects were made at this meeting and the remainder completed by correspondence.
The 54 successful applications represented 8% of the original 674 applications and 52% of the 103 full applications. Eighteen of the 54 funded projects addressed the first three priorities—namely, transfer of information across the interface, between health care professionals and other agencies; evaluation of clinical guidelines at the interface; appropriate access, use, and location of diagnostic facilities; and new technologies reflecting the large number of proposals submitted in these areas (n=181). Proposals were not funded in five priority areas (15, 17, 18, 20, and 21), all of which had been identified as further rather than top priorities by the advisory group.1 Seven proposals were funded at a lower level than sought by the initial application, either because the group thought that a pilot study was required or because the possibility of substantial savings had been identified. One successful proposal has not been taken forward owing to funding from local purchasers for a new service being withdrawn.
Feedback of comments by peer reviewers was provided to all unsuccessful applicants. The reason for the group's decision was also forwarded when this had not been made through peer review.
RESOURCES
A total of pounds sterling5964000 was committed to the programme after the first call for proposals. Most of the projects will run for up to two years (31 projects; 57%) or three years (20; 37%) and will be monitored by the programme manager. The project which could not be carried forward has reduced the total by about pounds sterling260.00.
MANAGEMENT COSTS
The first year's management costs (to April 1995) were pounds sterling65000 (including full time programme manager and clerical support). Management costs during 1996 and 1997 are expected to be around 2% of the resources committed. No provision has been made for the commissioning groups' and peer reviewers' time. Each commissioning group member read and assessed about 55 outline proposals and 40 full applications, together with the reviews in each case.
COMPARISON WITH OTHER NATIONAL NHS RESEARCH AND DEVELOPMENT PROGRAMMES
The response to the call for proposals in the primarysecondary care interface programme was somewhat higher than for other national programmes. However, when based on comparable available data the success rate was similar. From the first calls for proposals the cardiovascular disease and stroke programme funded 37 out of 380 projects (10%) (B Woodbridge, personal communication) and the physical and complex disabilities programme 30 out of 427 (7%) (R Simpkins, personal communication).
Satisfaction of applicants and reviewers
A questionnaire survey examined applicants' and peer reviewers' satisfaction with the commissioning process. Response rates were 74% among a random sample of 100 applicants who were not shortlisted, 65% (n=36) among 55 shortlisted applicants who did not secure funding (one applicant had two proposals), 90% (n=47) among 52 funded (two lead applicants had two projects funded) applicants, and 78% (n=112) among 144 peer reviewers (76 commented on the procedures for full proposals and 70 on outline proposals).
DOCUMENTATION
Applicants submitting outline proposals agreed that the clarity and standard of the supporting documentation (61/74 applicants (82%) not shortlisted; 79/83 (95%) shortlisted) and design of the application form (58/74 (78%); 75/83 (90%)) were good or adequate. Satisfaction was also recorded with the documentation for full applications and was reflected in the response from the peer reviewers. Other comments mostly concerned the lack of availability of the application forms on disk. The full application form was issued on disk but only in the software packages available at North Thames.
TIME SCALE
Only 15% (23/157) of applicants responding to the survey who submitted outline proposals and 17% (6/36) of those respondents unsuccessful at the second stage thought the decision making process too long, compared with 16 (34%) of the 47 successful applicants (owing to deferred proposals at the second stage caused by inadequate and late peer review reports). Around one third of applicants who responded thought the time allowed for completing both full and outline applications was too short.
PEER REVIEW AND FEEDBACK
Considerable dissatisfaction was expressed with the quality of peer review and feedback (tabletable I). Applicants suggested that they should have had an opportunity to address the peer reviewer's comments before the commissioning group meeting. However, many recommendations were subject to clarification or development, and shortlisted applicants were effectively given this opportunity at a later stage. Predictably, perhaps, criticisms of the peer review process were much greater among unsuccessful applicants. In contrast, the survey of peer reviewers found general satisfaction with the process. Around 90% of reviewers responding thought that enough time was allowed for the reviews and that the quality of the forms was good or adequate.
Views on peer review and feedback. Figures are numbers (percentages) of respondents
MULTIDISCIPLINARY APPLICATIONS
The applicants on each proposal were coded according to their discipline and organisation, and an analysis was undertaken of the extent to which the multidisciplinary nature of an application was a predictor of success (tabletable II). Only one unidisciplinary application was successful. Logistic regression analysis showed a highly significant association between the number of disciplines represented and the probability of success (P<0.0001). The odds of a successful application were nearly doubled (odds ratio 1.93; 95% confidence interval 1.53 to 2.44) with each unit increase in the number of disciplines represented.
Success of applications for funding by numbers of disciplines represented
Discussion
PEER REVIEW PROCESS AND FEEDBACK TO APPLICANTS
Clearly the peer review process needs improving. A more rigorous shortlisting process could allow more feedback on outline applications. However, the scope for providing feedback at this stage is limited by the administrative resources available to the programme and the number of applications submitted.
The logistics of involving so many peer reviewers for the full proposals were formidable and in many cases deadlines were not met, which resulted in delayed decisions. Because the subject is new in research terms and the topics often require multidisciplinary approaches it was sometimes difficult to match the project to the reviewers. This was, however, partly compensated for by ensuring that any deficits in specific disciplines were addressed at the commissioning group stage.
The time available for allocating reviewers will be increased for future calls. Allocation to reviewers will be by research discipline as well as clinical specialty, and this will be facilitated by asking applicants to identify key words describing the methodology of the proposal. As the portfolio of referees is developed for the programme, obtaining timely and good quality peer reviews should become easier. The workload of the peer reviewers is an important factor in determining the quality of the reviews, and future calls will be coordinated with other NHS research and development initiatives to minimise the burden on reviewers. There are other options for commissioning that would reduce the number of peer reviews—for example, limited tendering. However, under European Commission regulations projects costed at over pounds sterling149728 must be openly advertised.3
Giving more advanced notice of NHS research and development programmes may partly address criticisms about the timescale for submitting outline proposals. Feedback at the second stage could also be improved by ensuring that the lead commissioning group member for each project is responsible for providing feedback based on discussions within the committee, particularly where referees' comments do not cover the salient points.
SUCCESS RATE
The success rate of applications might be improved by more detailed documentation. This, however, would delay the commissioning process and should be balanced against the risk that the documentation would become too prescriptive.
Future calls should request that at least two disciplines are represented in applications, as there is a strong association between the number of disciplines represented and the probability of success. This could be partly due to a multidisciplinary approach being included among the criteria used by referees and partly because the greater the number of disciplines involved the more likely it is that the proposal is methodologically sound. A more rigorous selection process at the first stage in future calls would increase the success rate at the second stage and reduce the time spent developing, processing, and assessing unsuccessful proposals. However, this would increase the time taken to shortlist and should also be balanced against the increase in workload for reviewers at the outline stage. The failure of one project due to the withdrawal of funding for an innovative service by local purchasers suggests that effective mechanisms must be developed to ensure that funding is made available to support services that require evaluation.
The process described is a reasonably efficient way of commissioning a large number of projects. It shows the feasibility of generating a large number of well designed studies in an innovative area for research over a short period but also shows the need to improve peer review and feedback. The modifications we have described should substantially improve the commissioning process. A second call for proposals will be issued in November.
Acknowledgments
We thank members of the commissioning group (Dr Martin Bland, Professor Michael Clarke (chairman), Mr John James, Professor Roger Jones, Mr Nick Mays, Dr Jenny Roberts, Dr Richard Stevens, Ms Kate Thomas, Professor David Wilkin, Professor David Yates, and Dr Christina Davies (MRC observer)) and all the reviewers; Ms Stephanie Goubet for the statistical analysis; and Ms Melanie Baillie-Johnston, Mr David Knight, Dr Tim Riley, Mr Peter Richardson, and Ms Monika Temple for support and advice.
Footnotes
-
Funding None.
-
Conflict of interest AH is on secondment from the department of primary health care, University College London Medical School; members of the department are undertaking research funded by the NHS research and development programme.