This document discusses various methods that Cochrane entities use to prioritize topics for future Cochrane reviews. It identifies 17 entities that do not have a priority setting process and 27 that do or plan to. Common criteria for priority setting include clinical relevance, importance of the topic, impact on outcomes, and importance to specific populations. The document evaluates different approaches like the "Accountability for Reasonableness" framework and compares criteria like inclusiveness and equity. It poses discussion questions about selecting and applying criteria, evidence mapping, and integrating priority setting into the Collaboration's entities and strategies.
Evaluating the priority setting processes used across the Cochrane Collaboration
1. Workshop title: Ensuring relevance and building enthusiasm for Cochrane reviews:
determining appropriate methods for identifying priority topics for future Cochrane reviews
Evaluating the priority setting processes used
across the Cochrane Collaboration:
Accountability, Reasonability and Equity
Cochrane Colloquium, 3-7 October 2008
Freiburg, Germany
2. There is no simple way to set priorities. However, failure
to establish a process for priority setting has led to a
situation in which only about 10% of health research funds
from public and private sources are devoted to 90% of the
world’s health problems (measured in DALYs). This
extreme imbalance in research funding has a heavy
economic and social cost. To make matters worse, even
the 10% of funds allocated to the 90% of the world’s
health problems are not used as effectively as possible,
as health problems are often not prioritized using a
defined methodology.
Global Forum for Health Research
3. Prioritizing Cochrane Review Topics Relevant to Low- and Middle-
Income Countries
Peter Tugwell, Vivian Welch, Erin Ueffing, Zulma Ortiz, Mona Nasser, Elizabeth Waters,
Jodie Doyle, Andy Oxman
1) Identify what priority-setting approaches are underway by Cochrane entities
already and whether/how these can inform priorities for systematic reviews of
relevance to LMIC
2) Explore different methods of identifying priorities for systematic reviews for
specific audiences, recognizing that priorities may be different (eg for policy-
makers, for practitioners). We might convene small working groups of
practitioners to identify priorities for systematic reviews at a national or regional
level
3) Map existing systematic reviews to identified priorities to identify gaps/needs
for new systematic reviews, as well as showing whether the Cochrane Database
of Systematic Reviews already answers high-priority questions relevant to LMIC
4. Background:
• Ebrahim S, Moore T. Priority setting for review topics in the Cochrane Review Groups.
7th Cochrane Colloquium, 5-9 October 1999, Rome, Italy
• Ghersi D, Kennedy G, Rio P, Shea B. Consumer Setting Priorities for Cochrane Review
Groups. 5-9 October 1888, Rome, Italy
• Vet HCW, Korese MEAL, Scholten RJPM. The efficacy Of treatments for chronic benign
pain disorders: setting research priorities by literature searches With minimal reading.
8th Cochrane Colloquium 25-29 October 2000, Cape town, South Africa.
• Survey of prioritisation, commissioning and cochranisation of non-Cochrane reviews,
Adrian M Grant, CCSG, Khon Kaen, April 2006
• Bellorini J, Doree C. A method for priortising review topics: map of distribution of
randomized controlled trials (RCTs) in the Cochrane Review Group Trials Register. 14th
Cochrane Colloqiuium 23-26 October 2006, Dublin, Ireland.
• Doyle J, McDonald L, Bailie R, Waters E, Armstrong R. Prioritising and promoting topics
for systematic reviews to address the public health needs of indigenous people. 14th
Cochrane Colloqiuium 23-26 October 2006, Dublin, Ireland.
• Nasser M, Lodge M, Fedorowicz Z. The relevance of Cochrane Reviews to the Cancer
Priorities in Iran. 15th Cochrane Colloquium 23-27 October 2007, Sao Paulo, Brazil.
5. • We have contacted 67 Cochrane review groups and Cochrane fields.
• 52 groups responded (about 78%)
• 17 did not have a priority setting exercise
(about 25%)
• 8 had some relevant editorial process for accepting titles (about12%)
• 27 had a prioritsation process or planned one (about 40%)
6. Summary of ways the CRGs make decisions on prioritising new or
updated reviews:
1. Adapting priorities from other external organizations.
2. Using the database of uncertainties (DUETS)
3. Asking for recommendations and suggestions from editors (email, survey, face-to-face
meetings), authors or other members of the Cochrane Entities (CEs) (survey, email, web
based suggestion box). This might be accompanied by a further step on ranking the titles
in an editorial board discussion or emailing members of the CEs.
4. Identifying gaps in the existing literature (topics on which a number of trials have been
undertaken but no Cochrane review or systematic review exists) in areas which are
relevant (as judged by editors, members of the CEs, other stakeholders or identified in
guidelines).
5.Using a consensus process involving different stakeholders in the form of a workshop,
Delphi process, survey or focus group
6.Using health indicators e.g. Mortality or incidence to prioritise reviews.
7. Criteria for priority setting used by CEs
Clinical relevancy and importance
Knowledge of new trials available or numbers of them available
Improving patient outcomes
Importance to NHS priorities (UK based CRG)
Achievability and resources required
Impact on efficacy and efficiency
Timeliness
Relevancy to the country (Australian based CRG)
Focus of the CRG
Interset and priority as defined by individual members
How might this Cochrane review contribute to achieving the United Nationa‘s
Millenium Development Goals in infectious diseases, maternal motrality or
child health? (Infectious diseases CRG)
Burden of Disease, magnitude of problem and urgency
Importance to developing countries
Large scale impact on population.
Opportunity for action
8. Challenges:
1. Volunteer based contribution of the author’s
2. The priority setting could be useless if authors would not be interested to
take over any of the topics (Some fields provide bursary schemes in the
specific topic)
3. Inexperienced authors can not necessarily undertake a prioritize topics
which may encounter methodological difficulties
4. Lack of funding and time
9. How to evaluate or compare the processes?
Some examples:
• “Accountability for Reasonableness” (A4R)(1, 3)
• Comparison of various priority setting approaches in
the Combined Approach of Matrix (2)
• Programme budgeting and marginal analysis (PBMA)
(3)
References:
1. Martin D, Singer P. A Strategy to improve priority setting in health care institutions. Health care analysis 2003, 11
(1): 59-68.
2. Ghaffar A, Francisco A, Matlin S. The Combined Approach Matrix: A priority setting for health research. June 2004.
3. Gibson J, Mitton C, Martin D, Donaldsen C, Singer P. Ethics and economics: does programme budgeting and
marginal analysis contribute to fair priority setting? Journal of Health Services Research & policy 2006: 11(1) : 32-37.
10. “Accountability for Reasonableness” (A4R)
A4R is an ethical approach to priority setting that seeks to ensure
fairness in how priority-setting decisions are made.
Relevance: Decisions should be made on the basis of reasons (i.e. evidence,
principles, values, arguments) that ‘fair-minded’ stakeholders
can agree are relevant under the circumstances.
Publicity: Decisions and their rationales should be made available to
stakeholders.
Revision and appeals: There should be opportunities to revisit and revise
decisions in light of further evidence or arguments, and there
should be a mechanism for challenge and dispute resolution.
Enforcement: There is a voluntary or regulatory mechanism for ensuring that
the other three conditions are met
11. A possible framework to compare the strategies:
• Objective of priority Setting and process
• Inclusiveness
• Equity
• Evaluation of the existing evidence coverage
• Criteria for priority setting
• Continuity (expect of the normal procedure of a CRG) and
implementation
• Relevance (A4R criteria)
• Publicity (A4R criteria)
• Appeals (A4R criteria)
• Enforcement(A4R criteria)
14 of the processes used by Cochrane entities could be categorized
in this table.
12. Discussion Questions:
How to select and rank-order criteria for prioritisation?
Who should be consulted, and how, in setting and applying criteria?
Evidence mapping-how to judge applicability, extrapolation,
plausibility of existing, available systematic reviews?
How the prioritisation strategies could be integrated into the current
roles of all entities within the Collaboration;
How to align priority setting with new strategic funding opportunities
