This document discusses the GRADE system for developing clinical practice guidelines and making recommendations. It outlines some of the limitations of existing guideline development processes, including lack of transparency. GRADE provides a transparent and structured framework with three key steps: 1) assessing the quality of evidence through systematic review, 2) evaluating the risk-benefit profile and considering patient values, and 3) determining the strength of recommendations based on the quality of evidence and risk-benefit analysis. Examples are given of how GRADE has been applied to make both strong and weak recommendations. The speaker argues that adopting GRADE would help address many of the existing problems in guideline development.

1. Elaboración de
recomendaciones en las GPC
Sistema GRADE
Nicola Magrini
NHS CeVEAS, Centre for the Evaluation of the Effectiveness of
Health Care, Modena, Italy
WHO Collaborating Centre for Evidence Based Research Synthesis
and Guideline Development

2. Contents of the presentation
• What are the defects of existing guidelines and
systems of grading
• Why GRADE could help …
• A three pillar method: the GRADE system to
evaluate quality of evidence and define the
strength of a recommendation
• Three examples
• Conclusions

3. Contents of the presentation
• What are the defects of existing guidelines and
systems of grading
• Why GRADE could help …
• A three pillar method: the GRADE system to
evaluate quality of evidence and define the
strength of a recommendation
• Three examples
• Conclusions

4. http://clinicalevidence.bmj.com/ceweb/about/knowledge.jsp 2011

5. Tendency of recent guidelines

6. Trends in guideline production
(AHA guidelines, Tricoci JAMA 2009)
• Recommendations are increasing in size with every update
(+48% form first version)
• Quality of evidence: only a minority of recommendations are
based on good evidence (11%) and half (48%) on low quality
evidence
• Recommendations with high quality evidence are mostly
concentrated in class I (strong recommendation) but only 245
of 1305 class I recommendations have high quality evidence
(median, 19%)

7. Guidelines reassessment …
• … in ACC/AHA guidelines with at least 1 revision, the number of
recommendations increased 48% from the first guideline to the
most recent version. If there is a main message in such
guidelines, it is likely to be lost in the minutiae.
• Within a guideline document, individual recommendations also
need to be prioritized.
• Finally, guidelines need flexibility. Recommendations should
vary based on patient comorbidities, the health care setting,
and patient values and preferences.
• Physicians would be better off making clinical decisions based
on valid primary data.
Shaneyfelt TM, Centor RM.
Reassessment of clinical practice guidelines
JAMA 2009

8. How to improve guideline quality
Present limitations:
• Governance and composition of the guideline
committee (“what is to be decided is often already decided with the
selection of the deciders”)
• Unanimity in guideline (not a natural component in research)
• Lack of independent review (outside the accepted procedures of
scientific publications)
• Suboptimal management of Conflicts of interests
Sniderman AD, Furberg CD.
Why guidelines making requires reform
JAMA 2009

9. Too many grading systems?
Who is confused?
Evidence Recommendation
B Class I
C+ 1
IV C
Organization
AHA
ACCP
SIGN
Recommendation for use of oral
anticoagulation in patients with atrial
fibrillation and rheumatic mitral valve
disease

10. Contents of the presentation
• What are the defects of existing guidelines and
systems of grading
• Why GRADE could help …
• A three pillar method: the GRADE system to
evaluate quality of evidence and define the
strength of a recommendation
• Three examples
• Conclusions

11. Why using GRADE
GRADE is much more than a rating system
• offers a transparent and structured process for
developing and presenting summaries of quality of
evidence
• provides guideline developers with a comprehensive and
transparent framework for carrying out the steps involved
in developing recommendations
• specifies an approach to framing questions, choosing
outcomes of interest and rating their importance,
evaluating the evidence, and incorporating evidence with
considerations of values and preferences of patients and
society to arrive at recommendations

12. WHO guideline
development processes
update 2010

13. 1. Scoping the document: reasons for choosing the topic, problems
with existing guidelines, variations and gaps,
2. Group composition (or consultations)
3. Conflict of interest
4. Formulations of the questions and choice of
the relevant outcomes
5. Evidence retrieval, evaluation and synthesis
(balance sheet, evidence table)
6. Benefit/risk profile: integrating evidence with
values and preferences, equity and costs
7. Formulation of the recommendations
8. Implementation and evaluation of impact
9. Research needs or areas of further research
10. Peer-review process and updating
Title, responsible person, WHO Department
- responsible of the clearance process, WHO Departments involved, CC involved,
Standards for evidence: GRADE system
Reporting standard and process
Reporting standard and process

14. GRADE Working Group website
and publications
www.gradeworkinggroup.org

15. Contents of the presentation
• What are the defects of existing guidelines and
systems of grading
• Why GRADE could help …
• A three pillar method: the GRADE system to
evaluate quality of evidence and define the
strength of a recommendation
• Three examples
• Conclusions

16. GRADE: a 3 pillars approach
1. Formulate the question, choose and rate your
outcomes of interest and perform a systematic
review (quality of evidence)
2. Risk benefit evaluation, consider patients
values and preferences and also resource use
and feasibility
3. Direction (positive/negative) and strength
(strong/weak) of the recommendation

17. GRADE: a 3 pillars approach
1. Formulate the question, choose and rate
the outcomes of interest and perform a
systematic review (quality of evidence)
2. Risk benefit evaluation, consider patients
values and preferences and also
resource use and feasibility
3. Strength of the recommendation

19. Figure 1: Hierarchy of outcomes according to
their patient-importance to assess the effect of
enteral supplement nutrition for geriatric patients
with bed sores
Nutritional status 4
Importance
of endpoints
Microcirculation
of the wound 1
2
Energy supply 3
5
Function 6
Quality of life 7
Healing of the 8
bedsore
Mortality 9
Critical
for decision making
Important,
but not critical for
decision making
Not patient-important
Rating of
outcomes
… example:
patient with
bed sores

20. WHO Recommendations for the Prevention of PPH, 2007

21. WHO Recommendations for the Prevention of PPH, 2007

22. Study design is important
 Early systems of grading the quality of evidence
focused almost exclusively on study design
 Randomised trials provide, in general, stronger
evidence than observational studies:
–RCTs start at High Quality
–Observational studies start at Low Quality
 However, other factors may decrease or increase
the quality of evidence

23. Quality assessment criteria:
the big start

24. Factors that may decrease the
quality of evidence
Study limitations (risk of bias)
 well established
– concealment
– intention to treat principle observed
– blinding
– completeness of follow-up
– Choice of comparator (standard/optimal treatment)
 more recent
– early stopping for benefit
– selective outcome reporting bias

25. Factors that may decrease the
quality of evidence
 Study limitations (risk of bias)
 Inconsistency among studies
 Indirectness of evidence
 Imprecise results
 Reporting bias

26. Evidence synthesis (systematic review)
P
I
C
O
Outcome
Outcome
Outcome
Outcome
Critical
Important
Critical
Not
Summary of findings
& estimate of effect
for each outcome
Rate
overall quality of evidence
across outcomes based on
lowest quality
of critical outcomes
RCT start high,
obs. data start low
1. Risk of bias
2. Inconsistency
3. Indirectness
4. Imprecision
5. Publication bias
GradedownGradeup
1. Large effect
2. Dose
response
3. Confounders
Very low
Low
Moderate
High

27. GRADE: a 3 pillars approach
1. Formulate the question, choose and rate
your outcomes of interest and perform a
systematic review (quality of evidence)
2. Risk benefit evaluation, consider patients
values and preferences and also
resource use and feasibility
3. Strength of the recommendation

28. Determining the benefit risk profile:
positive/uncertain/unfavourable
Factors Impact on the strength of a
recommendation
Balance between
desirable and
undesirable
effects
Larger the difference between the
desirable and undesirable effects, more
likely a favourable benefit But
differences can arise depending on the
severuty of adverse events
Values and
preferences
More variability in values and
preferences, or more uncertainty in
values and preferences, more likely an
unfavourable profile.
Costs (resource
use)
Higher the costs of an intervention – that
is, the more resources consumed – less
likely a favourable profile.

29. GRADE Step 2: risk benefit profile,
values and preferences (1/3)

30. GRADE Step 2: risk benefit profile,
values and preferences (2/3)

31. GRADE Step 2: risk benefit profile,
values and preferences (3/3)

32. GRADE: a 3 pillars approach
1. Formulate the question, choose and rate
your outcomes of interest and perform a
systematic review (quality of evidence)
2. Risk benefit evaluation, consider patients
values and preferences and also
resource use and feasibility
3. Strength of the recommendation

33. Strength of recommendation
The degree of confidence that the
desirable effects of adherence to a
recommendation outweigh the undesirable
effects.
Desirable effects
•health benefits
•less burden
•savings
Undesirable effects
•harms
•more burden
•costs

34. Categories of recommendations
Although the degree of confidence is a
continuum, we suggest using two
categories: strong and weak.
 Strong recommendation: the panel is
confident that the desirable effects of
adherence to a recommendation
outweigh the undesirable effects.
 Weak recommendation: the panel
concludes that the desirable effects of
adherence to a recommendation
probably outweigh the undesirable
effects, but is not confident.
Recommend
 
Suggest
 

35. Why Grade Recommendations?
 Strong recommendations
– strong methods
– large precise effect
– few down sides of therapy
 Weak recommendations
– weak methods
– imprecise estimate
– small effect
– substantial down sides

36. Evidence synthesis (systematic review)
Making recommendations (guidelines)
P
I
C
O
Outcome
Outcome
Outcome
Outcome
Critical
Important
Critical
Not
Summary of findings
& estimate of effect
for each outcome
Rate
overall quality of evidence
across outcomes based on
lowest quality
of critical outcomes
RCT start high,
obs. data start low
1. Risk of bias
2. Inconsistency
3. Indirectness
4. Imprecision
5. Publication bias
GradedownGradeup
1. Large effect
2. Dose
response
3. Confounders
Very low
Low
Moderate
High
Formulate recommendations:
•For or against (direction)
•Strong or weak (strength)
By considering:
Quality of evidence
Balance benefits/harms
Values and preferences
Revise if necessary by considering:
Resource use (cost)
• “We recommend using…”
• “We suggest using…”
• “We recommend against using…”
• “We suggest against using…”

37. Contents of the presentation
• What are the defects of existing guidelines and
systems of grading
• Why GRADE could help …
• A three pillar method: the GRADE system to
evaluate quality of evidence and define the
strength of a recommendation
• Three examples
• Conclusions

38. Recommendations using
GRADE: Example 1
A flexible method:
quality of evidence independent
from strength of recommendation

39. WHO avian flu guideline 2006
Schünemann HJ et al. Lancet Infect Dis 2007;7:21-31

40. For opioid agonist maintenance treatment, most patients
should be advised to use methadone in adequate doses
in preference to buprenorphine.
– Strength of recommendation – Strong
– Quality of evidence – High
WHO Guidelines for the Psychosocially
Assisted Pharmacological Treatment of
Opioid Dependence (2009)
On average, methadone maintenance doses should be in
the range of 60–120 mg per day.
– Strength of recommendation – Strong
– Quality of evidence – Low

41. Recommendations using
GRADE: Example 2
Taking into account values and
preferences … and local context

42. Values and preferences
Stroke guideline: patients with TIA clopidogrel
over aspirin (Grade 2B).
Underlying values and preferences: This
recommendation to use clopidogrel over
aspirin places a relatively high value on a
small absolute risk reduction in stroke rates,
and a relatively low value on minimizing drug
expenditures.

43. Values and preferences
peripheral vascular disease: aspirin be
used instead of clopidogrel (Grade 2A).
Underlying values and preferences: This
recommendation places a relatively high
value on avoiding large expenditures to
achieve small reductions in vascular
events.

44. Recommendations using
GRADE: Example 3
Weak recommendations …
a blurred vision or a clear one?

45. Recommendations and expected adoption rate
Strength
Definition and implications
Expected
adoption
rate
Strong
positive
The drugs/interventions should offered to the vast majority of
patients and could be used as an indicator of good quality of care
> 60-70%
Weak
positive
It has the wider range of uncertainty since it could mean only for a
minority of patients (30%) or for a good proportion of them (50-
60%). It is necessary to inform patients of the expected benefits
and risks (and their magnitude), explore patients values and
discuss potential alternative treatments
30-60%
Weak
negative
In selected cases or a defined minority. The decision should go
along with a detailed information to patient of the benefit risk profile
(magnitude), patients values and expectations and the presentation
of potential alternative treatments
5-30%
Strong
negative
It should not be used neither routinely nor for a subgroup. Only in
few very selected (and documented) cases can be used since the
benefit/risk balance is negative/unknown and available alternative
are preferable
< 5%

46. Contents of the presentation
• What are the defects of existing guidelines and
systems of grading
• Why GRADE could help …
• A three pillar method: the GRADE system to
evaluate quality of evidence and define the
strength of a recommendation
• Three examples
• Conclusions

47. GRADE … in short
• Have an overall view of the process (see WHO), a good-
enough mandate and some governance of relevant CoI
• Make just a few (a reasonable number of)
recommendations
• Use systematic reviews (if not available, review key,
accessible evidence) – DO NOT meta-analyse if not
done
• Use GRADE criteria for quality of evidence
• Explain the reasons supporting the strength of
recommendations, including the benefit/risk profile and
values and preferences
• … just be (more) transparent

48. Where
am I?
You’re 30
metres
above the
ground in a
balloon
You must
be a
researcher
Yes.
How
did you
know?
Because what
you told me is
absolutely
correct but
completely
useless
You must
be a policy
maker
Yes,
how did
you
know? Because you
don’t know where
you are, you
don’t know where
you’re going, and
now you’re
blaming me
from: Jonathan Lomas, 2008