This document outlines guidelines for developing recommendations and appropriate use criteria for health technologies. It discusses using the GRADE framework to develop recommendations based on evidence synthesis. The RAND/UCLA method is described for developing appropriateness criteria when evidence is limited. Research recommendations are made to inform future research when evidence gaps are identified. The goal is to harmonize these procedures across Spanish HTA agencies for mutual recognition and transparency.

1. Zaragoza, 27th-28th April 2017
Guideline for the elaboration of recommendations
and appropriate use criteria in health
technologies.
Ruth Ubago Pérez
Andalusian Agency for Health Technology
Assessment (AETSA)
PUBLIC HEALTH CONFERENCE
10th anniversary of the Spanish Network of Health Technology Assessment (HTA) Agencies:
Insights for collaborative networking
Towards Patient and public engagement in HTA

2. BACKGROUND AND CONTEXT
METHODOLOGY
RESULTS
CONCLUSIONS
TABLE OF CONTENTS

3. BACKGROUND AND CONTEXT (I)
To harmonize procedures as a base for:
 mutual recognition
quality of work
To make those procedures and
methods explicit and public:
assuming a compromise with
transparency
quality in health technology
assessment (HTA )

4. BACKGROUND AND CONTEXT (II)
Health Technology Assessment as a support to the Decision
Making Process: from EVIDENCE to RECOMENDATIONS.
MAIN STEPS
EVIDENCE SYNTHESIS:
Formulate question
Summary of findings: efficacy/safety
RECOMMENDATION/DECISION

5. Sintetizar los
principios,
procedimientos y
metodología de la
elaboración de IRE en
el contexto de la Red
BACKGROUND AND CONTEXT (III)
Guideline for the elaboration
and adaptation of rapid health
technology assessment
reports. April, 2016.
Objectives:
To update methods and
processes.
To harmonize procedures.
EVIDENCE SYNTHESIS

6. Sintetizar los
principios,
procedimientos y
metodología de la
elaboración de IRE en
el contexto de la Red
BACKGROUND AND CONTEXT (IV)
Decision about:
What to do? What to cover?
Should this technology/device be
covered?
Should we stop doing a screening for a
cancer in asymptomatic men? In all
men?
RECOMMENDATION/DECISION

7. BACKGROUND AND CONTEXT (V)
MAKING RECOMMENDATIONS/DECISION
KEY POINTS
STAKEHOLDER INVOLVEMENT: policy makers, clinicians,
scientific societies and patients.
EXPLICIT AND TRANSPARENT PROCCES TO FORMULATE
A RECOMMENDATION/DECISION.

8. METHODOLOGY
 COORDINATION GROUP:
Avalia-t and AETSA.
WORKING GROUP: UETS-
ISCIII, Osteba, AQuAS, SCETS,
IACS, UETS Madrid.

9. 1. RECOMMENDATIONS
2. APPROPRIATE USE CRITERIA
3. RECOMMENDATIONS FOR FUTURE
RESEARCH
RESULTS

10. 1. RECOMMENDATIONS
GRADING OF RECOMMENDATIONS, ASSESSMENT,
DEVELOPMENT AND EVALUATION (GRADE) CRITERIAR
PURPOSE:
To inform decisions in the context of:
Clinical recommendations
Coverage decisions
Health system recommendations

11. WELL ESTABLISHED METHOD
GRADE methodology.
Practical experiences of application of GRADE at various
contexts: World Health Organization, HTA Agencies,
Scientific societies, Cochrane Collaboration.
1. RECOMMENDATIONS
DELIVERABLES SPANISH HTA REPORTS WITH GRADE
RECOMMENDATIONS
GRADING OF RECOMMENDATIONS, ASSESSMENT,
DEVELOPMENT AND EVALUATION (GRADE) CRITERIAR

12. MAIN STEPS:
PICO question: Setting perspective, Subgroups,
Background.
Assessment of evidence (GRADE methodology)
and judgements. DECIDE project. Evidence to
Decision (EtD) framework.
1. RECOMMENDATIONS
GRADING OF RECOMMENDATIONS, ASSESSMENT,
DEVELOPMENT AND EVALUATION (GRADE) CRITERIAR

13. DRAWING CONCLUSIONS/RECOMMENDATIONS
COVERAGE DECISIONS:
-Technology not covered.
-Covered only in the context of
research.
-Restricted coverage.
-Full coverage.
Direction and
strength
Justification
Subgroups
Implementation
Monitoring and
evaluation
Research priorities
1. RECOMMENDATIONS
GRADE CRITERIAR

14. •PANEL COMPOSITION IS A CRUCIAL STEP:
Patients, Policy makers, Clinicians, Scientific societies,
Methodologists.
•ADVANTAGES:
Consistent and transparent method.
How much outcomes are valued by all the stakeholders is
explicitely considered.
•POTENTIAL LIMITATION:
EtD frameworks are complex. Additional resources needed.
1. RECOMMENDATIONS
GRADE CRITERIAR KEY ISSUES

15. 1. RECOMMENDATIONS
2. APPROPRIATE USE CRITERIA.
3. RECOMMENDATIONS FOR FUTURE
RESEARCH.
RESULTS

16. 2. APPROPRIATE USE CRITERIA
PURPOSE:
To assess the
“appropriateness” of medical
procedures or services.
THE RAND/UCLA APPROPRIATENESS METHOD

17. 2. APPROPRIATE USE CRITERIA
DELIVERABLES SPANISH HTA REPORTS WITH
APPROPRIATENESS CRITERIA

18. THE RAND/UCLA APPROPRIATENESS METHOD
2. APPROPRIATE USE CRITERIA
WHEN TO BE USED
When scientific evidence is not
available or we cannot provide in
depth evidence sufficient to be
applied to the wide range of
patients seen in everyday clinical
practice.

19. Synthesis of the evidence
List of Indications/Clinical
Scenarios and Definitions
Criteria: Appropriate,
Uncertain, Inappropriate
Expert Panel
rates indications
in Two Rounds
MAIN STEPS
2. APPROPRIATE USE CRITERIA
THE RAND/UCLA APPROPRIATENESS METHOD
Core Panel

20. KEY ISSUES
ADVANTAGES:
1. Synthesis of published
literature prior to
consensus techniques
incorporated.
2. It allows for both
confidential ratings and
group discussion.
3. Multidisciplinary panel
encourages consensus
from a wider group.
POTENTIAL
LIMITATIONS:
1. Takes great deal of
time from gathering of
the evidence to
multiple rounds of
consensus.
2. Misclassification is
expected.
3. It lacks a clear rating
of the evidence.
2. APPROPRIATE USE CRITERIA
THE RAND/UCLA APPROPRIATENESS METHOD

21. 1. RECOMMENDATIONS
2. APPROPRIATE USE CRITERIA.
3. RECOMMENDATIONS FOR FUTURE
RESEARCH.
RESULTS

22. 3. RESEARCH RECOMMENDATIONS
PURPOSE:
To inform about primary biomedical research
after finding evidence gaps in the published
literature.

23. 3. RESEARCH RECOMMENDATIONS
EUnetHTA framework
Joint Action 2
Work package 7

24. WHEN TO BE USED
When finding EVIDENCE GAPS after developing HTA report.
“Missing evidence from a body of research that would
potentially answer the questions of decision makers. So final
decision cannot be made until these gaps are covered”.
3. RESEARCH RECOMMENDATIONS
FOR BIOMEDICAL PRIMARY RESEARCH
ADDITIONAL OBSERVATIONAL REAL WORLD DATA AFTER
COVERAGE FOR REINFORMING DECISION MAKING

25.  We have updated our methods and processes for
developing recommendations and appropriate use
criteria, as well as harmonizing all these procedures.
CONCLUSIONS

26. Zaragoza, 27th-28th April 2017
Thank you very much
PUBLIC HEALTH CONFERENCE
10th anniversary of the Spanish Network of Health Technology Assessment (HTA) Agencies:
Insights for collaborative networking
Towards Patient and public engagement in HTA
Follow us in
@AETSA_
www.aetsa.org