1. Good Practices for High Quality Decision
Analytic Models
Stephanie R. Earnshaw, RTI Health Solutions, RTP, NC
Joe Gricar, Independent Consultant, New York, NY
AMCP Seattle, WA: April 2006
LEADING RESEARCH…
MEASURES THAT COUNT
2. 2
Workshop Session Plan
What is decision-analytic modeling?
When it is appropriate to use modeling techniques?
What modeling standards/guidelines exist?
How do you translate information into decisions?
Summary of key points / findings
3. 3
Role of Decision-Analytic ModelingRole of Decision-Analytic Modeling
in Healthcare Decision-Makingin Healthcare Decision-Making
4. 4
What is a Decision Analytic Model?
A decision analytic model is:
Set of calculations laid out in a logical
sequence, and
Informs a decision process – it is not purely to
arrive ‘perfect’ scientific answers
Modeling should be used as a
decision aid
Its goal should be to provide the
decision-maker with information
that can allow them to judge
5. 5
What is a Decision Analysis Model?
Models are used when exact information/data is not
available
Decision modeling …
Synthesizes information from multiple sources
Sources: RCTs, observational studies, claims data,
expert opinion, preference studies…
Estimates clinical and economic consequences
Costs: drug, treatment, adverse events
Outcomes: life years saved, avoided secondary events
Acts as a conceptual framework to bring the data
together
6. 6
When to Use Decision-Analytic Modeling?
Examples of when a model is appropriate:
Treatment selection – To examine a large number of
potential treatment / management options which may vary by
setting
Patient selection – To extrapolate to a broader patient
population than used in available studies
Time periods – To extrapolate to a longer-term treatment time
horizon
Uncertain evidence base – To consider variation in effect
size, inadequate power, confounding variables, or data
sources
Flexibility - To develop analyses to cover alternative
healthcare settings / country-specific analysis
Timing and cost – Decision modeling can be performed at a
relatively low cost and results can be obtained quicker than
primary data-collection studies
7. 7
Challenges of Decision-Analytic Modeling
Economic evaluation is a standard requirement for many
reimbursement / review systems
National Institute of Clinical Excellence (NICE) – United Kingdom
Canadian Coordinating Office for Health Technology Assessment
(CCOHTA)
Pharmaceutical Benefits Advisory Committee (PBAC) - Australia
Academy of Managed Care Pharmacy (AMCP) – United States
Result: Modeling is accepted as a valid analytical
approach to performing economic evaluations
“It may be necessary to carry out an appraisal before the best quality
outcome data are available. In these circumstances modeling is
appropriate to adapt the best available data to the problem being
addressed….” (NICE 2001)
8. 8
Advantages and Challenges of Modeling
Advantages
Represent a complex reality
Estimate performance under projected conditions
Minimize data collection
Challenges
Lack of standardization/quality control in the development and
reporting of models
Potential for ‘hidden’ assumptions
Variation in quality of data
Lack of transparency makes it difficult to check
Difficulty in communicating results in an effective and balanced
way to a non-modeling audience
In general these challenges are applicable for other types
of analyses (i.e., prospective studies, RCTs, etc)
10. 10
Standards/Guidelines on Healthcare Modeling
Governmental (Reimbursement)
• NICE (Submission)
• AMCP (Format v2)
• Canada
• Australia
• Netherlands
Evaluation Checklists (Peer)
Journal/Academic
• BMJ (Drummond)
• NEJM
• Gold papers
• Sonnenberg et al,
1994
• Sheffield Workshop
• Sculpher et al, 2000
• McCabe et al, 2000
ISPOR Task Force -ISPOR Task Force -
Good Research PracticesGood Research Practices
in Modeling Studiesin Modeling Studies
Milton Weinstien. Value in Health 2003 (6)1 9-17
11. 11
Guidelines: Structural Form
Relevance to decision-maker
Perspective: Who is the decision-maker?
Relevance to decision
Complexity and scope should have clear rationale and be
relevant to the decision making perspective
Relevance to disease
Health state definitions and linkages hold clinical validity in the
face of current understanding
Solid grounding in disease theory through clinical concurrence
ModerateModerateMildMild SevereSevere
12. 12
Guidelines: Structural Form
Full acknowledgement of assumptions
Time horizon reflects and captures major clinical events and
costs related to the disease or treatment
Appropriate treatment comparators
Include treatment options of immediate interest
Consider treatment options reflecting novel treatments and
extremes
Appropriate level of model memory
Variations in patient history, sub-groups, and attributes must be
included if they have a logical and expected impact on event
rates and resource use
Age-adjusted mortality
Increased risk of CHD as patient ages
13. 13
Guidelines: Structural Form
Appropriate methodology
Clear justification for model methodology and health
states
Decision tree
Markov
Simulation
Transparency: calculations are clearly presented
Strike a balance between model structure and data
availability
Effective
Antibiotic
Inadequate response
Ineffective
Intolerable adverse events
14. 14
Guidelines: Data Selection
Model should include clinical, economic, and
outcomes data that have relevance to the
decision at hand
or
FractureFractureNo FractureNo Fracture
No FractureNo Fracture
Wrist FractureWrist Fracture
Hip FractureHip Fracture
15. 15
Guidelines: Data Selection
Evidence must show that consideration of all data and
their values have been made
Full details on the importance of a
parameter and its value
Appropriate ranges and distributions
representing data uncertainty
Clear acknowledgement where expert
opinion is used
Nuijten MJ. The selection of data
sources for use in modeling
studies. Pharmacoeconomics.
1998 Mar;13(3):305-16.
16. 16
Guidelines: Data Preparation
Data is seldom available ‘off the shelf’ and ‘ready to go’
Data often requires adaptation, translation, or mapping to
other value scales
Transparent presentation on data preparation methods
Calculation of baseline risks
Calculation of relative risks
Calculation of transition probabilities
Survival statistical modeling (weibull,
exponential etc)
Costs
Cost inflation
Cost and outcome discounting
17. 17
Guidelines: Validation
We can not formally ‘test’ the quality and validity of a model
in its true sense.
Validation approaches
Quality assurance ‘debugging’ plan
Run head-to-head comparisons of structures, inputs, and outputs
with other models in same area
Direct comparison of model predictions to a known set of
independent outcome or resource data
Examination of uncertainty in model inputs and structure for
sensitivity analyses
‘Putting the model
under the microscope’
19. 19
Documentation
Internal model documentation
Summary / introduction page to provide overview
Model schematic to discuss flow of data through model
Discussion of model calculations provided
All calculations are clearly identified
Discussion of models to allow a lay person to understand
Help buttons
Accompanying documentation
User guide
FAQ’s
One-page “cheat sheet”
Discussion points (scenario-based examples)
20. 20
Documentation - Model Schematic
Dx Population
Hospitalization
No Event
Drug 1
2nd
Event No Event
No Tx
2nd
Event
21. 21
Documentation – Default to Results
Input / Default Page
Medical
Costs
Side Effects Rx CostsClinical
Outcomes
Step 1
Step 2
Intermediate
Table
Final Results
Intermediate
Table
Final Results
Intermediate
Table
Final Results
Intermediate
Table
Final Results
Default
Values
Results Page
22. 22
Documentation – Default / Input Data
Default Data and Input Data Page
Category Sub-Category Default Data Data used in Model
Health Plan Data
Number of Member lives 1,000,000 1,000,000
Incidence Rate of Disease 4.6% 4.6%
% Hospitalized 25.0% 25.0%
% of population +65 25.0% 25.0%
Pharmacy cost data
AWP Price ($) - 30 day $95.00 $95.00
Co-Pay $7.50 $7.50
Dispensing Fee $3.00 $3.00
Table 2: Outcome Data: Drug 1
Category Sub-Category Default Data Data used in Model
All
Primary Event Rate 10.5% 10.5%
2nd Event Rate 0.6327 0.6327
Under 65
Primary Event Rate 9.0% 9.0%
2nd Event Rate 0.4327 0.4327
Over 65
Primary Event Rate 12.3% 12.3%
2nd Event Rate 0.7327 0.7327
23. 23
Documentation – Calculations
Category Value
Analysis Type PMPM
Total Membership (N) 1,000,000
Member Months 9,000,000
DX Patients (Total) 11,500
Patient Rx Therapy N of PTs
Scenario 1 5,750
Scenario 2 11,500
Data Link: This is a link
from the Default page
Drug therapy Data Cost ($)
AWP Price ($) - 30 day $95.00
Co-Pay ($) $7.50
Dispensing Fee ($) $5.00
24. 24
Documentation – Calculations
COST CATEGORY
Drug therapy Data
Scenario 1 Scenario 2 DIFFERENCE
TOTAL PLAN COSTS $2,587,500.00 $5,175,000.00 $2,587,500.00
AVG/PT $225.00 $450.00 $225.00
COST PER MEMBER $2.59 $5.18 $2.59
COST PMPM $0.29 $0.58 $0.29
COST CATEGORY
Drug therapy Data
Scenario 1 Scenario 2 DIFFERENCE
COST PMPM $0.29 $0.58 $0.29
Intermediate Table 2: Shows Cost information by 4 methods
1) Total Plan Costs
2) Avg/PT: Total plan Costs ÷ Number of Patients
3) Cost Per Member: Total plan Costs ÷ Number of Plan
Members
4) Cost PMPM: Total plan Costs ÷ Number of Plan Members
Months
Final Results: Data to be used in model
results based on user selection. This Data is
linked to Results Page.
Calculation: Adjusted Drug Cost
AWP Price ($) - 30 day $45.00
Minus co-pay $37.50
Plus Dispensing Fee (Total) $42.50
Intermediate Table 1: Calculation of Avg 30-day script
cost adjusting for co-pay and dispensing fee
26. 26
Transforming Information into Decisions:Transforming Information into Decisions:
Making models that are relevant to health plansMaking models that are relevant to health plans
27. 27
RCT
Study
Results
Nat. data on Prev. of Dx
(Race, Gender, Age)
Comorbid conditions (severity)
Age, Gender, Race, other
Assessment
of Direct medical costs
(Hosp., ER, Rx, etc.)
Indirect and intangibles costs
(QoL, Work loss, etc.)
Misc. information
(Compliance, persistency)
Population
based-model
Health Plan Specific data
Costs: Rx, Hosp/ER,)
Membership demo.
Expected change to HP
Plan specific model
for formulary DM
Transforming Information into Decisions
28. 28
Transforming Information into Decisions
What additional information is needed to make a
formulary decision?
Model results to “real world” setting
Age, gender and race
Patient severity (co-morbid conditions)
Prevalence of disease(s)
Compliance to drug therapy
Enrollment/dis-enrollment patterns
Economic assessment of reductions in “real world”
events
Translate information into “health plan” language
29. 29
Transforming Information into Decisions
Cost
QALY
Clinical Outcomes
•Event rate reduction
•Procedures avoided
•Rate of AE
Quality of Life
•Patient satisfaction w/care
•Compliance to Rx
•Discontinuation of Rx
•Rate of AE leading to switch
Economic Outcomes
•PMPM
•Utilization Rates
(Hosp./ER visits)
31. 31
Summary
Modeling should be used as a decision aid rather than to
defer decision making
Present elements of the decision to be made in a
focused, credible, structured, and transparent manner
Assist in making healthcare decisions by bringing
together all key influencing factors into consideration
Assist in prioritizing and handling uncertainty in key
outcomes
32. 32
Summary
‘Technical Elements’
Clinical Credibility
Disease Theory
Level of Data
Collection
Model Complexity
‘Artistic Impression’
Clarity to decision maker
Transparency
Appropriate to decision
Responsive to decision
timing
Ice Dancing
Analogy…………
Achieving a model that will be considered by peers to
be a ‘good’ model is about finding right balance in
scientific and decision credibility.
Notes de l'éditeur
Before we start, we would like to get a sense of the background that most of you in the audience have.
How many people are from managed care?
How many people are from pharma?
How many people have the responsibility of reviewing developed models?
How many people have the responsibility of developing models for products?
What other types of responsibilities do people have in the audience?
Basically, the aim of the session is to provide managed care decision-makers with the basic understanding of decision-analytic modeling in healthcare decision-making and important concepts around it.
Why do we have guidelines?
Decisions have to be made whether internal or external, model or no model, etc
How can we best inform these decisions using the most appropriate information available?
Specifically, the challenge is to support the development of a model…
fit for purpose
scientifically validity
decision-making credibility
transparent
‘Good practice’ guidelines can support quality assurance
Pragmatic approach is to establish a set of standards or criteria for rapid peer-review of models (as with RCTs)
Use these to ‘grade’ quality of models and identify potential improvements - to optimize decision quality
Published papers on model quality assessment
Sonnenberg FA, Roberts MS, Tsevat J, et al. Toward a peer review process for medical decision analysis models. Med Care. 1994 Jul;32(7 Suppl):JS52-64
Nuijten MJ. The selection of data sources for use in modeling studies. Pharmacoeconomics. 1998 Mar;13(3):305-16
Nuijten MJ, Starzewski J. Applications of modeling studies. Pharmacoeconomics. 1998 Mar;13(3):289-91
McCabe C, Dixon S. Testing the validity of cost-effectiveness models. Pharmacoeconomics. 2000; 17(5):501-13
Sculpher et al. Pharmacoeconomics 2000;17(5):461-77
Weinstien M. Value in Health 2003 (6)1 9-17 (ISPOR Workshop)
Types of recommendation and standards across ‘good practice’ guidelines
Structural form of the model (and justification of the structure)
Appropriateness and quality of data
Validation of the model (internal and external)
The model must have…
Relevance to decision-maker:
Ensure there is a clear decision and perspective to support
Relevance to decision:
Ensure that the complexity and scope have clear rationale and be relevant to the decision-making perspective
Relevance to disease:
Solid grounding in disease theory through clinical concurrence
Ensure that health state and linkages hold clinical validity in the face of current understanding
Full acknowledgement of assumptions and their relative importance on the model
Appropriate time horizon - reflect and capture major clinical events and costs related to the disease or treatment
Appropriate treatment comparators
Include treatment options of immediate interest
Consider treatment options reflecting other novel treatments and extremes (allows for context generalizability)
Appropriate level of model memory
Variations in patient history, sub-groups, and attributes must be included if they have a logical and expected impact on event rates and resource use
Age-adjusted mortality
Increased risk of fracture by BMD in osteoporosis
Increased risk of CHD as patient ages
Appropriate methodology
Clear justification for model methodology and health states
Decision tree
Markov
Simulation
State clearly why level of modeling has been adopted - allows clarity in methods and opens for peer review and challenge
Tranparency: calculations are clearly presented
Strike a balance between model structure and data availability
Beware of:
‘Pruning’ and over ‘simplification’
Over complicating the model to reflect ‘reality’
Application of blanket rules: Each situation is unique and requires different level of modeling and depends on both the disease and the decision to be made
Model should include clinical, economic, and outcomes data that have relevance to the decision at hand
Evidence must show that consideration of all data and their values have been made
Full details comprehensive and systematic approach - importance of parameter and the importance of its value
Clear justification for inclusion & exclusion of data items and values (‘optimal’ data set?) - hierarchy issues
Appropriate ranges and distributions representing data uncertainty
Clear acknowledgement where expert opinion is used
Details on approach used to gather expert opinion such as format of questions and consensus approach, selection criteria, sample size, and variation in opinion
Model should include clinical, economic, and outcomes data that have relevance to the decision at hand
Evidence must show that consideration of all data and their values have been made
Full details comprehensive and systematic approach - importance of parameter and the importance of its value
Clear justification for inclusion & exclusion of data items and values (‘optimal’ data set?) - hierarchy issues
Appropriate ranges and distributions representing data uncertainty
Clear acknowledgement where expert opinion is used
Details on approach used to gather expert opinion such as format of questions and consensus approach, selection criteria, sample size, and variation in opinion
Quality assurance ‘debugging’ plan
Confirm logical calculations and parameter values are applied correctly
Secondary review conducted by independent analysts
Confirm model matches results of data used to construct the model
Replicate model in alternative software
Run model at extreme values to observe direction of outcomes
In most cases, model should show some “negative” results
Run head-to-head comparisons of structures, inputs, and outputs with other models in same area
Highlight areas of divergence
Justify / explain reasons for differences
It is difficult: as often previous models are used as input to design of new model
In addition, we can not assume that previous models are more appropriate / better as they may have been designed for a different purpose
Direct comparison of model predictions to a known set of independent outcome or resource data
Challenges:
Data is not always available, comparable, or applicable
Data often collected for different purpose (not decision making) – flawed
Retrospective ‘validation’: New data may come post-model (model is intended to represent the best knowledge at the time of the decision)
However it is a useful exercise at the time of model development (pre-decision)
Identify differences in outcome patterns
Can lead to re-appraisal of model structure
Builds peer confidence in model
Appropriate representation of uncertainty in model inputs and structure for sensitivity analyses
Univariate (one-way) and multivariate (multi-way)
Probabilistic (Monte-Carlo)
Dependence / independence of input variables
Ability to replicate sensitivity analysis (fixed random seed)
Relevant comparisons - Choose between new interventions and routine care that differ between settings
Time periods – Extrapolate longer-term treatment impacts
Patient selection – Extrapolate to broader patient population than available studies
Scope of disease impact – Consider wider secondary clinical and economic impacts
Endpoint relevance – Consider economic meaning of primary clinical endpoints
Uncertain evidence base – Consider impact due to variations in effect size, inadequate power, or confounding variables
Treatment selection – Examine a large number of potential treatment / management options
Scoping – Consider impact of data from a range of disparate sources in a single framework
Flexibility - Develop analyses to cover alternative healthcare settings / country-specific analyses
Conceptualizing – Assess value of interventions from early concept to post-marketing
Timing and cost – Performed an analysis at a relatively low cost and quicker than performing primary data-collection studies
Discussion of Policy model vs. Health Plan model
Policy model
Static: no adjustments for different populations
Includes traditional C/E ratios and information
May include only a measure of QoL included, not separate
Usually only one time horizon (may not fit the health plan perspective)
Includes mostly clinical data.
Helps to drive the introduction of a new product / therapy or concept in care
Useful for development of thought leaders / overall agreement on the use of a treatment
Does not provide information on
“what will happen in my environment”
What will be the economic impact? Year 1, 2, 5 ?
What is the clinical impact? Year 1, 2, 5 ?
Why is this needed
Health plans do not believe the current information
Black box approach
Confusing (too much health economic terms) not relevant to day to day activities of health plan
Too many variables included in one ratio….what does $12,000/QALY mean to me?
QoL is not important / not provided in way to reflect issues relevant to health plan
We need to present data in usable form
Economics: Present both rational C/E ratios along with PMPM / Utilizations rates / hospital/ER/visits
Clinical: Discuss events reduction / avoided overall and by each individual event: Show effect of Rx on Adverse events that lead to medical treatment…
QoL: Useful but only in a tie: Again, what is a QALY mean to me? Is a difference in SF-36 scores mean I will lose patients, compliance will be an issue?
Provide QoL measures that are important to the health plan. This will get the drug approved. Traditional QoL allows health plan to sell itself to employers/patient groups and plan members.
Why is this needed
Health plans do not believe the current information
Black box approach
Confusing (too much health economic terms) not relevant to day to day activities of health plan
Too many variables included in one ratio….what does $12,000/QALY mean to me?
QoL is not important / not provided in way to reflect issues relevant to health plan
We need to present data in usable form
Economics: Present both rational C/E ratios along with PMPM / Utilizations rates / hospital/ER/visits
Clinical: Discuss events reduction / avoided overall and by each individual event: Show effect of Rx on Adverse events that lead to medical treatment…
QoL: Useful but only in a tie: Again, what is a QALY mean to me? Is a difference in SF-36 scores mean I will lose patients, compliance will be an issue?
Provide QoL measures that are important to the health plan. This will get the drug approved. Traditional QoL allows health plan to sell itself to employers/patient groups and plan members.