This presentation describes approaches to evaluating the effectiveness of risk minimisation programmes.

1. Risk minimisation activities – measuring effectiveness
Dr Jane Cook
Branch Head
Pharmacovigilance and Special Access Branch
11 November 2015

2. Risk Minimisation Programme
• You have identified the risks to be mitigated.
• You have selected the tools (activities) to be used in the programme.
• You now need to ensure the activities will meet their objectives and be successful
in mitigating the identified risks.
• You now need to develop an evaluation framework or protocol.
1

3. Areas to consider when developing a framework
• The framework/protocol should be available prior to the start of the
program
• The framework/protocol should provided in the Risk Management
Plan
2

4. Areas to consider when developing a framework
Area Description
Structure ü Reason for and description of the interventions and tools and their objectives
Design ü Key aspects, including any comparison, and duration and measures of success
Process
indicators
ü Those to address adequacy of the content
ü Those to assess adequacy of coverage, utilisation and maintenance
Outcome
indicators
ü Knowledge and awareness, attitudes, actual behaviours – depend on nature of activity
ü Those related to patient outcomes such as avoidance of C/I drugs, morbidity or mortality
measures
Analyses ü How measure of performance will be undertaken against pre-identified measures of success
ü When will data be collected
ü What type of analyses will be performed
3

5. Programme evaluation
• Need to measure effectiveness of overall programme as well as each individual component
• Indicators selected to evaluate the performance of the programme need to be:
– Relevant
– Well-defined
– Sensitive
– Reliable
– Evidence-based
– Objective
– Tailored to the programme
4

6. Evaluation method
• Comprehensive evaluation of overall effectiveness should involve measuring
performance in several aspects or domains.
• One methodology is the RE-AIM framework:
– Reach
– Efficacy/effectiveness
– Adoption
– Implementation
– Maintenance
5

7. Evaluation method: RE-AIM
Dimensions Definition
Reach Number and representativeness of participants selected for inclusion in evaluation
the protocol
Efficacy/effectiveness What are the changes expected – behavioural, patient outcome have these any
negative consequences?
Adoption Number and representativeness of those that agree to participate – do these
continue to be representative – bias?
Implementation The degree to which the intervention has been delivered as intended (could also
consider cost)
Maintenance The extent to which the intervention has been delivered overtime as intended
6

8. How did the intervention succeed in the
“real world”
• Implementation fidelity (how closely did the actual program follow the
framework/protocol?)
• Poor design or poor implementation?
• Good outcomes; could they be better?
7

9. Evaluating implementation fidelity
Area Description Examples of assessment methods
Exposure Were all aspects delivered to the
target recipients?
Survey, review administrative records of delivery of elements to
participants. Samples need to be representative.
Content Was what delivered the same as
originally designed? Was the
interpretation of the materials as
expected? I.e. was messaging “fit
for purpose”?
Focus groups.
Frequency Was it delivered consistently at
the frequency in the initial
activities plan?
Stakeholder survey, patient clinical reviews, drug utilisation
study.
Duration Was it delivered consistently
throughout the activity period?
Sampling as above.
8

10. Evaluating effectiveness of the risk minimisation
programme
• Outcome measures should be:
– Relevant (specific, measurable, time-bound)
– Consistent with overall aim of the risk minimisation programme
– Meaningful from both clinical and public health perspective
9

11. Types of suitable outcome measures
• Direct vs. indirect outcome indicators
– Direct (hard clinical endpoint)
– Indirect (surrogate marker)
• Single vs. composite outcome indicators
– Single
– Composite (consists of multiple single endpoints; can reduce sample size)
§ Example: hepatotoxicity vs. liver failure vs. hepatitis
10

12. Characteristics of suitable outcome measures
Characteristic Description
Preventable or mitigatable
Specific Clearly defined and requires little judgement
Easy to diagnose/detect Easy to identify and confirm
Rich in information Consider periodic rather than one off sampling
Responsive Is sensitive the proposed activity
Reliable Precise, reproducible, unlikely to be variable on repeated measures
Internally validity Linked to use
External validity Generalisable to general population
Clinical relevance Likely to influence treatment choices
Practical Easily implementable and low cost
11

13. Evaluation designs
• Comparator – can be difficult where has been required as part of registration.
• Staggered roll out.
• Before and after comparison, if available.
• Trend analysis through interrupted time series (periodic sampling) and use of
other existing databases e.g. health claims data.
• Should confirm persistence and maintenance of the programme impact over time.
12

14. Other considerations
• When would evaluation cease?
• When could the risk minimisation activity cease?
13

15. Summary
• Life-cycle approach
• Evaluation framework as part of development of the activity
• Evaluation of the parts and the whole of the risk minimisation programme
• Should be simple, pragmatic and user friendly to prevent undue clinician and
patient workload
• Measures should be clearly defined, easy to use, reproducible and practical
• Risk management programme should aim to in the real world ensure that “the
right prescriber provides the right medicine to the right patient at the right dose
and right time”.
14

16. Summary
From: Practical approaches to risk minimisation for medicinal products: Report of CIOMS Working Group IX.
Geneva, Switzerland: Council for International Organizations of Medical Sciences (CIOMS), 2014.
15