1. MAST: the Model for Assessment
of Telemedicine
and its application in the
RENEWING HEALTH Project
ANNA KOTZEVA, on behalf of the MAST Working group
Catalan Agency for Health Information, Assessment and Quality, Spain
MADoPA Seminar, June 2012, Paris

2. Contents
 Why do we need a global evaluation framework?
 How does MAST ensure such multidisciplinary assessment?
Application of MAST in the RENEWING HEALTH project
1. Project overview
2. Outcomes assessed
3. MAST tools to support study design, reporting and analysis
2

3. Why do we need
a global evaluation framework?

6. How does MAST ensure a
multidisciplinary assessment?

7. Comprehensive development
 MethoTelemed project, financed by the EC (SMART 2008/0064)
 Based on:
MAST
(Manual & Toolkit)

8. Assessment defined as:
 A multidisciplinary process that summarizes and evaluates
information about the clinical, economic, organizational and
socio-ethical issues related to the use of telemedicine, in a
systematic, unbiased and robust manner.
Approach

9. The framework suggests 3 stages of assessment:
I. Preceding considerations
II. Multidisciplinary Assessment
III. Transferability of results
Stages and elements

10.  This stage is about “setting the context”
 Various aspects have to be taken into consideration, such as:
 Maturity of the technology used
 Selection of appropriate comparator/s
 Level, on which the assessment should be carried out – single
hospital, region, state
 Existing legislation on TM-based services
 ….
I. Preceding considerations

11. WHAT should be assessed?
1. Health problem and characteristics of the application
2. Clinical effectiveness
3. Safety
4. Patient perspectives
5. Economic aspects
6. Organisational aspects
7. Socio-cultural, ethical and legal aspects
II. Multidisciplinary assessment
Descriptive
Descriptive
Evaluation by outcome
(systematic review OR
empirical study)
DOMAINS

12. HOW it should be assessed?
 Study design - Aim for highest possible level of evidence
 Outcome measures should be:
• patient-important outcomes
• based on scientific literature to enable comparison
• validated instruments
II. Multidisciplinary assessment

13.  Answers the question:
Can results be generalized to other settings?
This consideration has to be made within each of the domains!
 Examples of limitations for direct transferability of results:
 Country-specific reimbursement conditions
 Healthcare price/hour
 Values and culture influence user preferences and hence, satisfaction
with the new service
III. Transferability assessment
?

14. Application of MAST
in the RENEWING HEALTH project
 Project overview
 Outcomes assessed
(based on MAST and adapted to each clinical condition
and to the specific context)
 MAST tools to support design, reporting and analysis
(ensure quality and comparability)

15. Application of MAST
in the RENEWING HEALTH project
 Project overview
 Outcomes assessed
(based on MAST and adapted to each clinical condition
and to the specific context)
 MAST tools to support design, reporting and analysis
(ensure quality and comparability)

16. RENEWING HEALTH is the largest pragmatic
randomized clinical trial in the area of telemedicine
7158 patients
84 centers
21 pilots
10 clusters
3 diseases
1 observ.
9 RCTs
DM
COPD
CVD
DM: Diabetes mellitus; COPD: Chronic Obstructive Pulmonary Disease; CVD: Cardio-vascular Disease

17. Cluster 4 COPD
Short-term follow-up after
hospital discharge
Cluster 3 Diabetes
Ulcer monitoring
9 European regions involved
Cluster 1 Diabetes
Medium-term health coaching
and life-long monitoring
Cluster 2 Diabetes
Life-long monitoring
Cluster 5 COPD
Life-long
monitoring
Cluster 6 CVD
Medium-term health
coaching and life-long
monitoring
Cluster 7 CVD
Remote monitoring of
Congestive Heart Failure
Cluster 8 CVD
Remote monitoring of
implantable cardiac
devices
Cluster 9/10 Multi
pathology
Monitoring of frail patients
with chronic diseases
Cluster 10/11 CVD
Medium-term health
coaching and life-long
monitoring in CVD with
high blood pressure

18. General objective:
To produce high level evidence and decision support for EU
health policies and regional authorities regarding the future
deployment of Telemedicine services in those fields where they
can lead to improved care and reduced cost.
18

19. COPD (cluster 4 - Catalonia)
Data transmission
Data access
General Practitioner
Hospital & EPR
Hospital (Hospital Clínic & Hospital Mataró)Patient’s situation (at home)
Patient
Videoconference System & Patient
Portal
Telemonitoring
devices
Clinical Patient
Management System
NurseHospital
video
interface
Interface &
Gateway
Case Manager
Nurse
Hospital
Specialist
Call Centre &
Health coach

20. Telemedicine services have been piloted in many clinical areas in the
last 2 decades, but not always have been integrated into practice
Moreover, clinical outcomes of telemedicine on diabetes, COPD and CVD
have been measured in numerous trials internationally. And soon the
results of Whole System Demonstrator program will be available
RENEWING HEALTH Project is not envisioned to produce scientific
evidence on clinical outcomes of telehealth, BUT
a broader knowledge on benefits of telemedicine services through a
number of multidisciplinary outcomes

21. Application of MAST
in the RENEWING HEALTH project
 Project overview
 Outcomes assessed
(based on MAST and adapted to each clinical condition
and to the specific context)
 MAST tools to support design, reporting and analysis
(ensure quality and comparability)

22. Clinical
 Health-related quality of life
improvement measured via SF-36v2®
 Mental health status
measured via Hospital Anxiety and Depression Scale (HADS)
 Other specific clinical outcomes
for ex. FEV1 in COPD
FEV1: Forced Expiratory Volume in the 1st second; COPD: Chronic Obstructive Pulmonary Disease;

23. Assessed through a common questionnaire
Service User Technology Acceptability Questionnaire (SUTAQ)
• self-administered, generic, especially designed for telehealth and telecare users
• 22-items organized in 6 domains, Likert response scale
• During the Whole System Demonstrators program in UK was submitted to 3500
patients
• In RH, translated and validated*into 10 European languages
• Pilots in RH and WSD can be compared
*Collaboration Agreement with UCL, Prof. Stan Newman
Increased
accessibility
Privacy &
Discomfort
Substitution
to usual care
Satisfaction
Care
personnel
concerns
Enhanced
care
□ □ □ □ □ □
STRONGLY
AGREE
MODERATELY
AGREE
MILDLY
AGREE
MILDLY
DISAGREE
MODERATELY
DISAGREE
STRONGLY
DISAGREE
User perspective - patient

24. User perspective - professional
Assessed using qualitative research methods (sub-study)
 Semi-structured interviews, focus groups
 Themes investigated:
 Expectations
 Experiences
 Satisfaction with the new service
 Change in roles in terms of responsibilities and tasks
(nurse, physician)
 Perceived impact on patient-professional communication

25. Economic
Focus on reduction in healthcare cost due to:
 Progressive shift from hospital to primary, community and
homecare
 Increasing role of patients and informal caregivers in the care
process
Cost of investments in the telemedicine application
Running costs of delivering telemedicine service and the comparator (patient
level)
- Time used by the staff
- Time used by the patient
- Use of utensils, devices etc.
Each patient's use of health care service (patient level)
- Number of admissions
- Number of bed days
- Number of GP visits
- Number of visits to emergency department
Reimbursement of the telemedicine service (business case)

26.  Impact on PROCESSES
 Workflow: Effects on number of patients treated, procedures performed etc.
 Staff: Changes in distribution of work (task shifting)
 Resources: Changes in working hours for each profession
 Training: Time spent on training to learn to use the application
 Internal communication
 External communication
 Impact on STRUCTURE
 Description and number of units collaborating in the production of the service
 Changes in organisation of generalist and specialist tasks
 Changes in geographical spread
 Changes in time spent on travel
 Impact on CULTURE
 Staff attitudes towards the application
 Staff experiences with the use of the application
Data used from:
- economic data
- description of intervention
- interviews with staff
Organizational

27. Application of MAST
in the RENEWING HEALTH project
 Project overview
 Outcomes assessed
(based on MAST and adapted to each clinical condition
and to the specific context)
 MAST tools to support design, reporting and analysis
(ensure quality and comparability)

28. Application of MAST in practice stimulates its further
development and improvement
A number of common templates were developed as a
tool to ensure the quality of the data and the analysis:
 Minimum dataset of common outcome measures
 Common template for Case Report Form (CRF)
 Guidance on coding and monitoring of collected data
 Guidance on data analysis and reporting of results within each
domain

29. 1
• Minimum dataset of COMMON
OUTCOMES
2
• Compatible databases with
COMMON CODING OF VARIABLES
3
• Monthly monitoring of data
quality for each pilot
Agreement among pilots
Ensures quality and comparability of data

30. Common plan for analysis and reporting
Guide for analysis and reporting of results within each domain:
1. Health problem and characteristics of the application
2. Safety (adverse effects)
3. Clinical effectiveness
4. Patient perspectives
5. Economic aspects
6. Organisational aspects
7. Socio-cultural, ethical and legal aspects
Based on CONSORT recommendations*
Based on validation studies by WSD
Based on guide by Drummond et al 2005
Based on HTA Handbook (2007)
*CONSORT for pragmatic trials Zwarenstein et al. (2008)

31. Example for COPD
1. Health problem and characteristics of the application
2. Safety
3. Clinical effectiveness
4. Patient perspectives
5. Economic aspects
6. Organisational aspects
7. Socio-cultural, ethical and legal aspects
• Disease description
(epidemiology)
• Description of the
technology used
(videoconference)
Cost-effectiveness analysis, business case
Interviews with all professionals involved
Register of adverse events
FEV1, Quality of life, Mental health
Number of readmissions (PRIMARY OUTCOME)
Questionnaire SUTAQ (satisfaction, acceptability)
Synthesis and description of the regulatory
framework in Catalonia

32. Advantages and limitations
 Advantages of MAST
 Multidisciplinary and very comprehensive
 Based on scientific evidence, on criteria for quality and on
stakeholders’ needs
 Establishes a common standard for all stages of study development
(design, data collection, analysis and reporting)
 Limitations of MAST
 Time consuming
 Only relevant in assessment of mature telemedicine applications
 Doesn’t give indications for prioritization when resources are limited
 More operational criteria are needed
MAST 2.0ON THE WAY TO

33. By validating MAST in 21 different settings,
RENEWING HEALTH has given huge boost to establish
the first widely accepted methodology for the
evaluation of complex telemedicine interventions.
In conclusion…

34. www.renewinghealth.eu
Thank you!
Anna Kotzeva
akotzeva@gencat.cat