A technical presentation made at the eHealth Summer Semester in Braunschweg, Germany

1. eHealth Summer Semester:
International perspectives in
eHealth
Peter L. Reichertz Institute for Medical Informatics
University of Braunschweig - Institute of Technology and
Hannover Medical School

2. Evaluation of and evidence
for eHealth
Najeeb Al-Shorbaji,
Vice-President, e-Marefa
Director of Knowledge, Ethics and Research, WHO/HQ
(Retired)

3. Evidence on eHealth InterventionsEvidence on eHealth Interventions
What is eHealth?
•eHealth as the cost-effective and
secure use of information and
communication technologies in
support of health and health-related
fields, including health-care services,
health surveillance, health
literature, and health education.

4. Evidence on eHealth InterventionsEvidence on eHealth Interventions
Which eHealth application?
• Access to health information on the Internet
• Electronic health, medical or personal record
• Telemedicine, tele-health, tele-….;
• Disease surveillance, registry, monitoring, etc;
• Mobile based applications;
• eLearning;
• …..

5. Evidence on eHealth InterventionsEvidence on eHealth Interventions
What is evidence?
•Information that is collected in an
orderly way about a disease or its
treatment. This information often comes
from research. Evidence helps doctors
and scientists understand what
treatments work best on different
diseases. ww.cdc.gov/cancer/lung/glossary.htm

6. Evidence on eHealth InterventionsEvidence on eHealth Interventions
Evidence-Based Medicine
• Evidence-Based Medicine is the conscientious,
explicit and judicious use of current best evidence in
making decisions about the care of individual patients.
The practice of evidence-based medicine means
integrating individual clinical expertise with the best
available external clinical evidence from systematic
research.
www.mcg.edu/som/fmfacdev/fd_ebmconcepts.htm

7. Evidence on eHealth InterventionsEvidence on eHealth Interventions
Does EBM apply to eHealth?
‘‘health information systems should be
evaluated with the same rigor as a new drug or
treatment program, otherwise decisions about
future deployments of ICT in the health sector
may be determined by social, economic,
and/or political circumstances, rather than by
robust scientific evidence’’
Catwell L, Sheikh A (2009) Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Med 6:
e1000126. doi:10.1371/journal.pmed.1000126.

8. Evidence on eHealth InterventionsEvidence on eHealth Interventions
Is it a social science?
• The Summary Care Record (SCR) and HealthSpace programmes
spanned a number of different ‘worlds’ – political, clinical,
technical, commercial, academic – with different institutional
logics, as well as the personal world of the patient.
• Differences in norms, values, priorities and ways of working
between these six worlds, and imperfect attempts to bridge these
differences, accounted for much of the instability in the socio-
technical network – and this in turn explained many of the
challenges and frictions encountered as the complex collaborative
tasks of design, implementation, governance, front-line use and
evaluation were pursued.
Greenhalgh T, J, et al. (2010) The devil’s in the detail: final report of the independent evaluation of the Summary Care Record and
Health Space programmes. London: University College London. (www.ucl.ac.uk/news/scriesummary.pdf)

9. Evidence on eHealth InterventionsEvidence on eHealth Interventions
Potential and promise of eHealth
• eHealth Reduces cost and increases efficiency;
• eHealth Improves quality of service and patient
safety;
• eHealth Increases equity of access to health
information and services;
• eHealth Empowers individuals and enhances
accountability and transparency.

10. Evidence on eHealth InterventionsEvidence on eHealth Interventions
Summary Care Record (England):
as an example
Policy documents published in 2005-8 anticipated a
number of benefits of the SCR, including:
1.Safer care;
2.More efficient care;
3.Better care;
4.More equitable care;
5.Reduction in onward referral;
6.Greater patient satisfaction.
http://www.connectingforhealth.nhs.uk/systemsandservices/scr

11. Evidence on eHealth InterventionsEvidence on eHealth Interventions
•Technical and operational aspects of the
programme could not be meaningfully
isolated from subjective and contextual
issues such as what the introduction of the
SCR meant politically, professionally,
practically, and personally.
T. Greenhalgh. Adoption and non-adoption of a shared electronic summary record in
England: a mixed-method case study. BMJ 2010;340:c3111
(http://www.bmj.com/content/340/bmj.c3111.full)
Technology is only part of eHealth

12. Evidence on eHealth InterventionsEvidence on eHealth Interventions
Investment based on potential
• 2010-The Massachusetts Technology
Collaborative (MTC) and the New England
Healthcare Institute (NEHI) released a report
on December 1 that shows tele-ICU
technology could save 350 additional lives,
benefit hospitals financially, and save more
than $122 million annually if broadly and
effectively implemented across
Massachusetts
(http://www.masstech.org/ehealth/index.html)

13. Evidence on eHealth InterventionsEvidence on eHealth Interventions
Real money: big investments
• England National Programme for Information
Technology, US$ 20.6 billion (2003-2011);
• US Health Information Technology for Economic and
Clinical Health Act, US$ 20 billion (2011-2015);
• Australia: design, building and national rollout of a
personally controlled electronic health record US$ 473
million (2010-2011);
• Canada Health Infoway implementation program has
allocated US$ 2.16 billion (by 2010).

14. Evidence on eHealth InterventionsEvidence on eHealth Interventions
Why eHealth research?
• Build the knowledge base for eHealth;
• Establish evidence that aims and objectives of a an
intervention have been achieved, not achieved or
partially achieved;
• Introduce changes/improvements on the intervention
based on the available evidence;
• Establish value chain, cost-benefit, and return on
investment to support policy & decision making;
• Scale up, replicate, learn, improve in other settings.

15. Examples of evidence

16. Evidence on eHealth InterventionsEvidence on eHealth Interventions
eHealth IMPACT (eHI) study “eHealth is Worth It”
• The qualitative benefits were better informed patients and
carers, better focused information allowing for more streamlined
procedures, improved timeliness, improvements in safety, better
effectiveness, improved access to information and greater
efficiency. Timeliness, effectiveness and efficiency were
prevalent at all ten sites.
• For the economic benefit, when all ten cases are presented as
one virtual health economy, the results are economically
persuasive.
• ICT alone will never bring about change without substantial changes to
clinical and working practices, and the findings of the eHI study
highlight some important policy and management recommendations.
• Source: www.ehealth-impact.org

17. Evidence on eHealth InterventionsEvidence on eHealth Interventions
UK: Telehealth can deliver
• The early indications show that if used correctly
telehealth can deliver a 15% reduction in A&E
(Accident & Emergency ) visits, a 20% reduction
in emergency admissions, a 14% reduction in
elective admissions, a 14% reduction in bed days
and an 8% reduction in tariff costs. More strikingly
they also demonstrate a 45% reduction in mortality
rates.
From: Whole system demonstrator programme: Headline findings - December 2011
http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/D
H_131684

18. Evidence on eHealth InterventionsEvidence on eHealth Interventions
eIMCI Implementation
• Many of the limitations to proper
implementation of IMCI can be addressed by
electronic support tools. Indeed, the results
from a quantitative and qualitative study of
eIMCI show the impact of electronic support
tools on the adherence to clinical protocols,
consistency in care across providers, and
patient perceptions on quality of care.
H. Bethany. Evidence of eHealth Impact: The case for eIMCI in Tanzania
(http://www.dbmi.pitt.edu/seminar/evidence-ehealth-impact-case-eimci-tanzania)

19. World Health Bulletin Theme Issue on eHealth:
building the evidence base
• Call for papers in November
2011;
• Published on 1 May 2012;
• Received over 90 submissions;
• Published 14 studies articles;
• Rigorous peer review.

20. Evidence on eHealth InterventionsEvidence on eHealth Interventions
Highlights from the "Theme Issue"
• eHealth development must be holistic, evidence based, and
people centered;
• Improved collaboration between telemedicine networks could
help attenuate the lack of resources…and improve
sustainability;
• Literature review shows that eHealth systems of three types:
systems facilitating clinical practices, institutional systems and
systems facilitating care at distance;

21. Evidence on eHealth InterventionsEvidence on eHealth Interventions
Highlights from the "Theme Issue"
• Mobile health can enable behavior change and
improve health outcomes in resource-limited
settings;
• Real challenge for the development for eHealth lies
in establishing country level best practices that are
both cost-effective and supported by rigorous
research and evaluation;
• There is need for more research to continue to
prove that eHealth can result in economic benefits
and improves health outcomes.

22. Evaluation of eHealth projects

23. What is evaluation?
• An evaluation is a systematic and impartial assessment of an
activity, project, programme, strategy, policy, topic, them,
operational area or institutional performance. It focuses on
expected and achieved accomplishments, examining the
results chain, processes, contextual factors and causality, in
order to understand achievements or the lack of.

24. Why evaluation?
• Catwell and Sheikh argue that “health information systems
should be evaluated with the same rigor as a new drug or
treatment program, otherwise decisions about future
deployments of ICT in the health sector may be determined
by social, economic, and/or political circumstances, rather
than by robust scientific evidence”
[*]Catwell L, Sheikh A (2009) Evaluating eHealth interventions: the need for continuous systemic
evaluation. PLoS Med 6: e1000126. doi:10.1371/journal.pmed.1000126.

25. Different dimensions of evaluation
•Qualitative or quantitative;
•Patient or system;
•Formative (of learning) or summative (for
learning);
•Internal or external

26. What are the challenges?
• Greenhalgh at al believe that "eHealth “interventions” may
lie in the technical and scientific world, but eHealth dreams,
visions, policies, and programs have personal, social,
political, and ideological components, and therefore typically
prove fuzzy, slippery, and unstable when we seek to define
and control them".
Greenhalgh T, Stramer K, Bratan T, Byrne E, Russell J, et al. (2010) The devil's in the detail: final
report of the independent evaluation of the Summary Care Record and HealthSpace programmes.
London: University College London.

27. Challenges
• The scope of eHealth interventions requires diversity in
approaches as outcomes, impact, audience, cost and cost
structure, and technology are different. One model cannot
apply;
• Website for eLearning is different from a health information system
or an electronic medical record systems

28. Challenges
• Multiple players and stakeholders in eHealth interventions
require a multi-stakeholder approach to evaluation. These
stakeholders pose different and multiple challenges;
• Stakeholder representation includes legislative (legal) issues,
infrastructures issue, human resources issues, funding mechanisms,
technology adaptation and training needs.

29. Challenges
• eHealth interventions start usually as demonstration and pilot
projects. They are managed under a controlled environment
with dedicated staff and well defined budget and outcomes.
Scaling up becomes difficult (budget, legal, people, etc) and
when a scaled up project needs to be evaluated the whole
process changes and becomes less manageable.
• An mHealth pilot project is much easier to evaluate than a nation
wide implementation of mHealth or electronic health records.

30. Challenges
• Cost-benefit analysis of human resources vs. cost-benefit analysis of
technology. It is possible to calculate the cost of technology
infrastructure and applications including hardware and software. It
becomes more difficult to measure the cost of people's input to the
eHealth due to the diversity of roles physicians and nurses using the
system, ICT staff operating the system, patients benefiting from the
system.
• eHealth "time" challenges. Technology based systems develop much
faster than health systems. The pace of adoption of technology in
health makes it difficult to match.

31. Challenges
• Monitoring and evaluation cost money. They need to be planned form
the start and to be budgeted for and be performed by qualified
personnel using the right approach;

32. Do we have indicators?
• The eHealth Benchmarking report (2009) identified 79 sources of indicators
information, mainly surveys, which made an inventory of more than 4,400
eHealth-related indicators.
• The eHealth Benchmarking study developed a simple classification for data
sources in the course of its information gathering work that groups sources
according to four major purposes:
• Measuring of eHealth availability and use
• Evaluation of eHealth applications
• Measuring of attitudes towards eHealth
• Assessment of eHealth markets
http://www.ehealth-benchmarking.eu/results/documents/eHealthBenchmarking_Final-Report_2009.pdf

33. An indicator framework for eHealth
Benchmarking (EU Report, 2009)
1. eHealth Benchmarking indicators targeting citizens / patients;
2. eHealth Benchmarking indicators targeting General Practitioners;
3. eHealth Benchmarking indicators targeting specialists;
4. eHealth Benchmarking indicators hospital administrative / IT
staff;
5. eHealth Benchmarking indicators hospital medical staff;
6. eHealth Benchmarking indicators targeting therapists;
7. eHealth Benchmarking indicators targeting pharmacies;
8. eHealth Benchmarking indicators targeting care providers.
http://www.ehealth-benchmarking.eu/results/documents/eHealthBenchmarking_Final-Report_2009.pdf

34. Guiding principles for eHealth evaluation
• Greenhalgh and Russell provided an alternative set of guiding
principles for eHealth evaluation. These principles need to they will
need to be applied flexibly with attention to the particularities and
contingencies of different contexts and settings. Each principle will be
more or less relevant to a particular project, and their relative
importance will differ in different evaluations.
[*] Greenhalgh T, Russell J (2010) Why Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding
Principles. PLoS Med 7(11): e1000360. doi:10.1371/journal.pmed.1000360.
http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1000360

35. Guiding principles for eHealth evaluation
• First, think about your own role in the evaluation.
• Second, put in place a governance process (including a broad-based
advisory group with an independent chair) that formally recognizes that
there are multiple stakeholders and that power is unevenly distributed
between them.
• Third, provide the interpersonal and analytic space for effective
dialogue (e.g., by offering to feed back anonymized data from one group
of stakeholders to another).

36. Guiding principles for eHealth evaluation
• Fourth, take an emergent approach. An evaluation cannot be designed at the
outset and pursued relentlessly to its conclusions; it must grow and adapt in
response to findings and practical issues which arise in fieldwork
• Fifth, consider the dynamic macro-level context (economic, political,
demographic, technological) in which the eHealth innovation is being
introduced.
• Sixth, consider the different meso-level contexts (e.g., organisations,
professional groups, networks), how action plays out in these settings (e.g., in
terms of culture, strategic decisions, expectations of staff, incentives, rewards)
and how this changes over time. Include reflections on the research process
(e.g., gaining access) in this dataset .

37. Guiding principles for eHealth evaluation
• Seventh, consider the individuals (e.g., clinicians, managers, service users)
through whom the eHealth innovation(s) will be adopted, deployed, and used.
Explore their backgrounds, identities and capabilities; what the technology
means to them and what they think will happen if and when they use it.
• Eighth, consider the eHealth technologies, the expectations and constraints
inscribed in them (e.g., access controls, decision models) and how they “work”
or not in particular conditions of use.
• Ninth, use narrative as an analytic tool and to synthesize findings.

38. Evidence on eHealth InterventionsEvidence on eHealth Interventions
eHealth evaluation ''gap'‘:
Recap
• It is costly;
• It is not an integral part of eHealth project management;
• Fear of results (truth hurts);
• Lack of evaluation expertise;
• Lack of standard approaches or frameworks;
• Diversified technologies, environments and applications;
• Lack of standard indicators;
• Low priority for policy and decision makers;
• Conflict between internal and external evaluation.

39. Why do health ICT projects fail?
• Lack of proper needs assessment.
• Lack of vision, strategy, and national plans.
• Lack of information and awareness about ICT applications.
• Computer illiteracy.
• Insufficient resources to meet costs.
• Limited experience in medical informatics.
• Weak information and telecommunications infrastructures.
• Absence of legislative, ethical, and constitutional frameworks
WHO, 2004.

40. Evidence on eHealth InterventionsEvidence on eHealth Interventions
Comprehensive approach is needed
• …in light of the lack of evidence in relation to
improvements in patient outcomes, as well as the
lack of evidence on their cost-effectiveness, the
authors say that future eHealth technologies
should be evaluated against a comprehensive set
of measures, ideally throughout all stages of the
technology's life cycle, and include socio-
technical factors to maximize the likelihood of
successful implementation and adoption in a given
context".
D. Ashly, et al. The Impact of eHealth on the Quality and Safety of Health Care: A Systematic
Overview.
http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1000387

41. Thank you
Q & A