2. sician order entry systems have been shown to ibility of all studies identified in our search. A
reduce medical errors,15 but they can also in- second reviewer confirmed all relevant articles
crease error rates if not well designed and and retrieved full-text articles. Supplementary
implemented.16 methods of finding evaluations included a review
of article reference lists, informatics conference
proceedings, information provided by primary
Study Data And Methods study authors, requesting submissions from
STUDIES ELIGIBLE FOR REVIEW In our survey of stud- other researchers and implementers, and
ies for review, we included any qualitative or searching the RHINO Literature Database20 and
quantitative evaluation of information technol- other recent reviews.7,21–23
ogy affecting health care in developing coun- DATA ABSTRACTION AND SYNTHESIS We extracted
tries. We did not include telemedicine because data according to recurring themes, defined be-
other recent reviews exist.9,17 Developing countries low.We summarized these findings using tabular
were defined as those in the Emerging and De- techniques and descriptive statistics. Reported
veloping Economies List in the International analyses were too disparate to be pooled in a
Monetary Fund’s World Economic Outlook Report. meta-analysis.
Evaluations were excluded if (1) data complete- The systems described in the articles were
ness of the system was the only outcome, (2) the placed into one of eight categories correspond-
evaluation method was not described, (3) the ing to the typical applications used in developing
article only described the feasibility or technical countries. The order of these categories does not
evaluation of a system, (4) the evaluation was on infer any priority:
attitudes toward or knowledge of e-health (not (1) Electronic health record: an electronic rec-
an actual system), or (5) it was only an educa- ord of health-related information on an indivi-
tional tool.18,19 In the case of the Uganda Health dual that can be created, managed, or consulted
Information Network, we report on the e-health by clinicians or staff. In literature, the term elec-
component of the system. If an article did not tronic medical record is used interchangeably and
have an abstract, we attempted to find the article is used as a synonym in this paper.
through the Harvard or Massachusetts Institute (2) Laboratory information management sys-
of Technology (MIT) library systems. tem: a system for laboratory-specific activities or
FINDING RELEVANT STUDIES We conducted a for reporting results to administrators and
worldwide review of the literature and requested health care personnel.
submissions from researchers and those imple- (3) Pharmacy information system: any system
menting e-health in developing countries. Lit- used to order, dispense, or track medications or
erature searches were completed through Octo- medication orders including computerized or-
ber 2009 without language restrictions through der entry systems.
MEDLINE, EMBASE, Science Citation Index (4) Patient registration or scheduling system:
(Web of Science), Social Sciences Citation Index, any system used to monitor and manage the
the Cochrane Library, and the Latin American movement of patients through multistep proc-
and Caribbean Health Science Literature Data- esses or to maintain a census.24 An example is
base (LILACS). To find reports not in scientific admissions-discharge-transfer systems.
journals or conferences, we also used Google (5) Monitoring, evaluation, and patient track-
Scholar. For MEDLINE and EMBASE searches, ing system: any system used for aggregate report-
terms were derived from the MeSH database and ing of information, program monitoring, and
EMTREE tool, respectively. We searched for tracking of patients’ status. Examples include
more than forty commonly used terms to de- district health information systems or health
scribe e-health applications, found the broadest management information systems.
term within each tool that maintained its con- (6) Clinical decision support system: system
text, and then used that term for the search to designed to improve clinical decision making, in
ensure that we included all possible studies. which characteristics of individual patients are
Among the terms used in the final strategies were matched to a computerized knowledge base and
medical informatics applications, reminder system, software algorithms generate patient-specific
geographic information system, hospital informa- recommendations.25
tion systems, outcome and process assessment (7) Patient reminder system: a system used to
(Health Care), evaluation studies, attitude, costs prompt patients to perform a specific action—for
and cost analysis, developing countries, poverty, example, take medications or attend the clinic.
Africa, Latin America, eastern Europe, and central (8) Research/data collection system: any sys-
or southeastern Asia (complete strategies are tem used for collecting data from different loca-
available from the authors on request). An initial tions or for storing, managing, or reporting on
reviewer read the abstracts to evaluate the elig- data used for research purposes.
F E B R UA RY 2 0 1 0 29 : 2 HE A LT H A FFA IRS 245
3. POLICIES & POTENTIAL
Evaluations were classified into two major and abstracts, we found 126 articles that ap-
categories—qualitative and quantitative—as peared relevant. An additional five articles were
shown in Exhibit 1. Qualitative evaluations were identified by hand-searching bibliographies of
those where users gave opinions regarding a eligible articles and prior reviews. Of these,
system. These could be through questionnaires, forty-five fulfilled the inclusion criteria after full
focus groups, or interviews. (This definition is review of their abstracts. They are listed by type
different from the one proposed by Anselm of system and evaluation in Exhibit 1 and are
Strauss and Juliet Corbin of “any type of research categorized by systems in Appendix Exhi-
that produces findings not arrived at by statisti- bits 2a–5a.32 We included an evaluation from
cal procedures or other means of quantifica- the U.S. Indian Health Service, although it is
tion.”)26 Quantitative evaluations were those not in a developing country, because socioeco-
whose outcomes were data quality, administra- nomic and infrastructure conditions among the
tive changes, patient care, or economic assess- population treated are similar to those in devel-
ment. Evaluation designs were grouped accord- oping countries. If a system had multiple evalua-
ing to the definition by Charles Friedman and tions, only those with different outcomes are
Jeremy Wyatt:27(1) descriptive (uncontrolled) listed. If they had the same outcome, we took
study; (2) historically controlled (before-after) the one with the largest sample size. There were
study; (3) case-control (retrospective) study; two articles reporting an evaluation that did not
(4) prospective self-controls (subjects perform- occur because of a failed system implementa-
ing the same action in both systems; this cate- tion.33,34 These are not part of the results, but
gory was added by the authors); (5) simultaneous we considered them relevant to list because ar-
nonrandomized controls; (6) simultaneous ran- ticles on unsuccessful systems are not commonly
domized controls; and (7) externally and intern- published.
ally controlled before-after study. Two cost stud- Fifteen articles performed qualitative evalua-
ies and two studies modeling future medication tions, and forty performed quantitative evalua-
requirements were categorized as self-controls tions. If an evaluation performed both types, it
because they compared the impact of the system was counted in both categories. Two qualitative
against the same situation without the system. evaluations and sixteen quantitative performed
As a result of the inherent limitation of perform- statistical analysis. Of all evaluations, two
ing a case-control, descriptive, or qualitative (13 percent) of the qualitative and seven (18 per-
study without statistics, we do not comment cent) of the quantitative were performed by an
on the limitations of these studies. outside evaluator. The number of evaluations
has more than tripled comparing periods before
and after 2002.
Study Results ELECTRONIC HEALTH RECORDS Because EHRs are
Searches retrieved 2,043 citations. Five articles the core clinical application, they usually encom-
were excluded because they did not have ab- pass a variety of functionalities, which makes
stracts and full-text versions were not avail- their implementations complex35 and prone to
able.28–31 After the initial screening of article titles failure.36 This complexity provides an additional
EXHIBIT 1
Number Of Articles Included In Analysis, By E-Health Category And Evaluation Type
Quantitative
E-health category Qualitative Descriptive studies Controlled studies
Electronic health record 5 1 5
Laboratory information management systems 0 1 2
Pharmacy information systems 4 2 3
Patient registration or scheduling systems 1 0 2
Monitoring, evaluation, and patient tracking systems 0 2 4
Clinical decision support systems 1 0 3
Patient reminder systems 0 1 3
Research/data collection systems 5 1 11
Total 15 8 32
SOURCE Authors’ analysis. NOTES The articles (n ¼ 45) are classified by e-health category and by type of evaluation. If an article had both
qualitative and quantitative studies or multiple types of systems, it was counted in both categories. Details about the evaluated
projects are in Appendix Exhibits 2a–5a, available online as in Note 32.
246 HEA LT H AF FA IR S F E B R UA RY 2 0 1 0 29:2
4. challenge in their evaluation. Most evaluations in training and technical support and the need to
found provided insight into possible impacts of maintain a parallel paper system.
these systems, but had limited scientific rigor, as MONITORING , EVALUATION , AND PATIENT TRACKING
seen in Appendix Exhibit 2a.32,27 SYSTEMS Evaluations of systems to track and
The Indian Health Service’s Vista system was monitor patients’ status are limited to two
the most complete system we reviewed, and its case-control studies performed by the same or-
rigorous qualitative evaluation showed that a ganization in Haiti (Appendix Exhibit 4a).32
majority of clinicians viewed its implementation Both of these studies suggest that an electronic
positively and hence used it more. The Mosoriot system can effectively alert staff of patients who
Medical Record System evaluation in Kenya pro- have “fallen through the cracks” and prevent
vides data on the impact that an EHR can have on rates of patients lost to follow-up, which were
improving staff productivity and reducing pa- found to be as high as 76 percent (after two
tient wait times. All other evaluations were qual- years) as reported in some HIV programs.3
itative and provided insights into EHRs’ ability Two randomized controlled trials looked at
to improve staff satisfaction, providing higher- the effect of Global Positioning Systems (GPS)
quality data to relevant personnel and ultimately in finding households once a patient has been
improving patient care. identified. An evaluation from South Africa
LABORATORY INFORMATION MANAGEMENT SYSTEMS showed that GPS reduced the time to find a
There were only three evaluations of laboratory household by 20–50 percent, whereas one from
information management systems, all quantita- Nicaragua showed no difference between the pa-
tive, with only one having a control group (Ap- per and GPS systems. Both the South African and
pendix Exhibit 3a).32 However, they suggest two Nicaraguan systems were tested in similar urban
major benefits that such systems can provide: settings with novice users, so no immediate
(1) decreasing times for communication of re- reason for the difference can be found. Both
sults, and (2) improving the productivity of the studies had small sample sizes (identifying
laboratory. An additional impact, reduction in ten to fifty households) and lacked statistical
errors, has not yet been studied, although there analysis.
are groups currently performing such trials.37 Two evaluations, one descriptive and one cost
PHARMACY INFORMATION SYSTEMS Computerized analysis, looked at monitoring departments
order entry can provide a key incentive for clin- within a hospital in Cambodia and health estab-
ical staff, especially clinicians, to use an informa- lishments nationwide in Tanzania. They suggest
tion system, because such systems can reduce the that electronic systems can help allocate re-
time to order medications (especially repeat or- sources efficiently and improve infection control
ders) and provide easy access to past informa- and can be relatively low cost, respectively. How-
tion. The four qualitative evaluations shown in ever, both evaluations lacked detail on the tasks
Appendix Exhibit 3a32 cite these as their system’s affected, as well as control groups.
main advantages. The two quantitative evalua- CLINICAL DECISION SUPPORT SYSTEM Decision
tions with a control group (Socios en Salud in support systems have received attention for de-
Peru and Hamadan University of Medical veloping countries as a possible solution to the
Sciences in Iran) showed a reduction in errors, lack of trained clinical personnel, especially in
which is a main outcome cited in developed rural areas. The three quantitative evaluations
country studies. An additional benefit from some seen in Appendix Exhibit 4a32 were of high rigor.
pharmacy systems in developing countries is The expert system for mechanically ventilated
their ability to forecast medication requirements newborns showed that nurses performed better
(Socios en Salud in Peru). This is useful if a on a standardized test and felt that they had
country or organization needs to order medica- better judgment after receiving training on the
tions months in advance to get lower prices, system. The evaluation of the personal digital
which is currently the case for drug-resistant assistant (PDA) device to perform the Electronic
TB medications. Integrated Management of Childhood Illness ap-
PATIENT REGISTRATION AND SCHEDULING The two proach in Tanzania showed that more clinical
quantitative evaluations of registration systems, staff completed the electronic questionnaire
seen in Appendix Exhibit 4a,32 showed that fin- compared to the paper booklet. It also showed
gerprint scanners and barcode readers de- that it took the same amount of time (12.5 min-
creased the time to locate records by 74 percent utes) to fill out the questionnaire by either meth-
and 97 percent, respectively. The small sample od. The evaluation of the Early Diagnosis and
size of thirty in these randomized controlled Prevention System in India showed higher satis-
trials was their biggest limitation. In the quali- faction among patients if they were seen by a
tative evaluation of the Baobab system in Mala- computer operator before their clinical visit
wi, users preferred it to paper despite limitations and that there was a large increase in new pa-
F E B R UA RY 2 0 1 0 2 9 :2 HE A LT H A FFA IR S 247
5. POLICIES & POTENTIAL
tients at health centers with the system. compared the PDA system to paper and not to
However, the two studies with simultaneous a gold standard. The study performed by Socios
controls had major limitations. The evaluation of en Salud had a small number of users (n ¼ 4),
the Electronic Integrated Management of Child- and the study performed by the London School of
hood Illness was performed by the developers of Economics was performed seventeen years ago.
the systems, and because the technology was The organizations that implemented the PDA-
new to the users, the novelty rather than its use- based systems in Uganda and South Africa have
fulness could account for the additional comple- experience with hundreds of users and more
teness. In the case of the Early Diagnosis and than a dozen implementations combined, which
Prevention Systems, the increased attendance empirically shows the feasibility of such systems.
and patients’ opinions could have been easily The cost analyses show that these systems are
biased by the presence of the computers, the able to recoup the high initial costs by providing
motivation of computer operators, and the increased efficiency and continuous material
length of time spent with operator, none of costs. The Uganda system showed a cost savings
which were present at control sites. of 91 percent over the paper system. The South
PATIENT REMINDER SYSTEMS The quantitative African analysis calculated that after using the
evaluation of the South African text messaging PDA system for data collection in eight studies of
system (Appendix Exhibit 5a)32 found that after medium scale, it would equal the costs of paper.
the system was implemented, there were higher The PDA system in Peru would pay for expansion
completion rates of TB treatment. However, the to other health districts in three months as a
comparison was made to the city’s TB program result of increased efficiency.
register, for which the data quality was not ver-
ified and the data were different from the source
of the prospective data. A randomized trial in Discussion
Malaysia found that both text messaging and This review shows that with the exception of
mobile phone reminders significantly increased PDA-based data collection, there are still few
attendance (by 21 percent) over the control scientifically rigorous data on the effectiveness
group. Although they both had similar effective- and cost-effectiveness of e-health systems in de-
ness, the text messaging system was half the cost veloping countries. Further, the evaluations
of the mobile phone reminders. This evaluation have mostly been performed by organizations
had no major limitations. connected to academic settings and not by other,
The Malaysian study performed a well- larger recipients of donor funding.When looking
designed cost-effectiveness study showing that at the software systems included in the U.S. Pres-
text messaging, implemented correctly, can be a ident’s Emergency Plan for AIDS Relief (PEP-
cost-effective method to increase clinic atten- FAR) Anti-Retroviral Therapy (ART) Software
dance. This is especially important since both Inventory Report5 and EngenderHealth–Open-
TB and HIV treatments require constant super- Society software tools38 comparison, only three
vision of patients and strict adherence to a daily systems, the Partners in Health—Electronic
regimen of medications. Such systems can help Medical Record/HIV—Electronic Medical Rec-
patients in resource-poor settings who encoun- ord in Kenya, Mosoriot Medical Record System
ter many obstacles that can prevent them from in Kenya, and Vista in the U.S. Indian Health
getting their medications. Service, have had any evaluations performed.
RESEARCH / DATA COLLECTION SYSTEMS Research/ Although a few studies have been commissioned
data collection systems was the group with the by the U.S. Centers for Disease Control and Pre-
largest number and most rigorous evalua- vention (CDC), it is particularly important that
tions (Appendix Exhibit 5a).32 All systems, ex- large funders such as the U.S. Agency for Inter-
cept the Mexican National Institute of Public national Development or PEPFAR include re-
Health’s Audio Computer-Assisted Self-Inter- sources for the evaluation of e-health systems
view (ACASI) system, were on PDAs. Four ran- developed and deployed in developing countries
domized trials showed that the main benefits of and perhaps make them a requirement for con-
PDA-based systems were data qual- tinued funding. This could include
ity similar to paper systems or high- standard designs for studies that
er, less time taken to perform inter- are suitable for resource-poor en-
views, and decreased time to collect vironments, that minimize biases,
data. However, many of the studies and that are easily comparable to
had major limitations. The systems the results from other projects.
from the Universidad Peruana The overall pattern of e-health
Cayetano Heredia and the South evaluations in developed countries
African Medical Research Council reflects an initial focus on qualita-
248 HE A LT H A FFA IRS F E B R UARY 2 0 10 2 9 :2
6. tive and descriptive evaluations, with an increase health and cell phone–based tools, because these
in the number of quantitative and larger evalua- devices are also playing an increasing role in
tions published in the past decade. Developing communication directly with patients.
countries seem to be following this pattern as Evaluations of e-health systems are chal-
well, so in this study we found mostly qualitative lenging and require significant resources in ad-
and descriptive studies but saw an increase in the dition to funds creating and implementing sys-
number of randomized trials performed in the tems. Implementations should have evaluations
past few years. This suggests that as e-health built into the process. This will provide useful
implementations become more robust in devel- feedback to improve the project (formative eval-
oping countries, we can expect more rigorous uations) and will also demonstrate the impact of
studies, such as randomized trials or cost-effec- the system in the long term (summative evalua-
tiveness studies. tions). Evaluations in resource-poor environ-
Initial evaluations suggest that the following ments face many challenges when compared to
functions are of positive impact in developing those in developed countries, such as the physi-
countries: cal environment, power, networking, and avail-
(1) Ability to track patients through the treat- ability of technical staff. Measures of short- and
ment initiation process, monitor adherence, and long-term system usage and data completeness
detect those at risk for loss to follow-up. (2) Tools are important and a necessary prerequisite to a
to decrease communication times of information full evaluation study. Poor data quality is a con-
within and between institutions. (3) Tools to stant problem in health projects, whether they
label or register samples and patients. (4) Ability use paper or electronic systems, so tools that can
to electronically monitor and remind patients of reduce errors as well as benefiting other aspects
health care needs or treatment. (5) Collection of of care are likely to be well received.
clinical or research data using PDA applications. Some benefits of electronic systems are diffi-
(6) Reductions in errors in laboratory and med- cult to quantify. One is the ability to perform
ication data. operational research with greatly reduced costs.
Important findings include the user prefer- During our search we found eight studies that
ence for the Baobab touch-screen system in used electronic databases and probably could
Malawi, one of the only fully electronic point- not have been performed if manual data collec-
of-care systems in use in Africa, which is now tion was required. Another is the impact of emer-
in daily use for more than 35,000 HIV patients. gency communication across large distances,
The benefit shown for patient tracking and such as in the cholera outbreak in India or refu-
reminders is also important, given the loss to gee situations.39 The strongest evidence for ben-
follow-up rate of up to 76 percent for HIV pa- eficial impact of e-health on health care will come
tients in Africa.3 The Malaysian systems that from long-term follow-up of this sort carried out
texted patient reminders showed a significant by independent evaluators.
decrease in missed visits, at a reasonably low
cost, and the On Cue Compliance Service in
South Africa was well liked by users several years Conclusions
after implementation and, perhaps more impor- With the rapid growth of e-health in developing
tant, by an independent evaluation team. These countries, there is clearly an urgent need for
systems can be of high value because intermit- solid evidence of its impact to justify and guide
tent treatment puts patients at grave risk of the investment of resources in such systems.
deterioration and death, as well as causing in- Despite major increases in evaluations in recent
creased drug resistance and further transmis- years, most large e-health implementations have
sion of disease to the wider community. little or no evaluation data. To date, most studies
Tools to store and communicate such data with have been small; focused on process indicators
low error rates have been early successes in de- rather than patient outcomes, or on the attitudes
veloped countries, and the positive evaluations of users and patients; and performed mostly by
described here should drive their use in the de- academic groups. An increased focus on includ-
veloping world. Evaluations of PDAs and mobile ing evaluations as part of e-health implementa-
devices were particularly rigorous, and they con- tions is necessary and should be adopted by or-
vincingly demonstrate that such devices can be ganizations implementing or funding such
very effective in improving data collection time systems. One method is for large funders to in-
and quality. An additional benefit is their light clude resources for evaluations or make them a
weight and lack of printing costs compared to requirement for implementation.
large paper forms, which is crucial in remote Although evaluations of important indicators
areas with poor infrastructure. These results of care are difficult to do well, this review has
are important for the growing field of mobile confirmed that they are feasible even in very
F EB R UARY 2 0 1 0 29:2 H E ALT H AF FAI RS 249
7. POLICIES & POTENTIAL
challenging environments. Initial benefits were medications. Because of the lack of infrastruc-
shown in systems that track patients through ture and backup systems in resource-poor envir-
treatment initiation, monitor adherence, and de- onments, well-designed e-health solutions may
tect those at risk for loss to follow-up; tools to have a much larger impact on quality of care than
decrease information communication times in more developed areas. As e-health becomes
within and between institutions, as well as errors widespread in developing countries, these and
in reporting laboratory data; barcoding for pa- other benefits will need to be identified by more
tient identification cards and laboratory sam- rigorous evaluations that include long-term
ples; handheld devices for collecting and acces- follow-up and are carried out by independent
sing data; and the ordering and management of evaluators. ▪
An initial version of this paper was Chilean company that provides health Veronica Rojas, Adesina Iluyemi,
requested by the Rockefeller Foundation informatics consulting and technology in Mauricio Soto, Waldo Ortega, Chris
for the Making the eHealth Connection Latin America. The authors acknowledge Bailey, Patrick Whitaker, Gerry Douglas,
conference held in Bellagio, Italy, in July those who took the time to provide Natasha Kanagat, Steve Yoon, Zach
2008. This paper was funded by the additional information: Holly Ladd and Landis Lewis, Joel Selanikio, and Neal
Rockefeller Foundation. Joaquin A. Blaya Berhane Gebru from AED-Satellife, Lesh. Finally, the authors thank Claire
is cofounder of eHealth Systems, a Libby Levison, Heather Zornetzer, Mack for her invaluable editing.
NOTES
1 World Health Organization. 58th tively collected data. BMC Med In- lingual online physician education
World Health Assembly Report; 16– form Decis Mak. 2007;7(1):38. about electronic medical records.
25 May 2005. Geneva: WHO; 2005. 12 Garrido T, Jamieson L, Zhou Y, AMIA Annu Symp Proc. 2005:946.
2 Edworthy SM. Telemedicine in de- Wiesenthal A, Liang L. Effect of 20 Routine Health Information Net-
veloping countries. BMJ. 2001;323 electronic health records in ambu- work. RHINO Literature Database
(7312):524–5. latory care: retrospective, serial, [Internet]. Boston (MA): Routine
3 Rosen S, Fox MP, Gill CJ. Patient cross sectional study. BMJ. Health Information Network
retention in antiretroviral therapy 2005;330(7491):581. (RHINO); 2008 [cited 2010 Jan 4].
programs in sub-Saharan Africa: a 13 Wang SJ, Middleton B, Prosser LA, Available from: http://www
systematic review. PLoS Med. 2007 Bardon CG, Spurr CD, Carchidi PJ, .iphealth.info/refbase/index.php
Oct 16;4(10):e298. et al. A cost-benefit analysis of elec- 21 Fraser HS, Biondich P, Moodley D,
4 World Health Organization. Elec- tronic medical records in primary Choi S, Mamlin BW, Szolovits P.
tronic health records: a manual for care. Am J Med. 2003;114 Implementing electronic medical
developing countries. Geneva: (5):397–403. record systems in developing coun-
WHO; 2007. 14 Evans DC, Nichol WP, Perlin JB. tries. Inform Prim Care. 2005;13
5 U.S. President’s Emergency Plan for Effect of the implementation of an (2):83–95.
AIDS Relief. PEPFAR Software In- enterprise-wide electronic health 22 Forster M, Bailey C, Brinkhof MW,
ventory Report. Washington (DC): record on productivity in the Veter- Graber C, Boulle A, Spohr M, et al.
PEPFAR; 2004. ans Health Administration. Health Electronic medical record systems,
6 Rigby M. Impact of telemedicine Econ Policy Law. 2006;1(Pt data quality and loss to follow-up:
must be defined in developing 2):163–9. survey of antiretroviral therapy pro-
countries. BMJ. 2002;324 15 Bates DW, Teich JM, Lee J, Seger D, grammes in resource limited set-
(7328):47–8. Kuperman GJ, Ma’Luf N, et al. The tings. Bull World Health Organ.
7 Tomasi E, Facchini LA, Maia MF. impact of computerized physician 2008;86(12):939–47.
Health information technology in order entry on medication error 23 Fraser HS, Allen C, Bailey C, Douglas
primary health care in developing prevention. J Am Med Inform Assoc. G, Shin S, Blaya J. Information sys-
countries: a literature review. Bull 1999;6(4):313–21. tems for patient follow-up and
World Health Organ. 2004;82 16 Koppel R, Metlay JP, Cohen A, chronic management of HIV and
(11):867–74. Abaluck B, Localio AR, Kimmel SE, tuberculosis: a life-saving technology
8 Mitchell E, Sullivan F. A descriptive et al. Role of computerized physician in resource-poor areas. J Med In-
feast but an evaluative famine: sys- order entry systems in facilitating ternet Res. 2007;9(4):e29.
tematic review of published articles medication errors. JAMA. 2005;293 24 Shortliffe EH, Perreault LE, editors.
on primary care computing during (10):1197–203. Medical informatics: computer ap-
1980–97. BMJ. 2001;322 17 Hersh WR, Hickam DH, Severance plications in health care and bio-
(7281):279–82. SM, Dana TL, Pyle Krages K, Helfand medicine. 2nd edition. New York
9 Roine R, Ohinmaa A, Hailey D. As- M. Diagnosis, access, and outcomes: (NY): Springer; 2001.
sessing telemedicine: a systematic update of a systematic review of 25 Garg AX, Adhikari NK, McDonald H,
review of the literature. CMAJ. telemedicine services. J Telemed Rosas-Arellano MP, Devereaux PJ,
2001;165(6):765–71. Telecare. 2006;12(Suppl 2):S3–31. Beyene J, et al. Effects of compu-
10 Whitten PS, Mair FS, Haycox A, May 18 Mallapaty G, Kim S, Astion ML. terized clinical decision support
CR, Williams TL, Hellmich S. Sys- Using interactive software to teach systems on practitioner performance
tematic review of cost effectiveness image-based clinical laboratory tests and patient outcomes: a systematic
studies of telemedicine interven- in developing countries: a pilot trial review. JAMA. 2005;293
tions. BMJ. 2002;324(7351):1434–7. in Nepal. Clin Chem Lab Med. (10):1223–38.
11 Pollak VE, Lorch JA. Effect of elec- 2003;41(5):711–3. 26 Strauss A, Corbin J. Basics of quali-
tronic patient record use on mor- 19 Edmonson SR, Esquivel A, tative research: grounded theory
tality in end stage renal disease, a Mokkarala P, Johnson CW, Phelps procedures and techniques. New-
model chronic disease: retrospective CL. Using technology to teach tech- bury Park (CA): Sage; 1990.
analysis of nine years of prospec- nology: design and evaluation of bi- 27 Friedman CP, Wyatt JC. Evaluation
250 H E A LT H A FFA IRS F E B R UA RY 2 01 0 2 9 :2
8. methods in medical informatics. 2nd online; see the Appendix Exhibits cess, and local improvisations. In-
edition. New York: Springer- link in the box to the right of the formation Society. 2002;18
Verlag; 2005. article online. (2):101–12.
28 Halbwachs H. The technical and fi- 33 Littlejohns P, Wyatt JC, Garvican L. 37 Blaya JA, Shin SS, Yagui MJ, Yale G,
nancial impact of systematic main- Evaluating computerised health in- Suarez CZ, Asencios LL, et al. A Web-
tenance and repair services within formation systems: hard lessons still based laboratory information system
health systems of developing coun- to be learnt. BMJ. 2003;326 to improve quality of care of tuber-
tries. Health Estate. 1999;53(4):6– (7394):860–3. culosis patients in Peru: functional
8,10–1. 34 Iluyemi A, Briggs J, Fitch T. Elec- requirements, implementation, and
29 Desikan P, Koram MR, Trivedi SK, tronic health records in developing usage statistics. BMC Med Inform
Jain A. An evaluation of the effec- countries, integrating with mobile Decis Mak. 2007;7:33.
tiveness of the laboratory informa- technology and legacy systems for 38 EngenderHealth–Open Society In-
tion system (LIS) with special refer- community based health workers: stitute. Health toolkit: information
ence to the microbiology laboratory. organisational and end-users’ issues. management challenges and oppor-
Indian J Path Microbiol. Proceedings of the European Con- tunities for community-based orga-
2005;48(3):418. ference on Information Manage- nizations serving people living with
30 Janecki J, Podsiadly T. Computer- ment and Evaluation. Montpellier, HIV/AIDS. New York: Engender-
assisted analysis of patients’ medical France; 2007 Sep 20–21. Health–Open Society Insti-
records. Pol Tyg Lek. 1992;47(20– 35 Brender J, Ammenwerth E, Nykanen tute; 2004.
21):470–2. P, Talmon J. Factors influencing 39 Babille M, Decolombani P, Guerra R,
31 Swaminathan R, Black RJ, success and failure of health infor- Zagaria N, Zanetti C. Post-
Sankaranarayanan R. Database on matics systems—a pilot Delphi emergency epidemiological surveil-
cancer survival from developing study. Methods Inf Med. 2006;45 lance in Iraqi-Kurdish refugee camps
countries. IARC Sci Publ. 1998; (1):125–36. in Iran. Disasters. 1994;18(1):58–75.
(145):19–25. 36 Heeks R. Information systems and
32 The Appendix Exhibits are available developing countries: failure, suc-
ABOUT THE AUTHORS
Africa, and Asia. Blaya, 31, implementing an electronic Institute in the United United Kingdom. He also
who was born in Chile, is a health record for use in States, the Medical completed a fellowship in
Harvard and Massachusetts managing multidrug-resistant Research Council in South clinical decision making and
Institute of Technology TB patients in Peru. He and Africa, and others, have cardiology at MIT and the
(MIT)–trained Ph.D. in health Blaya teamed up to produce developed an “open source,” New England Medical Center.
sciences and technology. a Palm Pilot–based system or nonproprietary, electronic Blaya, who today is a
Fraser, age 47, was born in to collect laboratory results health record system for research fellow at Partners
Scotland and was educated on behalf of these patients. developing countries, called in Health, is also a National
and trained in medicine and In a study published in 2009 OpenMRS. The system is Library of Medicine Fellow
Joaquin A. Blaya cardiology in the United in the International Journal used by more than forty-five at Harvard Medical School.
Kingdom. They met in 2004 of Infectious Diseases, the organizations in twenty- In addition, he recently
when Blaya was at a joint system was shown to three countries and is cofounded a company,
Harvard-MIT program decrease delays in getting available for download at eHealth Systems, which aims
working on his Ph.D. and those results from thirty http://www.openmrs.org. to implement open-source
Fraser became his days to eight days, and to “My focus has been on technologies, including
supervisor. Then, as now, reduce errors in the practical systems that are OpenMRS, in health systems
Fraser was an assistant communication of these useful for doctors and other in Latin America. Having
professor of medicine at tests to clinicians by 59 health care staff,” says emigrated from Chile to
Harvard Medical School and percent. Fraser, who is also an Miami, Florida, twenty-two
director of informatics and Since then, the two have associate physician at the years ago, he plans to move
telemedicine at the worked on implementing a Brigham and Women’s back to Chile in 2010. His
nonprofit organization Web-based system to Hospital in Boston. In five-year goal is for a
Partners in Health, which communicate laboratory addition to his medical majority of public health
Hamish Fraser focuses on providing health results to TB clinicians in degree, he trained in the centers in Chile to use
Coauthors and frequent care for the poor in a more than 220 health development and use of so- OpenMRS and to expand
collaborators Joaquin Blaya number of developing centers throughout Peru. called knowledge-based their use in Nicaragua,
and Hamish Fraser share a countries, including Haiti, Fraser’s group (the systems—computer systems Argentina, Brazil, and other
passion for using e-health Rwanda, and Peru. Electronic Medical Records to diagnose and analyze countries.
technologies to improve Back then, Fraser was Team at Partners in Health), real-world data—at
health care in Latin America, working on developing and with the Regenstrief Edinburgh University in the
F E B R UA RY 2 0 1 0 2 9 :2 HE A LT H A FFA IRS 251
9. Blaya JA, Fraser HSF, Holt B. E-health technologies show promise in developing countries. Health Aff (Millwood). 2010;29(2):243-
50.
APPENDIX Exhibit 1a: Additional References
[1a] Merrell RC, Merriam N, Doarn C. Information support for the ambulant health worker.
Telemed J E Health. 2004 Winter;10(4):432-6.
[2a] Singh AK, Kohli M, Trell E, Wigertz O, Kohli S. Bhorugram (India): revisited. A 4 year
follow-up of a computer-based information system for distributed MCH services. International
journal of medical informatics. 1997 Apr;44(2):117-25.
[3a] Llido LO. The impact of computerization of the nutrition support process on the nutrition
support program in a tertiary care hospital in the Philippines: report for the years 2000-2003.
Clin Nutr. 2006 Feb;25(1):91-101.
[4a] Chae YM, Kim SI, Lee BH, Choi SH, Kim IS. Implementing health management
information systems: measuring success in Korea's health centers. Int J Health Plann Manage.
1994 Oct-Dec;9(4):341-8.
[5a] Al Farsi M, West DJ, Jr. Use of electronic medical records in Oman and physician
satisfaction. J Med Syst. 2006 Feb;30(1):17-22.
[6a] Weinhara M, Stoicu-Tivadar L, Dagres C. Early stage testing of user's satisfaction after
implementation of a central electronic health record (EHR) system in Serbia. Journal on
Information Technology in Healthcare. 2009;7(2):127-33.
[7a] Sequist TD, Cullen T, Hays H, Taualii MM, Simon SR, Bates DW. Implementation and
use of an electronic health record within the Indian Health Service. J Am Med Inform Assoc.
2007 Mar-Apr;14(2):191-7.
[8a] Ndira SR, Rosenberger KD, Wetter T. Assessment of data quality of and staff satisfaction
with an electronic health record system in a developing country (Uganda): A qualitative and
quantitative comparative study. Methods of Information in Medicine. 2008 2008;47(6):489-98.
[9a] Rotich JK, Hannan TJ, Smith FE, Bii J, Odero WW, Vu N, et al. Installing and
implementing a computer-based patient record system in sub-Saharan Africa: the Mosoriot
Medical Record System. J Am Med Inform Assoc. 2003 Jul-Aug;10(4):295-303.
[10a] Pourasghar F, Malekafzali H, Koch S, Fors U. May not fit Factors influencing the quality
of medical documentation when a paper-based medical records system is replaced with an
electronic medical records system: an Iranian case study. Int J Technol Assess Health Care. 2008
Fall;24(4):445-51.
[11a] Ayyagari A, Bhargava A, Agarwal R, Mishra SK, Mishra AK, Das SR, et al. Use of
telemedicine in evading cholera outbreak in Mahakumbh mela, Prayag, UP, India: An
encouraging experience. Telemedicine Journal and E-Health. 2003;9(1):89-94.
[12a] Alvarez Flores MG, Guarner J, Terres Speziale AM. [Productivity before and after
installing a computerized system in a clinical laboratorya]. Rev Invest Clin. 1995 Jan-
Feb;47(1):29-34.
[13a] Turhan K, Kayikcioglu T. Implementation of a virtual private network-based laboratory
information system serving a rural area in Turkey. Laboratory Medicine. 2006;37(9):527-31.
[14a] Cassiani SH, Freire CC, Gimenes FR. [Electronic medical prescription at a university
hospital: writing failures and users' opinions]. Rev Esc Enferm USP. 2003 Dec;37(4):51-60.
[15a] Costa AL, de Oliveira MM, Machado Rde O. An information system for drug
prescription and distribution in a public hospital. International journal of medical informatics.
2004 May;73(4):371-81.
[16a] Gimenes FRE, Miasso AI, De Lyra Jr DP, Grou CR. Electronic prescription as
contributing factor for hospitalized patients' safety. Pharmacy Practice. 2006;4(1):13-7.
1
10. Blaya JA, Fraser HSF, Holt B. E-health technologies show promise in developing countries. Health Aff (Millwood). 2010;29(2):243-
50.
[17a] Tan WS, Phang JS, Tan LK. Evaluating user satisfaction with an electronic prescription
system in a primary care group. Ann Acad Med Singapore. 2009 Jun;38(6):494-7.
[18a] Kazemi A, Ellenius J, Tofighi S, Salehi A, Eghbalian F, Fors UG. CPOE in Iran--a viable
prospect? Physicians' opinions on using CPOE in an Iranian teaching hospital. International
journal of medical informatics. 2009 Mar;78(3):199-207.
[19a] Kazemi A, Ellenius J, Pourasghar F, Tofighi S, Salehi A, Amanati A, et al. The Effect of
Computerized Physician Order Entry and Decision Support System on Medication Errors in the
Neonatal Ward: Experiences from an Iranian Teaching Hospital. Journal of Medical Systems.
2009.
[20a] Fraser H, Jazayeri D, Choi S, Blaya J, Bayona J, Levison L, et al. Forecasting three years
drug supply for a large MDR-TB treatment program in Peru. Int J Tuber Lung Dis. 2006;10(11
Suppl. 1):S245.
[21a] Yamanija J, Durand R, Bayona J, Blaya J, Jazayeri D, Fraser H. Comparing actual
medication consumption against the quantities ordered and a prediction using an information
system. Int J Tuber Lung Dis. 2006;10(11 Suppl. 1):S69-S70.
[22a] Choi SS, Jazayeri DG, Mitnick CD, Chalco K, Bayona J, Fraser HS. Implementation and
initial evaluation of a Web-based nurse order entry system for multidrug-resistant tuberculosis
patients in Peru. Medinfo. 2004;11(Pt 1):202-6.
[23a] CDC Global AIDS Program. Responses to the Touchscreen System User Survey: Queen
Elizabeth Central Hospital. Malawi: CDC Global AIDS Program; 2007.
[24a] Aviles W, Ortega O, Kuan G, Coloma J, Harris E. Quantitative assessment of the benefits
of specific information technologies applied to clinical studies in developing countries. Am J
Trop Med Hyg. 2008 Feb;78(2):311-5.
[25a] Fabre-Teste B, Sokha O. [Calmette Hospital, Phnom Penh, Cambodia. Assessment of the
implementation of the Medical Information System (SIM). Global analysis of the 1998 results].
Sante. 1999 Nov-Dec;9(6):367-75.
[26a] Rommelmann V, Setel PW, Hemed Y, Angeles G, Mponezya H, Whiting D, et al. Cost
and results of information systems for health and poverty indicators in the United Republic of
Tanzania. Bull World Health Organ. 2005 Aug;83(8):569-77.
[27a] Fraser HSF, Allen C, Bailey C, Douglas G, Shin S, Blaya J. Information systems for
patient follow-up and chronic management of HIV and tuberculosis: A life-saving technology in
resource-poor areas. Journal of Medical Internet Research. 2007;9(4):38.
[28a] Dwolatzky B, Trengove E, Struthers H, McIntyre JA, Martinson NA. Linking the global
positioning system (GPS) to a personal digital assistant (PDA) to support tuberculosis control in
South Africa: a pilot study. International journal of health geographics. 2006;5:34.
[29a] Jirapaet V. A computer expert system prototype for mechanically ventilated neonates
development and impact on clinical judgment and information access capability of nurses.
Comput Nurs. 2001 Sep-Oct;19(5):194-203.
[30a] DeRenzi B, Lesh N, Parickh T, Sims C, Mitchell M, Maokola W, et al. e-IMCI:
Improving Pediatric Health Care in Low-Income Countries. CHI. Florence, Italy 2008.
[31a] Peters DH, Kohli M, Mascarenhas M, Rao K. Can computers improve patient care by
primary health care workers in India? International Journal for Quality in Health Care.
2006;18(6):437-45.
[32a] Bridges.org. Evaluation of the On Cue Compliance Service Pilot: Testing the use of SMS
reminders in the treatment of Tuberculosis in Cape Town, South Africa. Cape Town: City of
2
11. Blaya JA, Fraser HSF, Holt B. E-health technologies show promise in developing countries. Health Aff (Millwood). 2010;29(2):243-
50.
Cape Town Health Directorate and the International Development Research Council (IDRC);
2005.
[33a] Leong KC, Chen WS, Leong KW, Mastura I, Mimi O, Sheikh MA, et al. The use of text
messaging to improve attendance in primary care: a randomized controlled trial. Fam Pract. 2006
Dec;23(6):699-705.
[34a] Shirima K, Mukasa O, Schellenberg JA, Manzi F, John D, Mushi A, et al. The use of
personal digital assistants for data entry at the point of collection in a large household survey in
southern Tanzania. Emerg Themes Epidemiol. 2007;4:5.
[35a] Bridges.org. Evaluation of the SATELLIFE PDA Project, 2002: Testing the use of
handheld computers for heathcare in Ghana, Uganda, and Kenya. Boston, MA: Satellife; 2003.
[36a] Satellife and Uganda Chartered HealthNet. Uganda Health Information Network, Phase-
III: June 9, 2006 – June 8, 2007. Boston: Satellife and Uganda Chartered HealthNet; 2007.
[37a] Kinkade S, Verclas K. Wireless Technology for Social Change. Washington, DC: UN
Foundation-Vodafone Group Foundation Partnership; 2008.
[38a] Missinou MA, Olola CH, Issifou S, Matsiegui PB, Adegnika AA, Borrmann S, et al.
Short report: Piloting paperless data entry for clinical research in Africa. Am J Trop Med Hyg.
2005 Mar;72(3):301-3.
[39a] Gutierrez JP, Torres-Pereda P. Acceptability and reliability of an adolescent risk behavior
questionnaire administered with audio and computer support. Revista Panamericana De Salud
Publica-Pan American Journal of Public Health. 2009 May;25(5):418-22.
[40a] Bernabe-Ortiz A, Curioso WH, Gonzales MA, Evangelista W, Castagnetto JM, Carcamo
CP, et al. Handheld computers for self-administered sensitive data collection: a comparative
study in Peru. BMC medical informatics and decision making. 2008;8:11.
[41a] Cheng K, Ernesto F, Truong K. Participant and Interviewer Attitudes toward Handheld
Computers in the Context of HIV/AIDS Programs in Sub-Saharan Africa. CHI: Healthcare in
the Developing World. Florence, Italy 2008.
[42a] Zwarenstein M, Seebregts C, Mathews C, Fairall L, Flisher AJ, Seebregts C, et al.
Handheld Computers For Survey and Trial Data Collection in Resource-Poor Settings:
Development and Evaluation of PDACT, a Palm™ Pilot Interviewing System. unpublished.
[43a] Blaya JA, Gomez W, Rodriguez P, Fraser H. Cost and implementation analysis of a
personal digital assistant system for laboratory data collection. Int J Tuberc Lung Dis. 2008
Aug;12(8):921-7.
[44a] Blaya JA, Cohen T, Rodriguez P, Kim J, Fraser HS. Personal digital assistants to collect
tuberculosis bacteriology data in Peru reduce delays, errors, and workload, and are acceptable to
users: cluster randomized controlled trial. Int J Infect Dis. 2009 May;13(3):410-8.
[45a] Forster D, Behrens RH, Campbell H, Byass P. Evaluation of a computerized field data
collection system for health surveys. Bull World Health Organ. 1991;69(1):107-11.
3
12. Blaya JA, Fraser HSF, Holt B. E-health technologies show promise in developing countries. Health Aff (Millwood). 2010;29(2):243-50.
APPENDIX Exhibit 2a Electronic Health Record Evaluations
System or Evaluation
Institution Country Type Outcome
Virginia System costs were US$750 for satellite
Commonwealth communication, and a fixed cost of a satellite
University phone (US$500), and monthly fees. They provided
[1a] Kenya Cost for 2700 patients.
Bhorugram Over 4 years immunizations increased from 45.4%
Rural Case- to 81.9% and 46.1% to 77.7% in DPT and polio
Dispensary control vaccines; antenatal registration increased from
[2a] India study 384 to 705 patients.
Decreased percentages of wrong entries and non-
St. Luke's Case- entries either of weight or height; Increases
Medical Philipp control seen in nutrition support services referrals to
Center [3a] ines study clinical dietitians and dietician productivity.
Kwonsun Staff & Increased staff productivity and satisfaction.
Health Center patient Did not increase staff decision abilities.
[4a] Korea surveys Increased visitors' satisfaction with services.
Advantages: physicians recorded improved
communication (95%); improved quality of care
(85%); accurate entry and retrieval of data
(80%); easy access to data (70%); usable in
physician liability cases (64%); reduced medical
errors (67%); enhanced productivity (59%);
Disadvantages: disease coding is a problem
Sur Hospital Physician (70%); system is time consuming (67% agree); and
[5a] Oman survey too slow (60%).
Advantages: improve clinical documentation,
consistency of health maintenance, access to
patients' data and research opportunities.
Euro Health Staff Disadvantages: negative impact on physician-
Group [6a] Serbia survey patient consultation time.
Advantages: EHR implementation was viewed
positively (66%); improved quality of care
(35%); 34% self-reported that EHRs improved
quality, this was associated with increased
utilization (odds ratio 3.03). IT could improve
quality of care in underserved settings (87%)
Indian Health Physician Disadvantages: decreased quality of patient–
Service [7a] USA survey doctor interaction (39%).
Higher availability of reports at district
Tororo health office compared to paper (79% vs. 100%),
District Before- no difference in quality, majority of staff
Hospital[8a] Uganda after interviewed appreciated system.
Hospital matron noticed a cluster of sexually
transmitted disease and therefore dispatched a
team to investigate. Also noted lack of child
Mosoriot immunizations and dispatched nurses to that
Medical site. Reports that previously took a clerk two
Record System User weeks, now take minutes; allowed the director to
[9a] Kenya opinion reassign two clerks to other duties
Mosoriot Duration of visits dropped from 41 to 31
Medical minutes; providers time with patients dropped
Record System Before- from a third to a sixth of workday; providers
[9a] Kenya after spent two thirds less time interacting with
13. Blaya JA, Fraser HSF, Holt B. E-health technologies show promise in developing countries. Health Aff (Millwood). 2010;29(2):243-50.
other staff and tripled their time spent in
personal activities; clerks spent two thirds
less time interacting with other staff and
almost doubled their time registering patients.
The EMR had higher overall completeness than the
paper system. High workloads, shortage of
Karolinska Random bedside hardware and lack of software features
Institute selection were prominent influential factors in the
[10a] Iran of records quality of documentation.
SOURCE: Authors’ Analysis
NOTES: Evaluations are in increasing order of strength with multiple
evaluations of a single system placed together. References can be found
in Appendix Exhibit 1a
14. Blaya JA, Fraser HSF, Holt B. E-health technologies show promise in developing countries. Health Aff (Millwood). 2010;29(2):243-50.
APPENDIX Exhibit 3a Laboratory Information Management
Systems (LIMS) and Pharmacy Information System Evaluations
System or Evaluation
Institution Country Type Outcome
Laboratory Information Management Systems (LIMS)
Cholera was isolated in 22.6% (7/31) of
samples sent to a central laboratory.
Information was relayed to hospital and
Sanjay Gandhi health authorities, who took strict measures
Post Graduate to improve hygiene at a festival.
Institute of Subsequently, the number of diarrhea cases
Medical during festival decreased and an epidemic was
Sciences [11a] India Descriptive averted.
Case- Productivity indexes showed an increase by
control 41% in number of patients handled and 28% in
Tesilab [12a] Mexico study number of tests processed.
Turn around times for routine samples
Karadeniz decreased from 1 to half day; number of
Technical samples processed increased a factor of 2;
University, Before- annual laboratory revenue increased 4 times,
[13a] Turkey after from 55,000 to 220,000 euro per month.
Pharmacy Information Systems
In 28.2% of medication orders there was
dubious or misleading information
Advantages: ease of data access and
ordering. Disadvantages: repetition of
Universidade de Descriptiv orders from previous days without a review
São Paulo [14a] Brazil e and incorrectly typed information.
Advantages: user-friendly interface;
quickness and clarity of information; ease
of use; reduction of time between drug
Hospital das prescription and administration; believed to
Clínicas da result in a drastic reduction in the risk of
Faculdade de error.
Medicina de Disadvantages: insufficient number of
Ribeirão Preto Staff terminals; system got stuck; technical
[15a] Brazil survey support was unsatisfactory.
Advantages: legibility (37.5%); less time to
order (20.5%); more practical and organized
(8%).
Disadvantages: repetition of previous
prescriptions (34%); typing mistakes (17%);
University of Staff dependence on computers (11%); alterations
São Paulo [16a] Brazil survey made manually (7%)
Over 70% of users preferred system over
paper, felt that it reduced the number of
prescription errors, and knew what to do
when system was down.
Its limitations were with system speed and
functionality in processing prescriptions.
National Satisfaction was more associated with
Healthcare Staff perceived impact on productivity than with
Group [17a] Singapore survey patient care.
Ekbatan Staff Clinician users of the prescribing system
Hospital [18a] Iran interviews were found to mostly rely on their memories
15. Blaya JA, Fraser HSF, Holt B. E-health technologies show promise in developing countries. Health Aff (Millwood). 2010;29(2):243-50.
and be overconfident which could lead to
errors.
Advantages: increased confidentiality,
reduction of medication errors and
educational benefits.
Disadvantages: high cost, social and
cultural barriers, data entry time and
problems with technical support.
Before intervention (Period 1), error rate
was 53%, which did not significantly change
after the implementation of CPOE without
Hamadan decision support (Period 2). However, errors
University of were significantly reduced to 34% after the
Medical Before- decision support was added to the CPOE
Sciences [19a] Iran after (Period 3).
Accuracy of prediction per medication was
Socios En Salud Model vs. 117% over-estimate in 2002, 5% underestimate
[20a] Peru actual use in 2003 and to 2% under-estimate 2004.
Model, For subgroup of 58 patients on
order individualized treatment, model predicted
Socios En Salud placed vs. 99% of actual use, the actual order placed
[21a] Peru actual use was 145% of actual use.
Externally 17.4% error rate fell significantly in the
controlled study group to 3.1% per patient. Error rate
Socios En Salud before- did not differ statistically in control
[22a] Peru after group (8.6% to 6.9%).
SOURCE: Authors’ Analysis
NOTES: Evaluations are in increasing order of strength with multiple
evaluations of a single system placed together. References can be found
in Appendix Exhibit 1a.
16. Blaya JA, Fraser HSF, Holt B. E-health technologies show promise in developing countries. Health Aff (Millwood). 2010;29(2):243-50.
Appendix Exhibit 4a Patient Registration and Scheduling,
Monitoring and Evaluation, and Clinical Decision Support
System Evaluations
System or Evaluation
Institution Country Type Outcome
Patient Registration and Scheduling
Most of the users (70%) expressed a clear
preference for the touch screen over the
paper system. However, every respondent
Baobab Clinical also identified on-going problems that need
Health[23a] Malawi user survey to be addressed.
Sustainable Mean time to locate record with fingerprint
Sciences Simultaneous scan was 7.0 (SD 3.5) seconds, versus 27.3
Institute randomized (SD 7.1) seconds using the traditional
[24a] Nicaragua controls method.
Average time to locate a patient’s chart
Sustainable using traditional methods was 2.9 (SD 2.1)
Sciences Simultaneous minutes, whereas using barcode-based
Institute randomized methods the average was 0.09 minutes, or
[24a] Nicaragua controls 5.5 (SD 1.2) seconds.
Monitoring, Evaluation, and Patient Tracking Systems
Data are invaluable for the short-term
Calmette management of the hospital. SIM helped set
Hospital [25a] Cambodia Descriptive up infection control committee.
Tanzanian Total annual systems cost was US$2,119,941,
Ministry of $0.13 per participant, and $0.06 per
Health [26a] Tanzania Cost capita.
For patients with CD4 counts between 101
and 350, those entered into the system
within 14 days had an odds ratio of 3.2 for
Case-control starting treatment within 14 days compared
HIV-EMR [27a] Haiti study to those without early CD4 entry.
Logged patient follow-up visits allowed
staff to rapidly identify a decline among
patients who had stopped receiving food
supplementation. New strategies were
HIV-EMR2.0 implemented within 3 weeks, and clinic
(OpenMRS) Case-control attendance returned to original level of
[27a] Haiti study over 90%.
University of Time taken to locate ten households was
the Simultaneous reduced by 20% and 50% in each of two
Witwatersrand South randomized communities using the PDA/GPS device
[28a] Africa controls compared to paper.
Sustainable
Sciences Simultaneous
Institute randomized GIS did not significantly decrease the time
[24a] Nicaragua controls necessary to locate a home.
Clinical Decision Support System (CDSS)
Chulalongkorn Nurses perceived they had better judgment
University Before-after and information access, all participants
[29a] Thailand qualitative wanted permanent installation.
Chulalongkorn
University Before-after Mean judgment performance score for case
[29a] Thailand quantitative simulations increased by 42%.
17. Blaya JA, Fraser HSF, Holt B. E-health technologies show promise in developing countries. Health Aff (Millwood). 2010;29(2):243-50.
Electronic
Integrated 84.7% of e-IMCI investigations had IMCI
Management of completed compared to 61% with the chart
Childhood Simultaneous booklet. Amount of time for both IMCI and
Illness (e- nonrandomize e-IMCI sessions averaged 12.5 minutes for
IMCI) [30a] Tanzania d controls the one clinician tested.
Increase of 430 new patient visits per
month at intervention sites, increase from
Early baseline of 18% at intervention sites
Diagnosis and compared with decline of 5% at control
Prevention sites. Intervention was associated with
System (EDPS) Longitudinal significant improvements in Global Patient
[31a] India RCT Assessment of Care Index.
SOURCE: Authors’ Analysis
NOTES: Evaluations are in increasing order of strength with multiple
evaluations of a single system placed together. References can be found
in Appendix Exhibit 1a.
18. Blaya JA, Fraser HSF, Holt B. E-health technologies show promise in developing countries. Health Aff (Millwood). 2010;29(2):243-50.
Appendix Exhibit 5a Patient Reminder and Research/Data
Collection Systems Evaluations
System or Country Evaluation
Institution Type Outcome
Patient Reminder Systems
On Cue
Compliance South Cost of 120 SMS reminders were
[32a] Africa Cost R13.90/patient/month (US$2.43).
Intervention had higher completion rate (10.6
vs. 3%), but similar cure rate (62.3 vs.
66.4%) and treatment success rate (73 vs.
On Cue 69%) compared to data from City of Cape
Compliance South Before- Town's TB Control Program for same clinic in
[32a] Africa after 2003.
It cost RM 0.45 per attendance for text
International messaging reminder as compared with RM 0.82
Medical Cost- per attendance for mobile phone reminder. The
University effectivene ratio of cost per unit attendance of text
Puchong [33a] Malaysia ss messaging versus mobile phone was 0.55.
Attendance rates of control, text messaging
and mobile phone reminder groups were 48.1,
59.0 and 59.6%, respectively. The text
messaging group was significantly higher than
International Simultaneou control group, no difference between text
Medical s messaging and mobile phone group. Text
University randomized messaging reminder system cost less than half
Puchong [33a] Malaysia controls of the mobile phone reminder per attendance.
Research/Data Collection Systems
There were no problems with the PDAs while
Ifakara collected data on 83,346 individuals over
Health seven weeks. Dataset was available within 24
Research & hours. Median time to form completion was 14
Development minutes during training and nine minutes
Centre [34a] Tanzania Descriptive during survey.
87% reported that health content received
Uganda helped them make faster more accurate
Health diagnoses. 86% integrated PDA into other
Information activities. 73% able to solve problems; 68%
Network reported problems with 41% of them being
[35a, 36a] Uganda User survey resolved due to lack of technical support.
System provides up to 91% saving per unit
Uganda spending compared to paper-based HMIS data
Health collection and reporting approaches.
Information Reporting compliance to MOH improved from
Network Cost national average of 63% to 94-100% for
[35a, 36a] Uganda analysis districts using UHIN.
Advantages: time savings (95 percent); the
ability to quickly mobilize or organize
individuals (91 percent); reaches audiences
previously difficult or impossible to reach
UN-Vodafone (74 percent); transmit data more quickly and
Partnership Multiple accurately (67 percent); gather data more
[37a] countries User survey quickly and accurately (59 percent).
Albert Gabon Self- Rate of discrepant entries was 1.7%.
