e-health technologies show promise in developing countries

By Joaquin A. Blaya, Hamish S.F. Fraser, and Brian Holt

E-Health Technologies ShowPromise In Developing Countries

ABSTRACT Is there any evidence that e-health—using informationtechnology to manage patient care—can have a positive impact indeveloping countries? Our systematic review of evaluations of e-healthimplementations in developing countries found that systems thatimprove communication between institutions, assist in ordering andmanaging medications, and help monitor and detect patients who mightabandon care show promise. Evaluations of personal digital assistantsand mobile devices convincingly demonstrate that such devices can bevery effective in improving data collection time and quality. Donors andfunders should require and sponsor outside evaluations to ensure thatfuture e-health investments are well-targeted.

E-health, defined as the “use of infor-mation and communications tech-nologies (ICT) in support of healthand health-related fields, includinghealth-care services, health surveil-

lance, health literature, and health education,knowledge and research,”1 has the potential togreatly improvehealth service efficiency, expandor scale up treatment delivery to thousands ofpatients in developing countries, and improvepatient outcomes.2 In this paper, the term is usedsynonymously with health information technol-ogy (IT).Information systems, such as electronic health

records (EHRs) and mobile phones and hand-held computers (also called m-health), can be ofenormous value in providing health care in mul-tiple settings. They can support a health workerperforming clinician duties where there are nodoctors and can help keep track of patients inHIV programs where the loss rate (patients whodrop out of treatment) can be as high as 76 per-cent.3 When used to monitor inventories, thesesystemscan save lives andprevent the increaseofdrug resistance by keeping medicines in stockand can provide accurate, timely information forstrategic planning, especially in areas where

hand-compiled data are often years out of date.Acknowledging this potential, the World HealthOrganization (WHO)has published amanual onimplementing EHRs for developing countries,4

and many agencies are funding e-health efforts.5

However, evaluations are essential to ensuringthat these systems are safe, beneficial, and not awaste of scant resources.6 The goal of this reviewwas to survey evaluations performed on e-healthsystems in developing countries, assess their po-tential impact, and guide future implementa-tions and evaluations.Evaluating the impact of e-health on patient

care is extremely difficult. Hence, there are fewrigorous evaluations worldwide.6 Systematic re-views of e-health in primary health care,7,8 tele-medicine,9 and its cost-effectiveness10 havefound that most articles “lacked any evaluationof their concrete application to health care.” Indeveloped countries, a few EHR system evalua-tions have shown that they have (1) improvedoutcomes for renal disease patients,11 (2) de-creased rates of clinical visits by 5–9 percent,12

(3) provideda five-yearbenefit ofUS$86,400perprovider at a large academic hospital,13 and(4) improved efficiency by 6 percent per yearin a large hospital network.14 Computerized phy-

doi: 10.1377/hlthaff.2009.0894HEALTH AFFAIRS 29,NO. 2 (2010): 244–251©2010 Project HOPE—The People-to-People HealthFoundation, Inc.

Joaquin A. Blaya ([email protected]) is a NationalLibrary of Medicine Fellow inthe Decision Systems Groupat Brigham and Women’sHospital in Brookline,Massachusetts.

Hamish S.F. Fraser is anassistant professor in theDivision of Global HealthEquity at Brigham andWomen’s Hospital and HarvardMedical School.

Brian Holt is a workflowanalyst in EMR workflowengineering at MassachusettsGeneral Hospital in Boston,Massachusetts.

244 HEALTH AFFAIRS FEBRUARY 2010 29:2

POLICIES & POTENTIAL

sician order entry systems have been shown toreduce medical errors,15 but they can also in-crease error rates if not well designed andimplemented.16

Study Data And MethodsSTUDIES ELIGIBLE FOR REVIEW In our survey of stud-ies for review, we included any qualitative orquantitative evaluation of information technol-ogy affecting health care in developing coun-tries. We did not include telemedicine becauseother recent reviews exist.9,17Developing countrieswere defined as those in the Emerging and De-veloping Economies List in the InternationalMonetary Fund’sWorld EconomicOutlook Report.Evaluations were excluded if (1) data complete-ness of the systemwas the only outcome, (2) theevaluation method was not described, (3) thearticle only described the feasibility or technicalevaluation of a system, (4) the evaluation was onattitudes toward or knowledge of e-health (notan actual system), or (5) it was only an educa-tional tool.18,19 In the case of the Uganda HealthInformation Network, we report on the e-healthcomponent of the system. If an article did nothave an abstract, we attempted to find the articlethrough the Harvard or Massachusetts Instituteof Technology (MIT) library systems.

FINDING RELEVANT STUDIES We conducted aworldwide review of the literature and requestedsubmissions from researchers and those imple-menting e-health in developing countries. Lit-erature searches were completed through Octo-ber 2009 without language restrictions throughMEDLINE, EMBASE, Science Citation Index(Web of Science), Social Sciences Citation Index,the Cochrane Library, and the Latin Americanand Caribbean Health Science Literature Data-base (LILACS). To find reports not in scientificjournals or conferences, we also used GoogleScholar. For MEDLINE and EMBASE searches,termswere derived from theMeSHdatabase andEMTREE tool, respectively. We searched formore than forty commonly used terms to de-scribe e-health applications, found the broadestterm within each tool that maintained its con-text, and then used that term for the search toensure that we included all possible studies.Among the termsused in the final strategiesweremedical informatics applications, reminder system,geographic information system, hospital informa-tion systems, outcome and process assessment(Health Care), evaluation studies, attitude, costsand cost analysis, developing countries, poverty,Africa, Latin America, eastern Europe, and centralor southeastern Asia (complete strategies areavailable from the authors on request). An initialreviewer read the abstracts to evaluate the elig-

ibility of all studies identified in our search. Asecond reviewer confirmed all relevant articlesand retrieved full-text articles. Supplementarymethodsof findingevaluations includeda reviewof article reference lists, informatics conferenceproceedings, information provided by primarystudy authors, requesting submissions fromother researchers and implementers, andsearching the RHINO Literature Database20 andother recent reviews.7,21–23

DATA ABSTRACTION AND SYNTHESIS We extracteddata according to recurring themes, defined be-low.Wesummarized these findingsusing tabulartechniques and descriptive statistics. Reportedanalyses were too disparate to be pooled in ameta-analysis.The systems described in the articles were

placed into one of eight categories correspond-ing to the typical applications used in developingcountries. The order of these categories does notinfer any priority:(1) Electronic health record: an electronic rec-

ord of health-related information on an indivi-dual that can be created, managed, or consultedby clinicians or staff. In literature, the term elec-tronic medical record is used interchangeably andis used as a synonym in this paper.(2) Laboratory information management sys-

tem: a system for laboratory-specific activities orfor reporting results to administrators andhealth care personnel.(3) Pharmacy information system: any system

used to order, dispense, or track medications ormedication orders including computerized or-der entry systems.(4) Patient registration or scheduling system:

any system used to monitor and manage themovement of patients through multistep proc-esses or to maintain a census.24 An example isadmissions-discharge-transfer systems.(5)Monitoring, evaluation, and patient track-

ing system: any systemused for aggregate report-ing of information, program monitoring, andtracking of patients’ status. Examples includedistrict health information systems or healthmanagement information systems.(6) Clinical decision support system: system

designed to improve clinical decisionmaking, inwhich characteristics of individual patients arematched to a computerized knowledge base andsoftware algorithms generate patient-specificrecommendations.25

(7) Patient reminder system: a system used topromptpatients to performa specific action—forexample, take medications or attend the clinic.(8) Research/data collection system: any sys-

tem used for collecting data from different loca-tions or for storing, managing, or reporting ondata used for research purposes.

FEBRUARY 2010 29:2 HEALTH AFFAIRS 245

Evaluations were classified into two majorcategories—qualitative and quantitative—asshown in Exhibit 1. Qualitative evaluations werethose where users gave opinions regarding asystem. These could be through questionnaires,focus groups, or interviews. (This definition isdifferent from the one proposed by AnselmStrauss and Juliet Corbin of “any type of researchthat produces findings not arrived at by statisti-cal procedures or other means of quantifica-tion.”)26 Quantitative evaluations were thosewhose outcomes were data quality, administra-tive changes, patient care, or economic assess-ment. Evaluation designs were grouped accord-ing to the definition by Charles Friedman andJeremy Wyatt:27(1) descriptive (uncontrolled)study; (2) historically controlled (before-after)study; (3) case-control (retrospective) study;(4) prospective self-controls (subjects perform-ing the same action in both systems; this cate-gorywasaddedby theauthors); (5)simultaneousnonrandomized controls; (6) simultaneous ran-domized controls; and (7) externally and intern-ally controlled before-after study. Two cost stud-ies and two studies modeling future medicationrequirements were categorized as self-controlsbecause they compared the impact of the systemagainst the same situation without the system.As a result of the inherent limitation of perform-ing a case-control, descriptive, or qualitativestudy without statistics, we do not commenton the limitations of these studies.

Study ResultsSearches retrieved 2,043 citations. Five articleswere excluded because they did not have ab-stracts and full-text versions were not avail-able.28–31 After the initial screeningof article titles

and abstracts, we found 126 articles that ap-peared relevant. An additional five articles wereidentified by hand-searching bibliographies ofeligible articles and prior reviews. Of these,forty-five fulfilled the inclusion criteria after fullreview of their abstracts. They are listed by typeof system and evaluation in Exhibit 1 and arecategorized by systems in Appendix Exhi-bits 2a–5a.32 We included an evaluation fromthe U.S. Indian Health Service, although it isnot in a developing country, because socioeco-nomic and infrastructure conditions among thepopulation treated are similar to those in devel-oping countries. If a system hadmultiple evalua-tions, only those with different outcomes arelisted. If they had the same outcome, we tookthe one with the largest sample size. There weretwo articles reporting an evaluation that did notoccur because of a failed system implementa-tion.33,34 These are not part of the results, butwe considered them relevant to list because ar-ticles onunsuccessful systems arenot commonlypublished.Fifteen articles performed qualitative evalua-

tions, and forty performed quantitative evalua-tions. If an evaluation performed both types, itwas counted in both categories. Two qualitativeevaluations and sixteen quantitative performedstatistical analysis. Of all evaluations, two(13 percent) of the qualitative and seven (18 per-cent) of the quantitative were performed by anoutside evaluator. The number of evaluationshas more than tripled comparing periods beforeand after 2002.

ELECTRONIC HEALTH RECORDS Because EHRs arethe core clinical application, they usually encom-pass a variety of functionalities, which makestheir implementations complex35 and prone tofailure.36 This complexity provides an additional

EXHIBIT 1

Number Of Articles Included In Analysis, By E-Health Category And Evaluation Type

E-health category Qualitative

Quantitative

Descriptive studies Controlled studiesElectronic health record 5 1 5Laboratory information management systems 0 1 2Pharmacy information systems 4 2 3Patient registration or scheduling systems 1 0 2Monitoring, evaluation, and patient tracking systems 0 2 4Clinical decision support systems 1 0 3Patient reminder systems 0 1 3Research/data collection systems 5 1 11

Total 15 8 32

SOURCE Authors’ analysis. NOTES The articles (n ¼ 45) are classified by e-health category and by type of evaluation. If an article had bothqualitative and quantitative studies or multiple types of systems, it was counted in both categories. Details about the evaluatedprojects are in Appendix Exhibits 2a–5a, available online as in Note 32.



challenge in their evaluation. Most evaluationsfound provided insight into possible impacts ofthese systems, but had limited scientific rigor, asseen in Appendix Exhibit 2a.32,27

The Indian Health Service’s Vista system wasthe most complete system we reviewed, and itsrigorous qualitative evaluation showed that amajority of clinicians viewed its implementationpositively and hence used it more. The MosoriotMedical Record System evaluation in Kenya pro-vides data on the impact that anEHRcanhave onimproving staff productivity and reducing pa-tient wait times. All other evaluations were qual-itative and provided insights into EHRs’ abilityto improve staff satisfaction, providing higher-quality data to relevant personnel and ultimatelyimproving patient care.

LABORATORY INFORMATION MANAGEMENT SYSTEMS

There were only three evaluations of laboratoryinformation management systems, all quantita-tive, with only one having a control group (Ap-pendix Exhibit 3a).32 However, they suggest twomajor benefits that such systems can provide:(1) decreasing times for communication of re-sults, and (2) improving the productivity of thelaboratory. An additional impact, reduction inerrors, has not yet been studied, although thereare groups currently performing such trials.37

PHARMACY INFORMATION SYSTEMS Computerizedorder entry can provide a key incentive for clin-ical staff, especially clinicians, to use an informa-tionsystem,because such systemscan reduce thetime to order medications (especially repeat or-ders) and provide easy access to past informa-tion. The four qualitative evaluations shown inAppendix Exhibit 3a32 cite these as their system’smain advantages. The two quantitative evalua-tions with a control group (Socios en Salud inPeru and Hamadan University of MedicalSciences in Iran) showed a reduction in errors,which is a main outcome cited in developedcountry studies. Anadditional benefit fromsomepharmacy systems in developing countries istheir ability to forecastmedication requirements(Socios en Salud in Peru). This is useful if acountry or organization needs to order medica-tions months in advance to get lower prices,which is currently the case for drug-resistantTB medications.

PATIENT REGISTRATION AND SCHEDULING The twoquantitative evaluations of registration systems,seen in Appendix Exhibit 4a,32 showed that fin-gerprint scanners and barcode readers de-creased the time to locate records by 74 percentand 97 percent, respectively. The small samplesize of thirty in these randomized controlledtrials was their biggest limitation. In the quali-tative evaluation of the Baobab system in Mala-wi, users preferred it to paper despite limitations

in training and technical support and the need tomaintain a parallel paper system.

MONITORING, EVALUATION, AND PATIENT TRACKING

SYSTEMS Evaluations of systems to track andmonitor patients’ status are limited to twocase-control studies performed by the same or-ganization in Haiti (Appendix Exhibit 4a).32

Both of these studies suggest that an electronicsystem can effectively alert staff of patients whohave “fallen through the cracks” and preventrates of patients lost to follow-up, which werefound to be as high as 76 percent (after twoyears) as reported in some HIV programs.3

Two randomized controlled trials looked atthe effect of Global Positioning Systems (GPS)in finding households once a patient has beenidentified. An evaluation from South Africashowed that GPS reduced the time to find ahousehold by 20–50 percent, whereas one fromNicaragua showed no difference between the pa-per andGPS systems. Both the SouthAfrican andNicaraguan systemswere tested in similar urbansettings with novice users, so no immediatereason for the difference can be found. Bothstudies had small sample sizes (identifyingten to fifty households) and lacked statisticalanalysis.Two evaluations, one descriptive and one cost

analysis, looked at monitoring departmentswithin a hospital in Cambodia and health estab-lishments nationwide in Tanzania. They suggestthat electronic systems can help allocate re-sources efficiently and improve infection controland can be relatively low cost, respectively. How-ever, both evaluations lacked detail on the tasksaffected, as well as control groups.

CLINICAL DECISION SUPPORT SYSTEM Decisionsupport systems have received attention for de-veloping countries as a possible solution to thelack of trained clinical personnel, especially inrural areas. The three quantitative evaluationsseen in Appendix Exhibit 4a32 were of high rigor.The expert system for mechanically ventilatednewborns showed that nurses performed betteron a standardized test and felt that they hadbetter judgment after receiving training on thesystem. The evaluation of the personal digitalassistant (PDA) device to perform the ElectronicIntegratedManagement of Childhood Illness ap-proach in Tanzania showed that more clinicalstaff completed the electronic questionnairecompared to the paper booklet. It also showedthat it took the same amount of time (12.5 min-utes) to fill out the questionnaire by eithermeth-od. The evaluation of the Early Diagnosis andPrevention System in India showed higher satis-faction among patients if they were seen by acomputer operator before their clinical visitand that there was a large increase in new pa-


tients at health centers with the system.However, the two studies with simultaneous

controlshadmajor limitations. Theevaluationofthe Electronic Integrated Management of Child-hood Illness was performed by the developers ofthe systems, and because the technology wasnew to the users, the novelty rather than its use-fulness could account for the additional comple-teness. In the case of the Early Diagnosis andPrevention Systems, the increased attendanceand patients’ opinions could have been easilybiased by the presence of the computers, themotivation of computer operators, and thelength of time spent with operator, none ofwhich were present at control sites.

PATIENT REMINDER SYSTEMS The quantitativeevaluation of the South African text messagingsystem (Appendix Exhibit 5a)32 found that afterthe system was implemented, there were highercompletion rates of TB treatment. However, thecomparison was made to the city’s TB programregister, for which the data quality was not ver-ified and the data were different from the sourceof the prospective data. A randomized trial inMalaysia found that both text messaging andmobile phone reminders significantly increasedattendance (by 21 percent) over the controlgroup. Although they both had similar effective-ness, the textmessaging systemwas half the costof the mobile phone reminders. This evaluationhad no major limitations.The Malaysian study performed a well-

designed cost-effectiveness study showing thattext messaging, implemented correctly, can be acost-effective method to increase clinic atten-dance. This is especially important since bothTB and HIV treatments require constant super-vision of patients and strict adherence to a dailyregimen of medications. Such systems can helppatients in resource-poor settings who encoun-ter many obstacles that can prevent them fromgetting their medications.

RESEARCH/DATA COLLECTION SYSTEMS Research/data collection systems was the group with thelargest number and most rigorous evalua-tions (Appendix Exhibit 5a).32 All systems, ex-cept the Mexican National Institute of PublicHealth’s Audio Computer-Assisted Self-Inter-view (ACASI) system, were on PDAs. Four ran-domized trials showed that the main benefits ofPDA-based systems were data qual-ity similar to paper systems or high-er, less time taken to perform inter-views, and decreased time to collectdata. However, many of the studieshad major limitations. The systemsfrom the Universidad PeruanaCayetano Heredia and the SouthAfrican Medical Research Council

compared the PDA system to paper and not toa gold standard. The study performed by Sociosen Salud had a small number of users (n ¼ 4),and the studyperformedby theLondonSchool ofEconomics was performed seventeen years ago.The organizations that implemented the PDA-based systems in Uganda and South Africa haveexperience with hundreds of users and morethan a dozen implementations combined, whichempirically shows the feasibility of such systems.The cost analyses show that these systems are

able to recoup the high initial costs by providingincreased efficiency and continuous materialcosts. The Uganda system showed a cost savingsof 91 percent over the paper system. The SouthAfrican analysis calculated that after using thePDA system for data collection in eight studies ofmedium scale, it would equal the costs of paper.The PDA system inPeruwould pay for expansionto other health districts in three months as aresult of increased efficiency.

DiscussionThis review shows that with the exception ofPDA-based data collection, there are still fewscientifically rigorous data on the effectivenessand cost-effectiveness of e-health systems in de-veloping countries. Further, the evaluationshave mostly been performed by organizationsconnected to academic settings andnot by other,larger recipients of donor funding.When lookingat the software systems included in theU.S. Pres-ident’s Emergency Plan for AIDS Relief (PEP-FAR) Anti-Retroviral Therapy (ART) SoftwareInventory Report5 and EngenderHealth–Open-Society software tools38 comparison, only threesystems, the Partners in Health—ElectronicMedical Record/HIV—Electronic Medical Rec-ord in Kenya, Mosoriot Medical Record Systemin Kenya, and Vista in the U.S. Indian HealthService, have had any evaluations performed.Although a few studies have been commissionedby the U.S. Centers for Disease Control and Pre-vention (CDC), it is particularly important thatlarge funders such as the U.S. Agency for Inter-national Development or PEPFAR include re-sources for the evaluation of e-health systemsdeveloped and deployed in developing countriesand perhaps make them a requirement for con-

tinued funding. This could includestandard designs for studies thatare suitable for resource-poor en-vironments, that minimize biases,and that are easily comparable tothe results from other projects.The overall pattern of e-health

evaluations in developed countriesreflects an initial focus on qualita-



tive anddescriptive evaluations,with an increasein the number of quantitative and larger evalua-tions published in the past decade. Developingcountries seem to be following this pattern aswell, so in this study we foundmostly qualitativeanddescriptive studies but sawan increase in thenumber of randomized trials performed in thepast few years. This suggests that as e-healthimplementations become more robust in devel-oping countries, we can expect more rigorousstudies, such as randomized trials or cost-effec-tiveness studies.Initial evaluations suggest that the following

functions are of positive impact in developingcountries:(1) Ability to track patients through the treat-

ment initiation process,monitor adherence, anddetect thoseat risk for loss to follow-up. (2)Toolstodecrease communication timesof informationwithin and between institutions. (3) Tools tolabel or register samples and patients. (4) Abilityto electronically monitor and remind patients ofhealth care needs or treatment. (5) Collection ofclinical or research data using PDA applications.(6) Reductions in errors in laboratory and med-ication data.Important findings include the user prefer-

ence for the Baobab touch-screen system inMalawi, one of the only fully electronic point-of-care systems in use in Africa, which is nowin daily use for more than 35,000 HIV patients.The benefit shown for patient tracking andreminders is also important, given the loss tofollow-up rate of up to 76 percent for HIV pa-tients in Africa.3 The Malaysian systems thattexted patient reminders showed a significantdecrease in missed visits, at a reasonably lowcost, and the On Cue Compliance Service inSouth Africa was well liked by users several yearsafter implementation and, perhaps more impor-tant, by an independent evaluation team. Thesesystems can be of high value because intermit-tent treatment puts patients at grave risk ofdeterioration and death, as well as causing in-creased drug resistance and further transmis-sion of disease to the wider community.Tools to store and communicate suchdatawith

low error rates have been early successes in de-veloped countries, and the positive evaluationsdescribed here should drive their use in the de-veloping world. Evaluations of PDAs andmobiledevices were particularly rigorous, and they con-vincingly demonstrate that such devices can bevery effective in improving data collection timeand quality. An additional benefit is their lightweight and lack of printing costs compared tolarge paper forms, which is crucial in remoteareas with poor infrastructure. These resultsare important for the growing field of mobile

health and cell phone–based tools, because thesedevices are also playing an increasing role incommunication directly with patients.Evaluations of e-health systems are chal-

lenging and require significant resources in ad-dition to funds creating and implementing sys-tems. Implementations should have evaluationsbuilt into the process. This will provide usefulfeedback to improve the project (formative eval-uations) and will also demonstrate the impact ofthe system in the long term (summative evalua-tions). Evaluations in resource-poor environ-ments face many challenges when compared tothose in developed countries, such as the physi-cal environment, power, networking, and avail-ability of technical staff. Measures of short- andlong-term system usage and data completenessare important and a necessary prerequisite to afull evaluation study. Poor data quality is a con-stant problem in health projects, whether theyuse paper or electronic systems, so tools that canreduce errors as well as benefiting other aspectsof care are likely to be well received.Some benefits of electronic systems are diffi-

cult to quantify. One is the ability to performoperational research with greatly reduced costs.During our search we found eight studies thatused electronic databases and probably couldnot have been performed if manual data collec-tionwas required. Another is the impact of emer-gency communication across large distances,such as in the cholera outbreak in India or refu-gee situations.39 The strongest evidence for ben-eficial impact of e-healthonhealth carewill comefrom long-term follow-up of this sort carried outby independent evaluators.

ConclusionsWith the rapid growth of e-health in developingcountries, there is clearly an urgent need forsolid evidence of its impact to justify and guidethe investment of resources in such systems.Despite major increases in evaluations in recentyears,most large e-health implementations havelittle or no evaluation data. To date,most studieshave been small; focused on process indicatorsrather thanpatient outcomes, or on the attitudesof users and patients; and performed mostly byacademic groups. An increased focus on includ-ing evaluations as part of e-health implementa-tions is necessary and should be adopted by or-ganizations implementing or funding suchsystems. One method is for large funders to in-clude resources for evaluations or make them arequirement for implementation.Although evaluations of important indicators

of care are difficult to do well, this review hasconfirmed that they are feasible even in very


challenging environments. Initial benefits wereshown in systems that track patients throughtreatment initiation,monitor adherence, andde-tect those at risk for loss to follow-up; tools todecrease information communication timeswithin andbetween institutions, aswell as errorsin reporting laboratory data; barcoding for pa-tient identification cards and laboratory sam-ples; handheld devices for collecting and acces-sing data; and the ordering and management of

medications. Because of the lack of infrastruc-ture and backup systems in resource-poor envir-onments, well-designed e-health solutions mayhave amuch larger impact onquality of care thanin more developed areas. As e-health becomeswidespread in developing countries, these andother benefits will need to be identified by morerigorous evaluations that include long-termfollow-up and are carried out by independentevaluators. ▪

An initial version of this paper wasrequested by the Rockefeller Foundationfor the Making the eHealth Connectionconference held in Bellagio, Italy, in July2008. This paper was funded by theRockefeller Foundation. Joaquin A. Blayais cofounder of eHealth Systems, a

Chilean company that provides healthinformatics consulting and technology inLatin America. The authors acknowledgethose who took the time to provideadditional information: Holly Ladd andBerhane Gebru from AED-Satellife,Libby Levison, Heather Zornetzer,

Veronica Rojas, Adesina Iluyemi,Mauricio Soto, Waldo Ortega, ChrisBailey, Patrick Whitaker, Gerry Douglas,Natasha Kanagat, Steve Yoon, ZachLandis Lewis, Joel Selanikio, and NealLesh. Finally, the authors thank ClaireMack for her invaluable editing.

NOTES

1 World Health Organization. 58thWorld Health Assembly Report; 16–25 May 2005. Geneva: WHO; 2005.

2 Edworthy SM. Telemedicine in de-veloping countries. BMJ. 2001;323(7312):524–5.

3 Rosen S, Fox MP, Gill CJ. Patientretention in antiretroviral therapyprograms in sub-Saharan Africa: asystematic review. PLoS Med. 2007Oct 16;4(10):e298.

4 World Health Organization. Elec-tronic health records: a manual fordeveloping countries. Geneva:WHO; 2007.

5 U.S. President’s Emergency Plan forAIDS Relief. PEPFAR Software In-ventory Report. Washington (DC):PEPFAR; 2004.

6 Rigby M. Impact of telemedicinemust be defined in developingcountries. BMJ. 2002;324(7328):47–8.

7 Tomasi E, Facchini LA, Maia MF.Health information technology inprimary health care in developingcountries: a literature review. BullWorld Health Organ. 2004;82(11):867–74.

8 Mitchell E, Sullivan F. A descriptivefeast but an evaluative famine: sys-tematic review of published articleson primary care computing during1980–97. BMJ. 2001;322(7281):279–82.

9 Roine R, Ohinmaa A, Hailey D. As-sessing telemedicine: a systematicreview of the literature. CMAJ.2001;165(6):765–71.

10 Whitten PS, Mair FS, Haycox A, MayCR, Williams TL, Hellmich S. Sys-tematic review of cost effectivenessstudies of telemedicine interven-tions. BMJ. 2002;324(7351):1434–7.

11 Pollak VE, Lorch JA. Effect of elec-tronic patient record use on mor-tality in end stage renal disease, amodel chronic disease: retrospectiveanalysis of nine years of prospec-

tively collected data. BMC Med In-form Decis Mak. 2007;7(1):38.

12 Garrido T, Jamieson L, Zhou Y,Wiesenthal A, Liang L. Effect ofelectronic health records in ambu-latory care: retrospective, serial,cross sectional study. BMJ.2005;330(7491):581.

13 Wang SJ, Middleton B, Prosser LA,Bardon CG, Spurr CD, Carchidi PJ,et al. A cost-benefit analysis of elec-tronic medical records in primarycare. Am J Med. 2003;114(5):397–403.

14 Evans DC, Nichol WP, Perlin JB.Effect of the implementation of anenterprise-wide electronic healthrecord on productivity in the Veter-ans Health Administration. HealthEcon Policy Law. 2006;1(Pt2):163–9.

15 Bates DW, Teich JM, Lee J, Seger D,Kuperman GJ, Ma’Luf N, et al. Theimpact of computerized physicianorder entry on medication errorprevention. J Am Med Inform Assoc.1999;6(4):313–21.

16 Koppel R, Metlay JP, Cohen A,Abaluck B, Localio AR, Kimmel SE,et al. Role of computerized physicianorder entry systems in facilitatingmedication errors. JAMA. 2005;293(10):1197–203.

17 Hersh WR, Hickam DH, SeveranceSM, Dana TL, Pyle Krages K, HelfandM. Diagnosis, access, and outcomes:update of a systematic review oftelemedicine services. J TelemedTelecare. 2006;12(Suppl 2):S3–31.

18 Mallapaty G, Kim S, Astion ML.Using interactive software to teachimage-based clinical laboratory testsin developing countries: a pilot trialin Nepal. Clin Chem Lab Med.2003;41(5):711–3.

19 Edmonson SR, Esquivel A,Mokkarala P, Johnson CW, PhelpsCL. Using technology to teach tech-nology: design and evaluation of bi-

lingual online physician educationabout electronic medical records.AMIA Annu Symp Proc. 2005:946.

20 Routine Health Information Net-work. RHINO Literature Database[Internet]. Boston (MA): RoutineHealth Information Network(RHINO); 2008 [cited 2010 Jan 4].Available from: http://www.iphealth.info/refbase/index.php

21 Fraser HS, Biondich P, Moodley D,Choi S, Mamlin BW, Szolovits P.Implementing electronic medicalrecord systems in developing coun-tries. Inform Prim Care. 2005;13(2):83–95.

22 Forster M, Bailey C, Brinkhof MW,Graber C, Boulle A, Spohr M, et al.Electronic medical record systems,data quality and loss to follow-up:survey of antiretroviral therapy pro-grammes in resource limited set-tings. Bull World Health Organ.2008;86(12):939–47.

23 Fraser HS, Allen C, Bailey C, DouglasG, Shin S, Blaya J. Information sys-tems for patient follow-up andchronic management of HIV andtuberculosis: a life-saving technologyin resource-poor areas. J Med In-ternet Res. 2007;9(4):e29.

24 Shortliffe EH, Perreault LE, editors.Medical informatics: computer ap-plications in health care and bio-medicine. 2nd edition. New York(NY): Springer; 2001.

25 Garg AX, Adhikari NK, McDonald H,Rosas-Arellano MP, Devereaux PJ,Beyene J, et al. Effects of compu-terized clinical decision supportsystems on practitioner performanceand patient outcomes: a systematicreview. JAMA. 2005;293(10):1223–38.

26 Strauss A, Corbin J. Basics of quali-tative research: grounded theoryprocedures and techniques. New-bury Park (CA): Sage; 1990.

27 Friedman CP, Wyatt JC. Evaluation



methods in medical informatics. 2ndedition. New York: Springer-Verlag; 2005.

28 Halbwachs H. The technical and fi-nancial impact of systematic main-tenance and repair services withinhealth systems of developing coun-tries. Health Estate. 1999;53(4):6–8,10–1.

29 Desikan P, Koram MR, Trivedi SK,Jain A. An evaluation of the effec-tiveness of the laboratory informa-tion system (LIS) with special refer-ence to the microbiology laboratory.Indian J Path Microbiol.2005;48(3):418.

30 Janecki J, Podsiadly T. Computer-assisted analysis of patients’ medicalrecords. Pol Tyg Lek. 1992;47(20–21):470–2.

31 Swaminathan R, Black RJ,Sankaranarayanan R. Database oncancer survival from developingcountries. IARC Sci Publ. 1998;(145):19–25.

32 The Appendix Exhibits are available

online; see the Appendix Exhibitslink in the box to the right of thearticle online.

33 Littlejohns P, Wyatt JC, Garvican L.Evaluating computerised health in-formation systems: hard lessons stillto be learnt. BMJ. 2003;326(7394):860–3.

34 Iluyemi A, Briggs J, Fitch T. Elec-tronic health records in developingcountries, integrating with mobiletechnology and legacy systems forcommunity based health workers:organisational and end-users’ issues.Proceedings of the European Con-ference on Information Manage-ment and Evaluation. Montpellier,France; 2007 Sep 20–21.

35 Brender J, Ammenwerth E, NykanenP, Talmon J. Factors influencingsuccess and failure of health infor-matics systems—a pilot Delphistudy. Methods Inf Med. 2006;45(1):125–36.

36 Heeks R. Information systems anddeveloping countries: failure, suc-

cess, and local improvisations. In-formation Society. 2002;18(2):101–12.

37 Blaya JA, Shin SS, Yagui MJ, Yale G,Suarez CZ, Asencios LL, et al. AWeb-based laboratory information systemto improve quality of care of tuber-culosis patients in Peru: functionalrequirements, implementation, andusage statistics. BMC Med InformDecis Mak. 2007;7:33.

38 EngenderHealth–Open Society In-stitute. Health toolkit: informationmanagement challenges and oppor-tunities for community-based orga-nizations serving people living withHIV/AIDS. New York: Engender-Health–Open Society Insti-tute; 2004.

39 Babille M, Decolombani P, Guerra R,Zagaria N, Zanetti C. Post-emergency epidemiological surveil-lance in Iraqi-Kurdish refugee campsin Iran. Disasters. 1994;18(1):58–75.

ABOUT THE AUTHORS

Joaquin A. Blaya

Hamish FraserCoauthors and frequentcollaborators Joaquin Blayaand Hamish Fraser share apassion for using e-healthtechnologies to improvehealth care in Latin America,

Africa, and Asia. Blaya, 31,who was born in Chile, is aHarvard and MassachusettsInstitute of Technology(MIT)–trained Ph.D. in healthsciences and technology.Fraser, age 47, was born inScotland and was educatedand trained in medicine andcardiology in the UnitedKingdom. They met in 2004when Blaya was at a jointHarvard-MIT programworking on his Ph.D. andFraser became hissupervisor. Then, as now,Fraser was an assistantprofessor of medicine atHarvard Medical School anddirector of informatics andtelemedicine at thenonprofit organizationPartners in Health, whichfocuses on providing healthcare for the poor in anumber of developingcountries, including Haiti,Rwanda, and Peru.

Back then, Fraser wasworking on developing and

implementing an electronichealth record for use inmanaging multidrug-resistantTB patients in Peru. He andBlaya teamed up to producea Palm Pilot–based systemto collect laboratory resultson behalf of these patients.In a study published in 2009in the International Journalof Infectious Diseases, thesystem was shown todecrease delays in gettingthose results from thirtydays to eight days, and toreduce errors in thecommunication of thesetests to clinicians by 59percent.

Since then, the two haveworked on implementing aWeb-based system tocommunicate laboratoryresults to TB clinicians inmore than 220 healthcenters throughout Peru.Fraser’s group (theElectronic Medical RecordsTeam at Partners in Health),with the Regenstrief

Institute in the UnitedStates, the MedicalResearch Council in SouthAfrica, and others, havedeveloped an “open source,”or nonproprietary, electronichealth record system fordeveloping countries, calledOpenMRS. The system isused by more than forty-fiveorganizations in twenty-three countries and isavailable for download athttp://www.openmrs.org.

“My focus has been onpractical systems that areuseful for doctors and otherhealth care staff,” saysFraser, who is also anassociate physician at theBrigham and Women’sHospital in Boston. Inaddition to his medicaldegree, he trained in thedevelopment and use of so-called knowledge-basedsystems—computer systemsto diagnose and analyzereal-world data—atEdinburgh University in the

United Kingdom. He alsocompleted a fellowship inclinical decision making andcardiology at MIT and theNew England Medical Center.

Blaya, who today is aresearch fellow at Partnersin Health, is also a NationalLibrary of Medicine Fellowat Harvard Medical School.In addition, he recentlycofounded a company,eHealth Systems, which aimsto implement open-sourcetechnologies, includingOpenMRS, in health systemsin Latin America. Havingemigrated from Chile toMiami, Florida, twenty-twoyears ago, he plans to moveback to Chile in 2010. Hisfive-year goal is for amajority of public healthcenters in Chile to useOpenMRS and to expandtheir use in Nicaragua,Argentina, Brazil, and othercountries.


Blaya JA, Fraser HSF, Holt B. E-health technologies show promise in developing countries. Health Aff (Millwood). 2010;29(2):243-50.

1

APPENDIX Exhibit 1a: Additional References [1a] Merrell RC, Merriam N, Doarn C. Information support for the ambulant health worker. Telemed J E Health. 2004 Winter;10(4):432-6. [2a] Singh AK, Kohli M, Trell E, Wigertz O, Kohli S. Bhorugram (India): revisited. A 4 year follow-up of a computer-based information system for distributed MCH services. International journal of medical informatics. 1997 Apr;44(2):117-25. [3a] Llido LO. The impact of computerization of the nutrition support process on the nutrition support program in a tertiary care hospital in the Philippines: report for the years 2000-2003. Clin Nutr. 2006 Feb;25(1):91-101. [4a] Chae YM, Kim SI, Lee BH, Choi SH, Kim IS. Implementing health management information systems: measuring success in Korea's health centers. Int J Health Plann Manage. 1994 Oct-Dec;9(4):341-8. [5a] Al Farsi M, West DJ, Jr. Use of electronic medical records in Oman and physician satisfaction. J Med Syst. 2006 Feb;30(1):17-22. [6a] Weinhara M, Stoicu-Tivadar L, Dagres C. Early stage testing of user's satisfaction after implementation of a central electronic health record (EHR) system in Serbia. Journal on Information Technology in Healthcare. 2009;7(2):127-33. [7a] Sequist TD, Cullen T, Hays H, Taualii MM, Simon SR, Bates DW. Implementation and use of an electronic health record within the Indian Health Service. J Am Med Inform Assoc. 2007 Mar-Apr;14(2):191-7. [8a] Ndira SR, Rosenberger KD, Wetter T. Assessment of data quality of and staff satisfaction with an electronic health record system in a developing country (Uganda): A qualitative and quantitative comparative study. Methods of Information in Medicine. 2008 2008;47(6):489-98. [9a] Rotich JK, Hannan TJ, Smith FE, Bii J, Odero WW, Vu N, et al. Installing and implementing a computer-based patient record system in sub-Saharan Africa: the Mosoriot Medical Record System. J Am Med Inform Assoc. 2003 Jul-Aug;10(4):295-303. [10a] Pourasghar F, Malekafzali H, Koch S, Fors U. May not fit Factors influencing the quality of medical documentation when a paper-based medical records system is replaced with an electronic medical records system: an Iranian case study. Int J Technol Assess Health Care. 2008 Fall;24(4):445-51. [11a] Ayyagari A, Bhargava A, Agarwal R, Mishra SK, Mishra AK, Das SR, et al. Use of telemedicine in evading cholera outbreak in Mahakumbh mela, Prayag, UP, India: An encouraging experience. Telemedicine Journal and E-Health. 2003;9(1):89-94. [12a] Alvarez Flores MG, Guarner J, Terres Speziale AM. [Productivity before and after installing a computerized system in a clinical laboratorya]. Rev Invest Clin. 1995 Jan-Feb;47(1):29-34. [13a] Turhan K, Kayikcioglu T. Implementation of a virtual private network-based laboratory information system serving a rural area in Turkey. Laboratory Medicine. 2006;37(9):527-31. [14a] Cassiani SH, Freire CC, Gimenes FR. [Electronic medical prescription at a university hospital: writing failures and users' opinions]. Rev Esc Enferm USP. 2003 Dec;37(4):51-60. [15a] Costa AL, de Oliveira MM, Machado Rde O. An information system for drug prescription and distribution in a public hospital. International journal of medical informatics. 2004 May;73(4):371-81. [16a] Gimenes FRE, Miasso AI, De Lyra Jr DP, Grou CR. Electronic prescription as contributing factor for hospitalized patients' safety. Pharmacy Practice. 2006;4(1):13-7.


2

[17a] Tan WS, Phang JS, Tan LK. Evaluating user satisfaction with an electronic prescription system in a primary care group. Ann Acad Med Singapore. 2009 Jun;38(6):494-7. [18a] Kazemi A, Ellenius J, Tofighi S, Salehi A, Eghbalian F, Fors UG. CPOE in Iran--a viable prospect? Physicians' opinions on using CPOE in an Iranian teaching hospital. International journal of medical informatics. 2009 Mar;78(3):199-207. [19a] Kazemi A, Ellenius J, Pourasghar F, Tofighi S, Salehi A, Amanati A, et al. The Effect of Computerized Physician Order Entry and Decision Support System on Medication Errors in the Neonatal Ward: Experiences from an Iranian Teaching Hospital. Journal of Medical Systems. 2009. [20a] Fraser H, Jazayeri D, Choi S, Blaya J, Bayona J, Levison L, et al. Forecasting three years drug supply for a large MDR-TB treatment program in Peru. Int J Tuber Lung Dis. 2006;10(11 Suppl. 1):S245. [21a] Yamanija J, Durand R, Bayona J, Blaya J, Jazayeri D, Fraser H. Comparing actual medication consumption against the quantities ordered and a prediction using an information system. Int J Tuber Lung Dis. 2006;10(11 Suppl. 1):S69-S70. [22a] Choi SS, Jazayeri DG, Mitnick CD, Chalco K, Bayona J, Fraser HS. Implementation and initial evaluation of a Web-based nurse order entry system for multidrug-resistant tuberculosis patients in Peru. Medinfo. 2004;11(Pt 1):202-6. [23a] CDC Global AIDS Program. Responses to the Touchscreen System User Survey: Queen Elizabeth Central Hospital. Malawi: CDC Global AIDS Program; 2007. [24a] Aviles W, Ortega O, Kuan G, Coloma J, Harris E. Quantitative assessment of the benefits of specific information technologies applied to clinical studies in developing countries. Am J Trop Med Hyg. 2008 Feb;78(2):311-5. [25a] Fabre-Teste B, Sokha O. [Calmette Hospital, Phnom Penh, Cambodia. Assessment of the implementation of the Medical Information System (SIM). Global analysis of the 1998 results]. Sante. 1999 Nov-Dec;9(6):367-75. [26a] Rommelmann V, Setel PW, Hemed Y, Angeles G, Mponezya H, Whiting D, et al. Cost and results of information systems for health and poverty indicators in the United Republic of Tanzania. Bull World Health Organ. 2005 Aug;83(8):569-77. [27a] Fraser HSF, Allen C, Bailey C, Douglas G, Shin S, Blaya J. Information systems for patient follow-up and chronic management of HIV and tuberculosis: A life-saving technology in resource-poor areas. Journal of Medical Internet Research. 2007;9(4):38. [28a] Dwolatzky B, Trengove E, Struthers H, McIntyre JA, Martinson NA. Linking the global positioning system (GPS) to a personal digital assistant (PDA) to support tuberculosis control in South Africa: a pilot study. International journal of health geographics. 2006;5:34. [29a] Jirapaet V. A computer expert system prototype for mechanically ventilated neonates development and impact on clinical judgment and information access capability of nurses. Comput Nurs. 2001 Sep-Oct;19(5):194-203. [30a] DeRenzi B, Lesh N, Parickh T, Sims C, Mitchell M, Maokola W, et al. e-IMCI: Improving Pediatric Health Care in Low-Income Countries. CHI. Florence, Italy 2008. [31a] Peters DH, Kohli M, Mascarenhas M, Rao K. Can computers improve patient care by primary health care workers in India? International Journal for Quality in Health Care. 2006;18(6):437-45. [32a] Bridges.org. Evaluation of the On Cue Compliance Service Pilot: Testing the use of SMS reminders in the treatment of Tuberculosis in Cape Town, South Africa. Cape Town: City of


3

Cape Town Health Directorate and the International Development Research Council (IDRC); 2005. [33a] Leong KC, Chen WS, Leong KW, Mastura I, Mimi O, Sheikh MA, et al. The use of text messaging to improve attendance in primary care: a randomized controlled trial. Fam Pract. 2006 Dec;23(6):699-705. [34a] Shirima K, Mukasa O, Schellenberg JA, Manzi F, John D, Mushi A, et al. The use of personal digital assistants for data entry at the point of collection in a large household survey in southern Tanzania. Emerg Themes Epidemiol. 2007;4:5. [35a] Bridges.org. Evaluation of the SATELLIFE PDA Project, 2002: Testing the use of handheld computers for heathcare in Ghana, Uganda, and Kenya. Boston, MA: Satellife; 2003. [36a] Satellife and Uganda Chartered HealthNet. Uganda Health Information Network, Phase-III: June 9, 2006 – June 8, 2007. Boston: Satellife and Uganda Chartered HealthNet; 2007. [37a] Kinkade S, Verclas K. Wireless Technology for Social Change. Washington, DC: UN Foundation-Vodafone Group Foundation Partnership; 2008. [38a] Missinou MA, Olola CH, Issifou S, Matsiegui PB, Adegnika AA, Borrmann S, et al. Short report: Piloting paperless data entry for clinical research in Africa. Am J Trop Med Hyg. 2005 Mar;72(3):301-3. [39a] Gutierrez JP, Torres-Pereda P. Acceptability and reliability of an adolescent risk behavior questionnaire administered with audio and computer support. Revista Panamericana De Salud Publica-Pan American Journal of Public Health. 2009 May;25(5):418-22. [40a] Bernabe-Ortiz A, Curioso WH, Gonzales MA, Evangelista W, Castagnetto JM, Carcamo CP, et al. Handheld computers for self-administered sensitive data collection: a comparative study in Peru. BMC medical informatics and decision making. 2008;8:11. [41a] Cheng K, Ernesto F, Truong K. Participant and Interviewer Attitudes toward Handheld Computers in the Context of HIV/AIDS Programs in Sub-Saharan Africa. CHI: Healthcare in the Developing World. Florence, Italy 2008. [42a] Zwarenstein M, Seebregts C, Mathews C, Fairall L, Flisher AJ, Seebregts C, et al. Handheld Computers For Survey and Trial Data Collection in Resource-Poor Settings: Development and Evaluation of PDACT, a Palm™ Pilot Interviewing System. unpublished. [43a] Blaya JA, Gomez W, Rodriguez P, Fraser H. Cost and implementation analysis of a personal digital assistant system for laboratory data collection. Int J Tuberc Lung Dis. 2008 Aug;12(8):921-7. [44a] Blaya JA, Cohen T, Rodriguez P, Kim J, Fraser HS. Personal digital assistants to collect tuberculosis bacteriology data in Peru reduce delays, errors, and workload, and are acceptable to users: cluster randomized controlled trial. Int J Infect Dis. 2009 May;13(3):410-8. [45a] Forster D, Behrens RH, Campbell H, Byass P. Evaluation of a computerized field data collection system for health surveys. Bull World Health Organ. 1991;69(1):107-11.

APPENDIX Exhibit 2a Electronic Health Record Evaluations

System or Institution Country

Evaluation Type Outcome

Virginia Commonwealth University [1a] Kenya Cost

System costs were US$750 for satellite communication, and a fixed cost of a satellite phone (US$500), and monthly fees. They provided for 2700 patients.

Bhorugram Rural Dispensary [2a] India

Case-control study

Over 4 years immunizations increased from 45.4% to 81.9% and 46.1% to 77.7% in DPT and polio vaccines; antenatal registration increased from 384 to 705 patients.

St. Luke's Medical Center [3a]

Philippines

Case-control study

Decreased percentages of wrong entries and non-entries either of weight or height; Increases seen in nutrition support services referrals to clinical dietitians and dietician productivity.

Kwonsun Health Center [4a] Korea

Staff & patient surveys

Increased staff productivity and satisfaction. Did not increase staff decision abilities. Increased visitors' satisfaction with services.

Sur Hospital [5a] Oman

Physician survey

Advantages: physicians recorded improved communication (95%); improved quality of care (85%); accurate entry and retrieval of data (80%); easy access to data (70%); usable in physician liability cases (64%); reduced medical errors (67%); enhanced productivity (59%); Disadvantages: disease coding is a problem (70%); system is time consuming (67% agree); and too slow (60%).

Euro Health Group [6a] Serbia

Staff survey

Advantages: improve clinical documentation, consistency of health maintenance, access to patients' data and research opportunities. Disadvantages: negative impact on physician-patient consultation time.

Indian Health Service [7a] USA

Physician survey

Advantages: EHR implementation was viewed positively (66%); improved quality of care (35%); 34% self-reported that EHRs improved quality, this was associated with increased utilization (odds ratio 3.03). IT could improve quality of care in underserved settings (87%) Disadvantages: decreased quality of patient–doctor interaction (39%).

Tororo District Hospital[8a] Uganda

Before-after

Higher availability of reports at district health office compared to paper (79% vs. 100%), no difference in quality, majority of staff interviewed appreciated system.

Mosoriot Medical Record System [9a] Kenya

User opinion

Hospital matron noticed a cluster of sexually transmitted disease and therefore dispatched a team to investigate. Also noted lack of child immunizations and dispatched nurses to that site. Reports that previously took a clerk two weeks, now take minutes; allowed the director to reassign two clerks to other duties

Mosoriot Medical Record System [9a] Kenya

Before-after

Duration of visits dropped from 41 to 31 minutes; providers time with patients dropped from a third to a sixth of workday; providers spent two thirds less time interacting with


other staff and tripled their time spent in personal activities; clerks spent two thirds less time interacting with other staff and almost doubled their time registering patients.

Karolinska Institute [10a] Iran

Random selection of records

The EMR had higher overall completeness than the paper system. High workloads, shortage of bedside hardware and lack of software features were prominent influential factors in the quality of documentation.

SOURCE: Authors’ Analysis NOTES: Evaluations are in increasing order of strength with multiple evaluations of a single system placed together. References can be found in Appendix Exhibit 1a


APPENDIX Exhibit 3a Laboratory Information Management Systems (LIMS) and Pharmacy Information System Evaluations



Laboratory Information Management Systems (LIMS)

Sanjay Gandhi Post Graduate Institute of Medical Sciences [11a] India Descriptive

Cholera was isolated in 22.6% (7/31) of samples sent to a central laboratory. Information was relayed to hospital and health authorities, who took strict measures to improve hygiene at a festival. Subsequently, the number of diarrhea cases during festival decreased and an epidemic was averted.

Tesilab [12a] Mexico

Case-control study

Productivity indexes showed an increase by 41% in number of patients handled and 28% in number of tests processed.

Karadeniz Technical University, [13a] Turkey

Before-after

Turn around times for routine samples decreased from 1 to half day; number of samples processed increased a factor of 2; annual laboratory revenue increased 4 times, from 55,000 to 220,000 euro per month.

Pharmacy Information Systems

Universidade de São Paulo [14a] Brazil

Descriptive

In 28.2% of medication orders there was dubious or misleading information Advantages: ease of data access and ordering. Disadvantages: repetition of orders from previous days without a review and incorrectly typed information.

Hospital das Clínicas da Faculdade de Medicina de Ribeirão Preto [15a] Brazil

Staff survey

Advantages: user-friendly interface; quickness and clarity of information; ease of use; reduction of time between drug prescription and administration; believed to result in a drastic reduction in the risk of error. Disadvantages: insufficient number of terminals; system got stuck; technical support was unsatisfactory.

University of São Paulo [16a] Brazil

Staff survey

Advantages: legibility (37.5%); less time to order (20.5%); more practical and organized (8%). Disadvantages: repetition of previous prescriptions (34%); typing mistakes (17%); dependence on computers (11%); alterations made manually (7%)

National Healthcare Group [17a] Singapore

Staff survey

Over 70% of users preferred system over paper, felt that it reduced the number of prescription errors, and knew what to do when system was down. Its limitations were with system speed and functionality in processing prescriptions. Satisfaction was more associated with perceived impact on productivity than with patient care.

Ekbatan Hospital [18a] Iran

Staff interviews

Clinician users of the prescribing system were found to mostly rely on their memories


and be overconfident which could lead to errors. Advantages: increased confidentiality, reduction of medication errors and educational benefits. Disadvantages: high cost, social and cultural barriers, data entry time and problems with technical support.

Hamadan University of Medical Sciences [19a] Iran

Before-after

Before intervention (Period 1), error rate was 53%, which did not significantly change after the implementation of CPOE without decision support (Period 2). However, errors were significantly reduced to 34% after the decision support was added to the CPOE (Period 3).

Socios En Salud [20a] Peru

Model vs. actual use

Accuracy of prediction per medication was 117% over-estimate in 2002, 5% underestimate in 2003 and to 2% under-estimate 2004.


Model, order

placed vs. actual use

For subgroup of 58 patients on individualized treatment, model predicted 99% of actual use, the actual order placed was 145% of actual use.


Externally controlled before-after

17.4% error rate fell significantly in the study group to 3.1% per patient. Error rate did not differ statistically in control group (8.6% to 6.9%).

SOURCE: Authors’ Analysis NOTES: Evaluations are in increasing order of strength with multiple evaluations of a single system placed together. References can be found in Appendix Exhibit 1a.


Appendix Exhibit 4a Patient Registration and Scheduling, Monitoring and Evaluation, and Clinical Decision Support System Evaluations



Patient Registration and Scheduling

Baobab Health[23a] Malawi

Clinical user survey

Most of the users (70%) expressed a clear preference for the touch screen over the paper system. However, every respondent also identified on-going problems that need to be addressed.

Sustainable Sciences Institute [24a] Nicaragua

Simultaneous randomized controls

Mean time to locate record with fingerprint scan was 7.0 (SD 3.5) seconds, versus 27.3 (SD 7.1) seconds using the traditional method.



Average time to locate a patient’s chart using traditional methods was 2.9 (SD 2.1) minutes, whereas using barcode-based methods the average was 0.09 minutes, or 5.5 (SD 1.2) seconds.

Monitoring, Evaluation, and Patient Tracking Systems

Calmette Hospital [25a] Cambodia Descriptive

Data are invaluable for the short-term management of the hospital. SIM helped set up infection control committee.

Tanzanian Ministry of Health [26a] Tanzania Cost

Total annual systems cost was US$2,119,941, $0.13 per participant, and $0.06 per capita.

HIV-EMR [27a] Haiti Case-control

study

For patients with CD4 counts between 101 and 350, those entered into the system within 14 days had an odds ratio of 3.2 for starting treatment within 14 days compared to those without early CD4 entry.

HIV-EMR2.0 (OpenMRS) [27a] Haiti

Case-control study

Logged patient follow-up visits allowed staff to rapidly identify a decline among patients who had stopped receiving food supplementation. New strategies were implemented within 3 weeks, and clinic attendance returned to original level of over 90%.

University of the Witwatersrand [28a]

South Africa


Time taken to locate ten households was reduced by 20% and 50% in each of two communities using the PDA/GPS device compared to paper.



GIS did not significantly decrease the time necessary to locate a home.

Clinical Decision Support System (CDSS) Chulalongkorn University [29a] Thailand

Before-after qualitative

Nurses perceived they had better judgment and information access, all participants wanted permanent installation.

Chulalongkorn University [29a] Thailand

Before-after quantitative

Mean judgment performance score for case simulations increased by 42%.


Electronic Integrated Management of Childhood Illness (e-IMCI) [30a] Tanzania

Simultaneous nonrandomized controls

84.7% of e-IMCI investigations had IMCI completed compared to 61% with the chart booklet. Amount of time for both IMCI and e-IMCI sessions averaged 12.5 minutes for the one clinician tested.

Early Diagnosis and Prevention System (EDPS) [31a] India

Longitudinal RCT

Increase of 430 new patient visits per month at intervention sites, increase from baseline of 18% at intervention sites compared with decline of 5% at control sites. Intervention was associated with significant improvements in Global Patient Assessment of Care Index.



Appendix Exhibit 5a Patient Reminder and Research/Data Collection Systems Evaluations

System or Institution

Country Evaluation Type Outcome Patient Reminder Systems

On Cue Compliance [32a]

South Africa Cost

Cost of 120 SMS reminders were R13.90/patient/month (US$2.43).

On Cue Compliance [32a]

South Africa

Before-after

Intervention had higher completion rate (10.6 vs. 3%), but similar cure rate (62.3 vs. 66.4%) and treatment success rate (73 vs. 69%) compared to data from City of Cape Town's TB Control Program for same clinic in 2003.

International Medical University Puchong [33a] Malaysia

Cost-effectivene

ss

It cost RM 0.45 per attendance for text messaging reminder as compared with RM 0.82 per attendance for mobile phone reminder. The ratio of cost per unit attendance of text messaging versus mobile phone was 0.55.

International Medical University Puchong [33a] Malaysia

Simultaneous

randomized controls

Attendance rates of control, text messaging and mobile phone reminder groups were 48.1, 59.0 and 59.6%, respectively. The text messaging group was significantly higher than control group, no difference between text messaging and mobile phone group. Text messaging reminder system cost less than half of the mobile phone reminder per attendance.

Research/Data Collection Systems

Ifakara Health Research & Development Centre [34a] Tanzania Descriptive

There were no problems with the PDAs while collected data on 83,346 individuals over seven weeks. Dataset was available within 24 hours. Median time to form completion was 14 minutes during training and nine minutes during survey.

Uganda Health Information Network [35a, 36a] Uganda User survey

87% reported that health content received helped them make faster more accurate diagnoses. 86% integrated PDA into other activities. 73% able to solve problems; 68% reported problems with 41% of them being resolved due to lack of technical support.

Uganda Health Information Network [35a, 36a] Uganda

Cost analysis

System provides up to 91% saving per unit spending compared to paper-based HMIS data collection and reporting approaches. Reporting compliance to MOH improved from national average of 63% to 94-100% for districts using UHIN.

UN-Vodafone Partnership [37a]

Multiple countries User survey

Advantages: time savings (95 percent); the ability to quickly mobilize or organize individuals (91 percent); reaches audiences previously difficult or impossible to reach (74 percent); transmit data more quickly and accurately (67 percent); gather data more quickly and accurately (59 percent).

Albert Gabon Self- Rate of discrepant entries was 1.7%.


Schweitzer Hospital [38a]

controls Categorical data were more commonly discrepant than were continuous “typed in” data (2.4% versus 1.2.


Self-controls

In 558 patient interviews accuracy of PDA and paper methods was 97.1% and 97.6%, respectively. For 1,543 field visits, accuracy rate for PDA and paper methods was 98.9% and 99.3%, respectively.

Mexican National Institute of Public Health [39a] Mexico

Focus groups

A majority of adolescents preferred the computer-assisted system to interviews.

Mexican National Institute of Public Health [39a] Mexico

Self-controls

Adolescents were 2-8 times more likely to report smoking, alcohol, and sexual behavior using the audio computer-assisted self-interview system compared to face-to-face interviews.

Universidad Peruana Cayetano Heredia [40a] Peru

Before-after (first

survey), RCT (second

survey)

First survey, almost perfect agreement between paper and PDA. Second survey, rates of responses to sensitive questions were similar between both kinds of questionnaires. PDA had 96% less inconsistencies and 66% less missing values than paper.

Charles R. Drew University of Medicine and Science [41a] Angola Block RCT

There was no difference between participants’ self-reported comfort across handheld and paper conditions. However, participants in the handheld condition were more likely to give socially desirable responses to the sexual behavior questions .

South African Medical Research Council [42a]

South Africa

Cost analysis

Cost of PDA survey is slightly less than paper when cost of hardware is annualized over four studies and the programming cost excluded. When programming cost is included, upfront costs need to be discounted over eight studies to obtain a comparative cost with paper.


South Africa

User surveys

85% of PDA users preferred PDA and 7% preferred paper for answering questions about sex. 53% of paper users preferred PDA and 28% preferred paper.


South Africa

Simultaneous

randomized controls

Comparing the data collected by the paper and PDAs using intra-scale and the test-retest reliability found them to be similar.

Socios En Salud[43a] Peru User Survey

User satisfaction higher for PDA (mean 5 of 5) than paper (3.5 of 5). PDA reduced mean work-time per result from 6.75 to less than 2 minutes. Mean 1.13 technical problems per month which could be fixed in the field (2 users) or back at the office (2 users).

Socios En Peru Cost Work hours required decreased by 60%. Total


Salud [43a] analysis cost and time to develop and implement was US$26,092 and 22 weeks. Cost to expand to 9 districts was $1,125 and to implement collecting patient weights $4,107.

Socios En Salud [44a] Peru Cluster RCT

PDA-based system decreased mean processing time from 23 to 8 days, and results with times greater than 90 days from 9.2% to 0.1%. It reduced errors by 57.1%.

London School of Economics [45a] Gambia

Cross-over simultaneou

s randomized controls

A custom-built PDA system showed a 30% improvement for collection of identification data and a 100% improvement for dates and times [system automatically time stamped]. Significant reduction in inter-individual variability in data accuracy. By the third week the average interview times were 31% shorter for field workers who used handheld.



e-health technologies show promise in developing countries

Health & Medicine