e-health evidence and evalaution

eHealth Summer Semester:International perspectives

in eHealth

Peter L. Reichertz Institute for Medical InformaticsUniversity of Braunschweig - Institute of Technology and

Hannover Medical School

Evaluation of and evidence for

eHealthNajeeb Al-Shorbaji,

Vice-President, e-MarefaDirector of Knowledge, Ethics and Research, WHO/HQ

(Retired)

Evidence on eHealth InterventionsEvidence on eHealth Interventions

What is eHealth?

•eHealth as the cost-effective and secure use of information and communication technologies in support of health and health-related fields, including health-care services, health surveillance, health literature, and health education.


Which eHealth application?• Access to health information on the Internet• Electronic health, medical or personal record• Telemedicine, tele-health, tele-….;• Disease surveillance, registry, monitoring, etc;• Mobile based applications;• eLearning;• …..


What is evidence?

• Information that is collected in an orderly way about a disease or its treatment. This information often comes from research. Evidence helps doctors and scientists understand what treatments work best on different diseases. ww.cdc.gov/cancer/lung/glossary.htm

http://www.cdc.gov/cancer/lung/glossary.htm


Evidence-Based Medicine

• Evidence-Based Medicine is the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients. The practice of evidence-based medicine means integrating individual clinical expertise with the best available external clinical evidence from systematic research.www.mcg.edu/som/fmfacdev/fd_ebmconcepts.htm

http://www.mcg.edu/som/fmfacdev/fd_ebmconcepts.htm


Does EBM apply to eHealth?

‘‘health information systems should be evaluated with the same rigor as a new drug or treatment program, otherwise decisions about future deployments of ICT in the health sector may be determined by social, economic, and/or political circumstances, rather than by robust scientific evidence’’Catwell L, Sheikh A (2009) Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Med 6: e1000126. doi:10.1371/journal.pmed.1000126.


Is it a social science?• The Summary Care Record (SCR) and HealthSpace programmes

spanned a number of different ‘worlds’ – political, clinical, technical, commercial, academic – with different institutional logics, as well as the personal world of the patient.

• Differences in norms, values, priorities and ways of working between these six worlds, and imperfect attempts to bridge these differences, accounted for much of the instability in the socio-technical network – and this in turn explained many of the challenges and frictions encountered as the complex collaborative tasks of design, implementation, governance, front-line use and evaluation were pursued.

Greenhalgh T, J, et al. (2010) The devil’s in the detail: final report of the independent evaluation of the Summary Care Record and Health Space programmes. London: University College London. (www.ucl.ac.uk/news/scriesummary.pdf)

http://www.ucl.ac.uk/news/scriesummary.pdf








Potential and promise of eHealth

• eHealth Reduces cost and increases efficiency;• eHealth Improves quality of service and patient

safety;• eHealth Increases equity of access to health

information and services;• eHealth Empowers individuals and enhances

accountability and transparency.


Summary Care Record (England):as an example

Policy documents published in 2005-8 anticipated a number of benefits of the SCR, including:

1.Safer care;2.More efficient care;3.Better care;4.More equitable care;5.Reduction in onward referral;6.Greater patient satisfaction.

http://www.connectingforhealth.nhs.uk/systemsandservices/scr

http://www.connectingforhealth.nhs.uk/systemsandservices/scr


• Technical and operational aspects of the programme could not be meaningfully isolated from subjective and contextual issues such as what the introduction of the SCR meant politically, professionally, practically, and personally.

T. Greenhalgh. Adoption and non-adoption of a shared electronic summary record in England: a mixed-method case study. BMJ 2010;340:c3111 (http://www.bmj.com/content/340/bmj.c3111.full)

Technology is only part of eHealth

http://www.bmj.com/content/340/bmj.c3111.full


Investment based on potential

• 2010-The Massachusetts Technology Collaborative (MTC) and the New England Healthcare Institute (NEHI) released a report on December 1 that shows tele-ICU technology could save 350 additional lives, benefit hospitals financially, and save more than $122 million annually if broadly and effectively implemented across Massachusetts (http://www.masstech.org/ehealth/index.html)

http://www.masstech.org/ehealth/index.html


Real money: big investments• England National Programme for Information

Technology, US$ 20.6 billion (2003-2011);• US Health Information Technology for Economic and

Clinical Health Act, US$ 20 billion (2011-2015);• Australia: design, building and national rollout of a

personally controlled electronic health record US$ 473 million (2010-2011);

• Canada Health Infoway implementation program has allocated US$ 2.16 billion (by 2010).


Why eHealth research?

• Build the knowledge base for eHealth;• Establish evidence that aims and objectives of a an

intervention have been achieved, not achieved or partially achieved;

• Introduce changes/improvements on the intervention based on the available evidence;

• Establish value chain, cost-benefit, and return on investment to support policy & decision making;

• Scale up, replicate, learn, improve in other settings.

Examples of evidence


eHealth IMPACT (eHI) study “eHealth is Worth It”

• The qualitative benefits were better informed patients and carers, better focused information allowing for more streamlined procedures, improved timeliness, improvements in safety, better effectiveness, improved access to information and greater efficiency. Timeliness, effectiveness and efficiency were prevalent at all ten sites.

• For the economic benefit, when all ten cases are presented as one virtual health economy, the results are economically persuasive.

• ICT alone will never bring about change without substantial changes to clinical and working practices, and the findings of the eHI study highlight some important policy and management recommendations.

• Source: www.ehealth-impact.org

http://www.ehealth-impact.org/


UK: Telehealth can deliver

• The early indications show that if used correctly telehealth can deliver a 15% reduction in A&E (Accident & Emergency ) visits, a 20% reduction in emergency admissions, a 14% reduction in elective admissions, a 14% reduction in bed days and an 8% reduction in tariff costs. More strikingly they also demonstrate a 45% reduction in mortality rates.

From: Whole system demonstrator programme: Headline findings - December 2011 http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_131684

http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_131684

http://www.dh.gov.uk/en/Publicationsandstatistics/Publications/PublicationsPolicyAndGuidance/DH_131684


eIMCI Implementation

• Many of the limitations to proper implementation of IMCI can be addressed by electronic support tools. Indeed, the results from a quantitative and qualitative study of eIMCI show the impact of electronic support tools on the adherence to clinical protocols, consistency in care across providers, and patient perceptions on quality of care.

H. Bethany. Evidence of eHealth Impact: The case for eIMCI in Tanzania (http://www.dbmi.pitt.edu/seminar/evidence-ehealth-impact-case-eimci-tanzania)

http://www.dbmi.pitt.edu/seminar/evidence-ehealth-impact-case-eimci-tanzania

World Health Bulletin Theme Issue on eHealth: building the evidence base• Call for papers in November

2011;• Published on 1 May 2012;• Received over 90 submissions;• Published 14 studies articles;• Rigorous peer review.


Highlights from the "Theme Issue"• eHealth development must be holistic, evidence based, and

people centered;• Improved collaboration between telemedicine networks could

help attenuate the lack of resources…and improve sustainability;

• Literature review shows that eHealth systems of three types: systems facilitating clinical practices, institutional systems and systems facilitating care at distance;


Highlights from the "Theme Issue"

• Mobile health can enable behavior change and improve health outcomes in resource-limited settings;

• Real challenge for the development for eHealth lies in establishing country level best practices that are both cost-effective and supported by rigorous research and evaluation;

• There is need for more research to continue to prove that eHealth can result in economic benefits and improves health outcomes.

Evaluation of eHealth projects

What is evaluation?

• An evaluation is a systematic and impartial assessment of an activity, project, programme, strategy, policy, topic, them, operational area or institutional performance. It focuses on expected and achieved accomplishments, examining the results chain, processes, contextual factors and causality, in order to understand achievements or the lack of.

Why evaluation?

• Catwell and Sheikh argue that “health information systems should be evaluated with the same rigor as a new drug or treatment program, otherwise decisions about future deployments of ICT in the health sector may be determined by social, economic, and/or political circumstances, rather than by robust scientific evidence”

[*]Catwell L, Sheikh A (2009) Evaluating eHealth interventions: the need for continuous systemic evaluation. PLoS Med 6: e1000126. doi:10.1371/journal.pmed.1000126.

http://dx.doi.org/10.1371/journal.pmed.1000126

Different dimensions of evaluation

•Qualitative or quantitative;•Patient or system;•Formative (of learning) or summative (for learning);

• Internal or external

What are the challenges?

• Greenhalgh at al believe that "eHealth “interventions” may lie in the technical and scientific world, but eHealth dreams, visions, policies, and programs have personal, social, political, and ideological components, and therefore typically prove fuzzy, slippery, and unstable when we seek to define and control them".

Greenhalgh T, Stramer K, Bratan T, Byrne E, Russell J, et al. (2010) The devil's in the detail: final report of the independent evaluation of the Summary Care Record and HealthSpace programmes. London: University College London.

Challenges

• The scope of eHealth interventions requires diversity in approaches as outcomes, impact, audience, cost and cost structure, and technology are different. One model cannot apply;

• Website for eLearning is different from a health information system or an electronic medical record systems

Challenges

• Multiple players and stakeholders in eHealth interventions require a multi-stakeholder approach to evaluation. These stakeholders pose different and multiple challenges;

• Stakeholder representation includes legislative (legal) issues, infrastructures issue, human resources issues, funding mechanisms, technology adaptation and training needs.

Challenges

• eHealth interventions start usually as demonstration and pilot projects. They are managed under a controlled environment with dedicated staff and well defined budget and outcomes. Scaling up becomes difficult (budget, legal, people, etc) and when a scaled up project needs to be evaluated the whole process changes and becomes less manageable.

• An mHealth pilot project is much easier to evaluate than a nation wide implementation of mHealth or electronic health records.

Challenges

• Cost-benefit analysis of human resources vs. cost-benefit analysis of technology. It is possible to calculate the cost of technology infrastructure and applications including hardware and software. It becomes more difficult to measure the cost of people's input to the eHealth due to the diversity of roles physicians and nurses using the system, ICT staff operating the system, patients benefiting from the system.

• eHealth "time" challenges. Technology based systems develop much faster than health systems. The pace of adoption of technology in health makes it difficult to match.

Challenges

• Monitoring and evaluation cost money. They need to be planned form the start and to be budgeted for and be performed by qualified personnel using the right approach;

Do we have indicators?

• The eHealth Benchmarking report (2009) identified 79 sources of indicators information, mainly surveys, which made an inventory of more than 4,400 eHealth-related indicators.

• The eHealth Benchmarking study developed a simple classification for data sources in the course of its information gathering work that groups sources according to four major purposes:

• Measuring of eHealth availability and use • Evaluation of eHealth applications • Measuring of attitudes towards eHealth • Assessment of eHealth markets

http://www.ehealth-benchmarking.eu/results/documents/eHealthBenchmarking_Final-Report_2009.pdf


An indicator framework for eHealth Benchmarking (EU Report, 2009)

1. eHealth Benchmarking indicators targeting citizens / patients;2. eHealth Benchmarking indicators targeting General Practitioners;3. eHealth Benchmarking indicators targeting specialists;

4. eHealth Benchmarking indicators hospital administrative / IT staff;5. eHealth Benchmarking indicators hospital medical staff;6. eHealth Benchmarking indicators targeting therapists; 7. eHealth Benchmarking indicators targeting pharmacies;

8. eHealth Benchmarking indicators targeting care providers.



Guiding principles for eHealth evaluation

• Greenhalgh and Russell provided an alternative set of guiding principles for eHealth evaluation. These principles need to they will need to be applied flexibly with attention to the particularities and contingencies of different contexts and settings. Each principle will be more or less relevant to a particular project, and their relative importance will differ in different evaluations.

[*] Greenhalgh T, Russell J (2010) Why Do Evaluations of eHealth Programs Fail? An Alternative Set of Guiding Principles. PLoS Med 7(11): e1000360. doi:10.1371/journal.pmed.1000360. http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1000360


• First, think about your own role in the evaluation.• Second, put in place a governance process (including a broad-based

advisory group with an independent chair) that formally recognizes that there are multiple stakeholders and that power is unevenly distributed between them.

• Third, provide the interpersonal and analytic space for effective dialogue (e.g., by offering to feed back anonymized data from one group of stakeholders to another).


• Fourth, take an emergent approach. An evaluation cannot be designed at the outset and pursued relentlessly to its conclusions; it must grow and adapt in response to findings and practical issues which arise in fieldwork

• Fifth, consider the dynamic macro-level context (economic, political, demographic, technological) in which the eHealth innovation is being introduced.

• Sixth, consider the different meso-level contexts (e.g., organisations, professional groups, networks), how action plays out in these settings (e.g., in terms of culture, strategic decisions, expectations of staff, incentives, rewards) and how this changes over time. Include reflections on the research process (e.g., gaining access) in this dataset .


• Seventh, consider the individuals (e.g., clinicians, managers, service users) through whom the eHealth innovation(s) will be adopted, deployed, and used. Explore their backgrounds, identities and capabilities; what the technology means to them and what they think will happen if and when they use it.

• Eighth, consider the eHealth technologies, the expectations and constraints inscribed in them (e.g., access controls, decision models) and how they “work” or not in particular conditions of use.

• Ninth, use narrative as an analytic tool and to synthesize findings.


eHealth evaluation ''gap'‘:Recap

• It is costly;• It is not an integral part of eHealth project management;• Fear of results (truth hurts);• Lack of evaluation expertise;• Lack of standard approaches or frameworks;• Diversified technologies, environments and applications;• Lack of standard indicators;• Low priority for policy and decision makers;• Conflict between internal and external evaluation.

Why do health ICT projects fail?

• Lack of proper needs assessment.• Lack of vision, strategy, and national plans.• Lack of information and awareness about ICT applications.• Computer illiteracy.• Insufficient resources to meet costs.• Limited experience in medical informatics.• Weak information and telecommunications infrastructures.• Absence of legislative, ethical, and constitutional frameworksWHO, 2004.


Comprehensive approach is needed

• …in light of the lack of evidence in relation to improvements in patient outcomes, as well as the lack of evidence on their cost-effectiveness, the authors say that future eHealth technologies should be evaluated against a comprehensive set of measures, ideally throughout all stages of the technology's life cycle, and include socio-technical factors to maximize the likelihood of successful implementation and adoption in a given context".

D. Ashly, et al. The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview. http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1000387

http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.1000387

Thank you Q & A