latin america and caribbean zika extension for community … · 2020-06-29 · latin america and...

Latin America and Caribbean Zika Extension for Community Healthcare Outcomes Evaluation

RESEARCH AND EVALUATION REPORT

JUNE 2020

This research and evaluation report on the Extension for Community Healthcare Outcomes (ECHO) program implemented with support from the American Academy of Pediatrics and the University of the Republic of Uruguay was prepared by University Research Co., LLC (URC) for review by the United States Agency for International Development (USAID) through the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project, which was made possible by the generous support of the American people through USAID. It was authored by the Monitoring and Evaluation Team of Project ECHO Uruguay.

RESEARCH AND EVALUATION REPORT

Latin America and Caribbean Zika Extension for Community Healthcare Outcomes Evaluation

JUNE 2020

Monitoring and Evaluation Team of Project ECHO Uruguay

Acknowledgements

The authors thank the Ministries of Health of Ecuador, El Salvador, Guatemala, Honduras, Nicaragua, Paraguay, Peru, and the Dominican Republic as well as all the providers who participated in this evaluation of the Zika Extension for Community Healthcare Outcomes (ECHO) program. Thanks are also due to Shannon Limjuco from the American Academy of Pediatrics for her contribution to the evaluation report.

This evaluation report was prepared by the Monitoring and Evaluation team of Project ECHO Uruguay for review by the United States Agency for International Development (USAID), the American Academy of Pediatrics (AAP), and University Research Co., LLC (URC) under the USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project., which is funded by the American people through USAID’s Bureau for Global Health, Office of Health Systems. The project was managed by URC under the terms of Cooperative Agreement Number AID-OAA-A-12-00101. URC's global partners for USAID ASSIST included the American Academy of Pediatrics; EnCompass LLC; FHI 360; Institute for Healthcare Improvement; and WI-HER, LLC.

For more information on the work of the USAID ASSIST Project, please visit www.urc-chs.com/assist or write [email protected].

Recommended citation

Monitoring and Evaluation Team of Project ECHO Uruguay. Latin America and Caribbean Zika Extension for Community Healthcare Outcomes Evaluation. 2020. Research and Evaluation Report. Published by the USAID ASSIST Project. Chevy Chase, MD: University Research Co., LLC (URC).

LAC Zika ECHO Program i

TABLE OF CONTENTS

List of Tables ................................................................................................................ i List of Figures ............................................................................................................... i Acronyms ....................................................................................................................iii

Executive Summary ....................................................................................................... iv I. Introduction ............................................................................................................. 1 II. Methodological Approach ........................................................................................ 3

a. Overview ......................................................................................................... 3 b. Data collection ................................................................................................. 3 c. Sampling ......................................................................................................... 4

III. Findings ................................................................................................................... 5 a. Description of the sample ................................................................................ 5 b. Changes in knowledge, skills and self-efficacy ............................................... 6 c. Satisfaction with the LAC Zika ECHO program ............................................. 19 d. Self-reported intention to change practices ................................................... 21 e. Focus group discussion findings ................................................................... 22 f. Identification of barriers to change practices ................................................. 24

IV. Limitations ............................................................................................................... 27 V. Conclusion ............................................................................................................... 27

List of Tables

Table 1. Number of ECHO sessions attended by survey respondents by country ....................... 5 Table 2. Comparison of knowledge, skills, and self-efficacy indicators before and after participation in the LAC Zika ECHO program ............................................................................... 6 Table 3. Program satisfaction indicators ..................................................................................... 20 Table 4. Participants’ self-reported changes following participation in the LAC Zika ECHO program ....................................................................................................................................... 21 Table 5. Additional barriers to change identified by participants in the LAC Zika ECHO program ....................................................................................................................................... 25 Table 6. Self-reported solutions to barriers identified by participants in the LAC Zika ECHO program ....................................................................................................................................... 26

List of Figures Figure 1. Self-reported mean change in knowledge, skills, and self-efficacy before and after participation in LAC Zika ECHO program (n=57) .......................................................................... 7 Figure 2. Index of increase in self-reported capability after participation in LAC Zika ECHO program: knowledge, skills, self-efficacy (n=57) ........................................................................... 7 Figure 3: Percentage of participants who report capacity to provide primary care to children potentially exposed to Zika before and after participation in the LAC Zika ECHO program (Indicator 1) ................................................................................................................................... 8 Figure 4. Percentage of participants who report capacity to understand lab tests results for Zika infection before and after participation in the LAC Zika ECHO program (Indicator 2) ................... 9 Figure 5. Percentage of participants who report capacity to order neurological exams for children potentially affected by Zika before and after participation in the LAC Zika ECHO

ii LAC Zika ECHO Evaluation

program (Indicator 3) .................................................................................................................... 9 Figure 6. Percentage of participants who report capacity to order visual exams before and after participation in the LAC Zika ECHO program (Indicator 4) ......................................................... 10 Figure 7. Percentage of participants who report capacity to identify developmental delays before and after participation in the LAC Zika ECHO program (Indicator 5) .......................................... 10 Figure 8. Percentage of participants who report capacity to know when to refer to a specialist before and after participation in the LAC Zika ECHO program (Indicator 6) ............................... 11 Figure 9. Percentage of participants who report capacity to evaluate pathologies related to Zika before and after participation in the LAC Zika ECHO program (Indicator 7) ............................... 11 Figure 10. Percentage of participants who report capacity to clinically manage associated pathologies before and after participation in the LAC Zika ECHO program (Indicator 8)............ 12 Figure 11. Percentage of participants who report capacity to communicate with families about comorbidities associated with Congenital Syndrome associated with Zika before and after participation in the LAC Zika ECHO program (Indicator 9) ......................................................... 12 Figure 12. Percentage of participants who report capacity to train health care team on Zika before and after participation in the LAC Zika ECHO program (Indicator 10) ............................. 13 Figure 13. Percentage of participants who report capacity to share knowledge about Zika with co-workers and professional peers before and after participation in the LAC Zika ECHO program (Indicator 11) ................................................................................................................ 13 Figure 14. Self-reported changes in capacity levels for Indicator 1 before and after participation in the LAC Zika ECHO program .................................................................................................. 14 Figure 15. Self-reported changes in capacity levels for Indicator 2 before and after participation in the LAC Zika ECHO program .................................................................................................. 15 Figure 16. Self-reported changes in capacity levels for Indicator 3 before and after participation in the LAC Zika ECHO program .................................................................................................. 15 Figure 17. Self-reported changes in capacity levels for Indicator 4 before and after participation in the LAC Zika ECHO program .................................................................................................. 16 Figure 18. Self-reported changes in capacity levels for Indicator 5 before and after participation in the LAC Zika ECHO program .................................................................................................. 16 Figure 19. Self-reported changes in capacity levels for Indicator 6 before and after participation in the LAC Zika ECHO program .................................................................................................. 17 Figure 20. Self-reported changes in capacity levels for Indicator 7 before and after participation in the LAC Zika ECHO program .................................................................................................. 17 Figure 21. Self-reported changes in capacity levels for Indicator 8 before and after participation in the LAC Zika ECHO program .................................................................................................. 18 Figure 22. Self-reported changes in capacity levels for Indicator 9 before and after participation in the LAC Zika ECHO program .................................................................................................. 18 Figure 23. Self-reported changes in capacity levels for Indicator 10 before and after participation in the LAC Zika ECHO program .................................................................................................. 19 Figure 24. Self-reported changes in capacity levels for Indicator 11 before and after participation in the LAC Zika ECHO program .................................................................................................. 19 Figure 25. LAC Zika ECHO program satisfaction indicators ....................................................... 20 Figure 26: Percentage of participants who reported that they intend to change practices following participation in the Zika ECHO program ...................................................................... 21 Figure 27: Barriers to practice changes identified by participants in the LAC Zika ECHO program ....................................................................................................................................... 24

LAC Zika ECHO Program iii

Acronyms

AAP American Academy of Pediatrics

ASSIST USAID Applying Science to Strengthen and Improve Systems Project

CSaZ Congenital Syndrome associated with Zika virus

ECHO Extension for Community Healthcare Outcomes

LAC Latin America and Caribbean

USAID United States Agency for International Development

iv LAC Zika ECHO Evaluation

Executive Summary

The Extension for Community Healthcare Outcomes (ECHO) Project relies on an interdisciplinary strategy in which a group of specialists addresses a health issue through a short didactic lecture followed by discussion of clinical cases presented by physicians and health care teams. Using Zoom to facilitate video and audio connection among participants, the technology and the case-based presentation format allows local medical teams to collaborate with specialists through “learning loops” which involve discussion and follow-up of cases.

The USAID Applying Science to Strengthen and Improve Systems (ASSIST) Project created a Zika ECHO program in Latin America and the Caribbean (LAC) to connect a team of experts with local providers from eight countries who provide care for babies and children affected by the Zika virus. The objective of the program was to increase participants’ knowledge, skills, and self-efficacy to improve care and health outcomes for children and families affected by the Zika virus infection. The American Academy of Pediatrics (AAP) worked closely with the ASSIST LAC ECHO regional team to identify the infrastructure needed to effectively coordinate the ASSIST LAC Zika ECHO program. A total of 15 ASSIST LAC Zika ECHO sessions were held between September 2018 and July 2019. ECHO sessions lasted 1.5 hours and included a brief didactic lecture followed by a presentation and discussion of a de-identified patient case.

AAP commissioned an evaluation of LAC Zika ECHO participants’ self-reported competencies and skills, satisfaction with the LAC Zika ECHO program, and changes in practice following participation in the program. The evaluation was conducted in Spanish by AAP’s partner Project ECHO Uruguay. The evaluation also sought to identify barriers to changing practices and assess providers’ perception of ways to overcome these barriers. Data was collected through a retrospective survey and a focus group discussion.

Participants reported improved knowledge and self-efficacy in key aspects of care including screening for Zika and comorbidities faced by infants and children potentially exposed to Zika as well as high satisfaction with the ASSIST LAC Zika ECHO program. Participants also reported sharing knowledge with coworkers and serving as local resources regarding care of infants and children with confirmed or suspected Zika exposure. Focus group participants described improved outcomes in terms of patient care and provision of services as well as practice- and system-level changes. Participants’ feedback regarding the program include the desire for the program to continue and expand to include other health topics. Specific suggestions include continuing patient case presentations and/or presenting updates on cases previously presented and incorporating the Zika ECHO program into undergraduate and post-graduate student curricula. Many participants noted the lack of specialists in different localities and the potential ECHO offers to create and strengthen health teams.

Overall, the ASSIST LAC Zika ECHO evaluation revealed positive outcomes for ECHO participants, their patients, and the health system in which they care for children, youth, and families. ECHO participants have become an asset that should be leveraged for future knowledge sharing in the LAC region.

LAC Zika ECHO Evaluation 1

I. Introduction

Project ECHO was created in 2003 by the University of New Mexico in the United States in order to improve access to Hepatitis C virus early detection and treatment in a context where thousands of infected individuals were not treated due to lack of specialists, especially in rural areas. Through sessions leveraging video conference technology, Project ECHO trained 3000 doctors, nurses, and community health workers to treat Hepatitis C in their own communities. Research conducted in 2011 showed that the quality of care provided by clinicians trained by Project ECHO was equivalent to that provided by university-trained specialists.1 The success of Project ECHO has led to the ECHO model being used for over 60 pathologies or clinical conditions by over 170 universities or specialized centers in 37 countries.

The ECHO model uses an interdisciplinary strategy in which a group of specialists addresses a health issue through a short didactic lecture followed by the analysis of clinical cases presented by physicians and health care teams. Through this methodology, the participation of health care teams is encouraged in an iterative exchange process. The ECHO sessions serve as “knowledge networks” which bring together specialists, experts, and interdisciplinary teams of primary care providers with participants at different training and specialization levels (doctors, nurses, nutritionists, psychologists, social workers, etc.) and allow guided feedback from specialists and iterative learning with the opportunity to discuss a patient case multiple times during the course of the patient’s care. Providers are therefore able to operate with greater independence as skills and self-efficacy increase. The technology and the case-based presentation format allow local medical teams to collaborate with specialists through “learning loops” which involve guided feedback from specialists and iterative learning with the opportunity for discussion and follow-up of cases.

Under the ECHO model, local health care teams learn from three main sources:

Discussion of patients among specialists offers case-based learning and an opportunity to develop both content knowledge and self-efficacy in practice. The model is based on guided feedback from specialists and iterative learning.

The interaction with other health care providers on equal terms allows for learning through others’ experiences. This generates a ‘community of practice’ that allows a horizontal transfer of knowledge among different localities.

Content knowledge is supported through short educational presentations. The presentations are conducted by ECHO-trained faculty.

The ECHO model is based in educational theories about learning and behavioral changes including: (I) Bandura Social Cognitive Theory2, Vygotsky Situated Learning Theory3, and Communities of Practice.4 Social Cognitive Theory identifies three influencing factors affecting individuals’ belief in their own capacity to act and perform behaviors that would lead to a

1 Sanjevv A, Karla T, Glen M, et al. Outcomes of treatment for Hepatitis C virus infection by primary care providers. New England Journal of Medicine. 2011; 364:2199-2207. 2 Bandura A. The evolution of social cognitive theory. In K. G. Smith & M. A. Hitt (Eds.) Great Minds in Management. 2005; 9-35. Oxford: Oxford University Press. 3 Vygotsky L. Mind in Society: Development of Higher Psychological Processes. 1978. Cambridge MA: Harvard University Press. 4 Lave J, Wenger, E. Situated Learning: Legitimate Peripheral Participation. 1991. Cambridge: Cambridge University Press.

2 LAC Zika ECHO Evaluation

desired outcome: a) knowledge of learning benefits over its costs; b) self-sufficiency; and c) reinforcement of positive changes by significant others. ECHO sessions are also based on the situated learning theory which emphasizes that learning requires social interaction and collaboration.

Building from the Project ECHO model, the ASSIST LAC Zika ECHO program was created and integrated into the larger USAID ASSIST Project Zika program focused on improving the quality of prevention, care, and psycho-emotional support to families affected by the Zika virus. The objectives of the overall ASSIST Zika program in the LAC region were the following:

Increase the number and proportion of babies affected by Congenital Syndrome associated with Zika virus (CSaZ) receiving the recommended care.

Increase the provision of quality psycho-emotional support services for women and families affected by Zika.

Increase the clinical detection of Congenital Syndrome associated with Zika in newborns and the implementation of the recommended initial actions.

Increase the clinical detection of Zika virus infection in prenatal care, as well as knowledge of routes of transmission, risks, and personal and domestic measures to prevent vector transmission and the use of a condom as a preventive measure against sexual transmission among pregnant women in prenatal care services.

Special attention was paid to prevention as both health professionals and mothers can detect early risk factors that allow adequate and timely clinical care. Emphasis was also placed on the relationship between the providers and the families of the children served, since they are an important part of the therapeutic process. One of the ways in which these objectives was achieved was through the participation of health care providers in the ASSIST LAC Zika ECHO program.

The ASSIST LAC Zika ECHO program connected a team of Spanish-speaking experts from the American Academy of Pediatrics (AAP) with local health care providers from eight countries who provide care for babies and children affected by Zika virus (Dominican Republic, Ecuador, El Salvador, Guatemala, Honduras, Nicaragua, Paraguay, and Peru). Through regular virtual meetings, the AAP experts and other ASSIST colleagues aimed to increase participants’ knowledge, skills, and self-efficacy to provide the best care for children and families affected by the Zika virus infection, with the goal of improving health outcomes for infants and children.

The AAP worked closely with the ASSIST LAC Zika ECHO regional team to identify the infrastructure needed to effectively coordinate the ASSIST LAC Zika ECHO program. This included training the ASSIST team on the ECHO methodology, identifying and ensuring that Spanish-speaking faculty supported the core ASSIST LAC Zika ECHO team, supporting the design of patient case forms, supporting the development and implementation of the ASSIST LAC Zika ECHO program, and providing technical assistance and feedback during the implementation of the ECHO program. The AAP identified and recruited three experts to serve as faculty for the ASSIST LAC Zika ECHO program. The role of the faculty was to contribute to curriculum planning discussions, create and deliver the content of ECHO sessions, review de-identified patient cases, and facilitate case discussion and recommendations during ECHO sessions.


A total of 15 ASSIST LAC Zika ECHO sessions were held between September 2018 and July 2019. Each ECHO session lasted 1.5 hours and included a brief didactic lecture followed by a presentation and discussion of a de-identified patient case.

The objective of this evaluation was to assess self-reported competencies and skills, satisfaction with the ASSIST LAC Zika ECHO program, and changes in practice following participation in the program. The evaluation also sought to identify barriers to changing practices and assess providers’ perception of ways to overcome these barriers.

II. Methodological Approach

a. Overview

The ECHO model uses evaluation tools validated by the ECHO Institute in New Mexico for the short-term evaluation of its hepatitis ECHO program.5 The tools were adapted to account for the specificity of each ECHO session, local terminology,6 and to incorporate input from the AAP technical team who had previously adapted the tool for an AAP Zika ECHO program.

It was not possible to obtain baseline and follow-up measures within this program’s timeframe. Therefore, a retrospective self-perception web-based survey was administered followed by a focus group discussion.

b. Data collection

Retrospective web-based survey

A web-based survey was sent to all 522 clinical staff participating in the ASSIST LAC Zika ECHO. The survey collected information regarding the following key variables:

Knowledge, skills, and self-efficacy as an index measure. These indicators were assessed using a Likert scale of five values before and after starting the ASSIST LAC Zika ECHO program. The scale items/indicators below were measured:

1. Capacity to provide primary care 2. Capacity to understand lab tests results 3. Capacity to order neurological exams 4. Capacity to order vision exams 5. Capacity to identify developmental issues 6. Capacity to know when referral to a specialist is necessary 7. Capacity to evaluate the existence of related pathologies 8. Capacity to clinically manage related pathologies 9. Capacity to communicate with families about comorbidities 10. Capacity to train health care team on Zika 11. Capacity to share knowledge about Zika with coworkers and other professionals

5 Arora S, Kalishman S, Thornton K, Dion D, Murata G, Deming P et al. 2010. Expanding Access to HCV Treatment - Extension for Community Healthcare Outcomes (ECHO) Project: Disruptive Innovation in Specialty Care. Hepatology (Baltimore, Md.), 52(3), 1124–1133. http://doi.org/10.1002/hep.23802 6 Giachetto G, Casuriaga AL, Santoro A, Kanopa A, Garrido G, Fernández J, Cohen H, Sohl K. Extension for Community Healthcare Outcomes Uruguay: A New Strategy to Promote Best Primary Care Practice for Autism. Global Pediatric Health, (January 2019). doi:10.1177/2333794X19833734.


Personal satisfaction with the ASSIST LAC Zika ECHO program as an index measure. Satisfaction with training and mentoring activities was measured using six items/indicators:

1. Use of time (assessment of the usefulness of the time dedicated to the ECHO program)

2. Professional networking (perception of joining the community of practice) 3. Improvement in the quality of work 4. Balance between instruction and practice 5. Zika knowledge 6. Overall satisfaction

Self-reported intention to change practices due to participation in the ASSIST LAC

Zika ECHO program. This variable included three indicators: 1. Changes in the use of clinical guides. 2. Changes in professional practice. 3. Changes in policies or procedures.

Self-reported barriers to changes in practice. This variable included the following

indicators: 1. Insufficient knowledge. 2. Insufficient abilities. 3. Lack of support from workmates. 4. Lack of support from management.

Self-reported solutions to overcome barriers (open-ended questions)

Focus Group

A focus group was conducted after the completion of the survey to better understand the impact of the LAC Zika ECHO program from the participants' perspective and further explore responses from the survey. Information was collected on successes, challenges, and how learning translated into better clinical care. Participants were informed about the focus group activity through a brief description of the purpose and logistics of the focus group. All potential attendees of the focus group were informed that participation was voluntary.

The focus group was implemented using the video conference Zoom platform and was coordinated by the Project ECHO Uruguay evaluation team. The ASSIST LAC Zika ECHO team sought the participation of at least one participant from each of the eight countries. The focus group discussion lasted an hour and forty-four minutes with 24 participants connected for the entire focus group discussion.

c. Sampling

The web-based survey was sent by email to all 522 registered participants of the ASSIST LAC Zika ECHO program. The survey was sent four successive times between June 11 and 26, 2019. There were 78 responses in total; 21 responses were removed from analysis due to refusal or non-eligibility (not meeting the minimum participation threshold for participation).


The initial plan was to include in the analysis respondents who participated in at least six out of 15 ECHO sessions. However, this would have resulted in a very small sample; hence the analysis included all respondents who attended at least three sessions. As a result, the analysis included responses from 57 respondents (see Table 1 below for breakdown by country).

III. Findings

a. Description of the sample

Table 1 shows the number of ECHO sessions attended by survey respondents by country. The maximum number of ECHO sessions attended by respondents was 13 out of 15 sessions. In terms of professional distribution of the sample, there was a strong participation of physicians and to a lesser extent nurses; other professionals were less represented in the sample. In terms of specialization, pediatrics was most frequently cited (25%) followed by neonatology (15%). Other indicated areas of specialization included epidemiology, public health, and obstetrics and gynecology. Gender distribution was relatively balanced with an eight-point difference in favor of women (58% women and 42% men).

The median age of participants was 45 years, and the average years of experience in health care was 20 years. The wide range of ages and years of experience may have implications with regard to shared learning among individuals from different generations and levels of experience.

Table 1. Number of ECHO sessions attended by survey respondents by country

Country

Number of ECHO sessions attended

Number of

respondents

3 4 5 6 8 9 10 11 12 13

Ecuador 1 4 0 1 0 0 1 1 1 1

El Salvador 1 5 1 1 2 0 2 1 1 0

Guatemala 0 0 1 0 1 0 0 1 4 2

Honduras 0 0 0 1 1 0 0 0 1 2

Nicaragua 1 0 1 0 0 0 0 3 1 0

Paraguay 0 0 0 0 0 1 0 0 0 0

Peru 1 1 2 0 1 0 1 0 0 0

Dominican

Republic

1

1

1

0

0

0

2

0

1

0 6

Total

5 11 6 3 5 1 6 6 9 5

8.8% 19.3% 10.5% 5.3% 8.8% 1.8% 10.5% 10.5% 15.8% 8.8% 100.0%

10

14

9

5

6

1

6

57


b. Changes in knowledge, skills and self-efficacy

Analysis of overall project impact

Regarding knowledge, skills, and self-efficacy, all 11 indicators increased following participation in the LAC Zika ECHO program (Table 2 and Figure 1). The four indicators theoretically linked to an increase in self-efficacy (1, 9, 10, and 11) show the highest increase. Paired t tests comparing values of the items before and after participation in the LAC Zika ECHO program were all significant (p value<0.001).

Table 2. Comparison of knowledge, skills, and self-efficacy indicators before and after participation in the LAC Zika ECHO program

Scale items/indicators Zika ECHO

Mean p value for

difference

1. Capacity to provide primary exposed to Zika

care to children potentially After 4.35 <0.001 Before 2.74

2. Capacity to understand lab tests results associated to with Zika infection

After 4.28 <0.001 Before 2.84

3. Capacity to order neurological exams for children potentially affected by Zika

After 3.93 <0.001 Before 2.67

4. Capacity to order visual exams to children with potentially affected by Zika

After 4.04 <0.001 Before 2.53

5. Capacity to identify developmental issues in children potentially affected by Zika

After 4.21 <0.001 Before 2.82

6. Capacity to know when children need referral to specialists After 4.53

<0.001 Before 2.93

7. Capacity to evaluate pathologies related to Zika After 4.14 <0.001 Before 2.7

8. Capacity to clinically manage pathologies related to Zika After 3.86 <0.001 Before 2.51

9. Capacity to communicate with families about Zika comorbidities. After 4.39 <0.001 Before 2.58

10. Capacity to train healthcare team about Zika. After 4.35

<0.001 Before 2.82

11. Capacity to share knowledge about Zika with workmates and other professionals.

After 4.42 <0.001 Before 2.77


Figure 1. Self-reported mean change in knowledge, skills, and self-efficacy before and after participation in the LAC Zika ECHO program (n=57)

4.534.35 4.28 4.39 4.35 4.42

3.93 4.04 4.21

3.86

2.7 2.7

Item 1 Item 2 Item 3 Item 4 Item 5 Item 6 Item 7 Item 8 Item 9 Item 10 Item 11

Before Zika ECHO After Zika ECHO

2.52.52.7

2.5 2.6

2.8 2.9

2.8

2.8

13 7

2 2 4 4 7 8

4.14

3

The Cronbach Alpha coefficient for the knowledge, skills, and self-efficacy scale before and after participation in the LAC Zika ECHO program was 0.974 and 0.934, respectively, suggesting that the scale items have high internal consistency. A score index was therefore obtained to show the average increase in self-reported knowledge, skills, and self-efficacy across all 11 items/indicators following participation in the program. The index score shows that 99% of respondents have increased their rating to at least “competent” and in most cases “above-average” or “expert” after participating in the LAC Zika ECHO program (Figure 2).

Figure 2. Index of increase in self-reported capability after participation in LAC Zika ECHO program: knowledge, skills, self-efficacy (n=57)

54.4%

36.8%

8.8%

Average relative to my peers Above Average Expert; teach others


Changes by indicator

This section shows the results of two types of analyses for each of the 11 indicators that constitute the knowledge, skills, and self-efficacy scale:

Percentage of respondents who report no capacity, limited capacity, average capacity, above average capacity, and expert capacity before and after participating in the LAC Zika ECHO program (Figures 3-13).

Percentage of respondents who report the following changes in self-reported skills following participation in the LAC Zika ECHO program: no changes, one level higher, two levels higher, three levels higher, or more than three levels higher (Figures 14- 24).

Figures 3-13

Figures 3-13 show that although there are variations across indicators, following participation in the LAC Zika ECHO program, the majority of participants reported either above average or expert level expertise for each of the 11 indicators. Before participation in the program, most patients reported either no capacity, limited, or average capacity.

Figure 3: Percentage of participants who report capacity to provide primary care to children potentially exposed to Zika before and after participation in the LAC Zika ECHO program (Indicator 1)

51

42

35

26

19

12

No capaciy Limited knowledge or

capacity

Average relative to my peers

Above average Expert; teach others



Figure 4. Percentage of participants who report capacity to understand lab tests results for Zika infection before and after participation in the LAC Zika ECHO program (Indicator 2)

Figure 5. Percentage of participants who report capacity to order neurological exams for children potentially affected by Zika before and after participation in the LAC Zika ECHO program (Indicator 3)

54

37

28 3023

12

No change Limited knowledge Average relative to Above average or capacity my peers

Expert; teach others


47

30

25 26

21 23

11 11

No change Limited knowledge Average relative to Above average

or capacity my peersExpert; teach

others



Figure 6. Percentage of participants who report capacity to order visual exams before and after participation in the LAC Zika ECHO program (Indicator 4)

Figure 7. Percentage of participants who report capacity to identify developmental delays before and after participation in the LAC Zika ECHO program (Indicator 5)

Capacity to order visual exams (%)

32 30 23

1923

11

No change Limited knowledge or

capacity

Average relative Above average to my peers



47

37

32 28

16 18

12 11





Figure 8. Percentage of participants who report capacity to know when to refer to a specialist before and after participation in the LAC Zika ECHO program (Indicator 6)

Figure 9. Percentage of participants who report capacity to evaluate pathologies related to Zika before and after participation in the LAC Zika ECHO program (Indicator 7)

56

4035

2823

No change

Limited knowledge Average relative to Above averageor capacity my peers



54

37

28 30

14 18 16

No change

Limited knowledge Average relative to

Above averageor capacity my peers




Figure 10. Percentage of participants who report capacity to clinically manage associated pathologies before and after participation in the LAC Zika ECHO program (Indicator 8)

Figure 11. Percentage of participants who report capacity to communicate with families about comorbidities associated with Congenital Syndrome associated with Zika before and after participation in the LAC Zika ECHO program (Indicator 9)

51

42

23 21 2116 14

No change Limited knowledge Average relative to Above average Expert; teach others or capacity my peers


49

40

33

26

19 19

11

No change

Limited knowledge Average relative to

Above averageor capacity my peers




Figure 12. Percentage of participants who report capacity to train health care team on Zika before and after participation in the LAC Zika ECHO program (Indicator 10)

Figure 13. Percentage of participants who report capacity to share knowledge about Zika with co-workers and professional peers before and after participation in the LAC Zika ECHO program (Indicator 11)

46 46

32

23 23

16

No change

Limited knowledge or

capacity

Average relative Above average Expert; teach to my peers others


51

40

30 26

16 18

11





Figures 14-24

Figures 14-24 show that following participation in the LAC Zika ECHO program, most participants improved their capacity by at least two levels (e.g., from limited knowledge to average to above average, or expert level). However, self-reported improvements varied by indicator. The percentage of participants whose self-reported capacity increased by one level ranged from 28%-51%; those who reported changes by two levels ranged from 19%-39%, while the percentage of participants who reported increases by three or more levels ranged from 12%-23%. The percentage of participants whose self-reported capacity did not change ranged from 12%-19%.

Figure 14. Self-reported changes in capacity levels for Indicator 1 before and after participation in the LAC Zika ECHO program

No change

Improved capacity by 1 level

Improved capacity by 2 levels

Improved capacity by 3 or more levels




No change




No change






.

No change





c. Satisfaction with the LAC Zika ECHO program

Participants’ satisfaction with the LAC Zika ECHO was assessed using six indicators (Figure 25). The majority of participants strongly agreed or agreed that the LAC ECHO program was a valuable use of their time (63% and 33%, respectively ), contributed to their professional work (63% and 30%, respectively), made them better at their job (61% and 28%, respectively), provided an appropriate balance between instruction and practice (54% and 40%, respectively), and improved their understanding of Zika (67% and 25%, respectively). Overall, 93% of participants reported being satisfied with the LAC Zika ECHO program.

No change





The Zika ECHO program made me better at my job (%)

61

The Zika ECHO program provided an appropriate balance between instruction and practice (%)

54 40

28

4 7

Strongly Neutral disagree

Agree Strongly agree

Strongly disagree

Neutral Agree Strongly agree

My understanding of Zikaimproved due to my

participatoin in the Zika ECHO program (%)

I am generally satisfied with my particiption in the Zika ECHO

program (%)

63

6730

4 5 25

Totally disagree

4

Neutral

4

Agree Strongly agreeStrongly disagreeNeutral Agree Strongly agree

The Zika ECHO program was a valuable use of my time (%)

63

33

4

The Zika ECHO program contributed to my professional network %

63

30

4 4

Strongly disagree

Agree Strongly Agree Strongly disagree

Neutral Agree Strongly agree

4 2

Figure 25. LAC Zika ECHO program satisfaction indicators

Table 3 shows means values of each satisfaction indicator. In order to measure overall personal satisfaction, a personal satisfaction index was obtained using the six satisfaction indicators (Cronbach alpha coefficient= 0.97). The resulting mean satisfaction index score indicated a high satisfaction level (4.61 out of 5).

Table 3. Program satisfaction indicators

Indicators Mean Standard Deviation

Valuable use of time. 4.53 0.826

Contributed to my personal network. 4.49 0.869

Made me better at my job. 4.44 0.907

Provided an appropriate balance between instruction and practice. 4.42 0.844

My participation in ECHO improved my understanding about Zika. 4.51 0.889

I am satisfied in general terms with my participation in the Zika ECHO program. 4.49 0.869


91

Clincal guidelines Professional Practice. Policies and/or procedures.

77

65

d. Self-reported intention to change practices

Participants’ self-reported intention to change practices following participation in the LAC Zika ECHO program was measured with three indicators/items (Figure 26). Findings show that the majority of providers reported that they intended to change practices related to clinical guidelines (65%), professional practice (91%), and policies and procedures (77%).

Figure 26: Percentage of participants who reported that they intend to change practices following participation in the Zika ECHO program

In addition, qualitative open-ended comments asking about intent to change were also obtainedthrough the retrospective survey. Table 4 summarizes responses to this question.

Table 4. Participants’ self-reported changes following participation in the LAC Zika ECHO program

Change in awareness of Zika virus in terms of how it affects Awareness children, families, and society, and how to provide the necessary and care and improve their quality of life motivation, Awareness related to Zika in my job self-efficacy and

Knowledge applies to pregnant women Promote all the people who participate in any way in these

empowerment trainings to continue and sustain improvements and to identify gaps that could be turned into strengths

Motivate hospital providers where no technical assistance or resources from ECHO teleclinics were provided to empower them on Zika issues


Involve learning institutions, including undergraduate and graduate levels

Share my knowledge with nurse professional training institutions Disseminate knowledge and promote the use of ECHO providing advice to use the platform and sharing the presentations Disseminate this knowledge among colleagues and students of medicine Institutionalize this knowledge in learning institutions

Concrete actions related to patients, families, and health system

Assessment of premature children with WHO charts especially designed for this type of children Promoting prevention and eradicating mosquitos Extend implementation of proceedings to actively seek and assess Zika cases and provide timely referral Train the couple in terms of gender, which shall result in the effective use of male condoms

I am interested in creating a neuropsychological assessment clinic and enhance the early stimulation area

Systems changes

Apply learnings to other health conditions Promote the use of teleclinics to authorities as a tool to continuously improve quality of care to children affected by Zika Replicate the content learned Teamwork in the different types of care is essential and should be constant

e. Focus group discussion findings

Participants’ perceptions of the ASSIST LAC Zika ECHO program were further explored during a focus group discussion. All participants reported they had never taken part in a similar program and that they quickly realized that the ASSIST LAC Zika ECHO was different from typical teleconference meetings or webinars.

Perceived strengths: The elements of the ASSIST LAC Zika ECHO highlighted as particularly relevant included:

The opportunity to interact with providers from other countries regarding a new/emerging disease for which there was previously little knowledge and experience.

The opportunity to participate in the program despite not having expertise or knowledge about the disease.

The opportunity to see and analyze patient cases from different points of view.

Some participants stated that they joined the program due to the enthusiasm of other participants. Participants also reported that the ASSIST LAC Zika ECHO sessions enabled them to learn from others’ experiences. This was considered by many as very relevant, because it is often difficult to see diverse cases, especially in small cities. Participants also reported that the program was accessible, and the videoconference component allowed them to share experiences more effectively. Participants also reported that the instruction approach was original with useful, clear, and updated technical material delivered by experienced lecturers.

The most concrete examples of the impact of the LAC Zika ECHO program is gaining the


confidence necessary to develop a new kind of relationship with families and children. With these changes in the attitudes and dynamics of teams, providers have reported that families have felt more supported and better served and are more receptive to providers’ suggestions.

Additional perceptions of the impact of the LAC Zika ECHO program include:

Equipment was bought to provide timely early infant stimulation based on the information in the Zika ECHO lectures.

Parents and teams showed commitment to patients. The combination of satisfied parents and committed teams creates a feedback loop. Parents realized that health care providers have more equipment, interest, and information while providers witness parents’ positive reactions, including parents coming from rural areas and committing to the care process.

Teams received tools that made them feel empowered.

Patients felt that their opinions and needs were taken into account. A mother’s committee was established, and parents no longer felt alone.

Staff felt more knowledgeable and more confident when interacting with families. Visit schedules changed from monthly to weekly or based on needs. There was also more awareness and sense of closeness between providers and families.

Mothers felt more supported in the process. They have been empowered to provide care to children including those with other disabilities.

Perceived challenges and recommendations for improvement: Although participants thought the opportunity to ask questions was useful, they reported that the “Question and Answers” component of the sessions was often too short. Participants suggested that one element worthy of continued discussion is “how to talk to caregivers”, especially regarding providers’ skills to deliver children’s test results to parents. Participants further reported that the lectures focused on clinical exams and more emphasis could have been placed on psycho-emotional support, nutrition, neurodevelopment, and language therapy. Additionally, participants would like to see lecture topics tied to the presented cases.

Regarding the case presentations, presenters noted a feeling of discomfort asking questions about clinical cases but felt that this was also empowering from the collective and individual point of view. Participants reported that gathering information for the cases and presenting the cases required collective hard work. For some teams, the greatest challenge was filling in the information for all the visits and care provided to children. Participants stated that it would be helpful to receive updates on the cases.

Participants considered both the lectures and patient case presentations portions of the ECHO sessions to be important but suggested additional time for both. In instances where it was possible to identify a patient case in the region for presentation, participants suggested they needed support to bring in an expert to present.

Participants identified flexibility as an important component of the ECHO sessions. For example, they enjoyed when case presentations were conducted via PowerPoint as a more visually stimulating option. Timing must also be flexible, as there was often more time needed to review concepts being presented. Issues regarding the scheduling of ECHO sessions and internet connection were also raised. Although overall, participants were satisfied with the Zoom platform, some participants suggested additional options (other than Zoom) to connect or having


sessions available on demand. One focus group participant suggested having more lecturers from LAC regions to enable more advice within the local context.

Finally, participants suggested it would be important to integrate ECHO sessions in their daily activities and would consider using this approach to manage other diseases, especially in areas where it is difficult to gain access to specialists. It was strongly suggested that ECHO be used in undergraduate and postgraduate courses for medical, nursing, and other health care professional students.

f. Identification of barriers to change practices

Participants were asked to identify barriers to change practices in the retrospective survey. The most relevant barriers were related to lack of support from co-workers (43.9% of participants) and management (31.6% of participants). In addition, barriers related to insufficient knowledge (10.5%) and skills (15.8%) were also identified (Figure 27). An open-ended question was also posed to assess individual perception of barriers. Responses are summarized in Table 5.

Figure 27: Barriers to practice changes identified by participants in the LAC Zika ECHO program

43.9%

31.6%

15.8%

10.5%

Insufficient knowledge

Insufficient skill set Lack of support from Lack of support fromco-workers management


Table 5. Additional barriers to change identified by participants in the ASSIST LAC Zika ECHO program

Barriers related to public policies and leadership support

Barriers linked to limited access as a result of not being part of the USAID ASSIST Project or of not being part of the project target

Barriers related lack of infrastructure, supplies, and human resources

Barriers related to teleclinic schedules and

Public policies related to arbovirus infection Limited support from authorities Lack of support from operational units Attitudinal issues; resistance to change

Lack of team training in facilities not involved in the USAID ASSIST Project Not all of providers have the same level of training Lack of updates through teleclinics Having a portal to access teleclinics while they are being conducted even if not part of the project Diminished interest in current Zika conditions. An expansion in the range of similar situations to those dealt with should beconsidered.

Lack of necessary medical equipment and difficult access to them from health units that are far away, for electrocardiograms or electromyography. Appropriate materials and supplies and labs Lack of specialists in hospitals Lack of specialist doctors to make referrals

Assess appropriate schedules to replicate knowledge based on needs

connectivity issues Lack of time Teleclinic schedules are at a critical time for a hospital

network It is difficult to allocate time to clinicians without affecting

other services

Professional availability and availability of equipment and internet network

In addition, participants were asked about solutions to address the barriers they identified. Table 6 summarizes solutions reported by participants to overcome barriers. Reported solutions include solutions to improve team and clinic dynamics as well as management.


Table 6. Self-reported solutions to barriers identified by participants in the ASSIST LAC Zika ECHO program

Solutions related to the team and clinics dynamics

Updates /working materials/laboratories/supplemental exams To keep myself updated and keep on learning about Zika virus disease, as it is a new disease and constantly changing. Some incentives to encourage participation. Expand teleclinics to more personnel in the facilities. Reinforcement to non-trained personnel. Communication and teamwork. Professional support and commitment (operational in the territory) Multidisciplinary health personnel involvement such as physiotherapists, psychology, nutrition, and dentistry.

Economic support and donations. Budget (appropriate and/or needed funding) Supplemental hours.

Logistic support.

Supply and stationery logistics. Provision of materials and supplies.

Having the Department of Health set the mechanism to facilitate the acquisition of a timely and rapid service.

Solutions Communication plan, training in the model, and support with basic equipment. related to Package of changes at the central level. political and institutional

Continuous education program for administrative and operational personnel.

planning and

Case studies and epidemiological profile assessment of cases and audits of care.

aspects Publication of psychosocial support policies. Political willingness and commitment of authorities.

Management commitment.

Coordination with authorities.

Greater involvement of authorities.

Awareness of teammates in different areas and having support of the authorities in facilities supported by the project.

Application of quality improvement within the facility.

Institutionalize knowledge and strengthen supervision/mechanisms to ensure compliance with standards.

Specific aspects

Need technological support. Need audiovisual supplies. Having instruments to measure appropriate head circumference in premature babies. PAHO guides printed for the health services in my area of influence.


IV. Limitations

Several limitations should be noted. Only 11% of registered participants responded to the survey email. This low response rate may lead to biased findings if participants who responded are different from those who did not in terms of characteristic that may affect answers to survey questions. Another limitation is the use of a retrospective survey to assess participants’ knowledge, skills, and self-efficacy as before and after their participation in the ASSIST LAC Zika ECHO sessions. Such surveys are subject to bias as participant’s perceptions of past knowledge, skills and self-efficacy may be affected by their current knowledge, skills, and self-efficacy. A prospective survey assessing knowledge and skills at two time periods-prior and following participation- in the ASSIST LAC Zika ECHO sessions would have produced more valid responses. In addition, while the virtual focus group discussion brought participants from eight countries, online focus group discussions require adjustments due not lack of non-verbal cues and interactions typically visible in face-to-face focus groups. The large number of participants and the relatively limited time for discussion may have also limited the depth of responses.

V. Conclusion

Key conclusions arising from the ASSIST LAC Zika ECHO program evaluation include the followin

There were important self-reported improvements in terms of knowledge, skills, and self- efficacy following participation in the ASSIST LAC Zika ECHO program.

On average, participants reported a high level of satisfaction with the program.

Self-reported intention to change practices was high, with the highest values linked to “professional practice” (91%) and “policies and procedures” (65%).

Noteworthy barriers to change practices identified include lack of institutional support, both on a political management level as well as in a lower level of management; lack of infrastructure, human resources, materials, and specialists; and elements linked to the internal management of the ASSIST LAC Zika ECHO sessions, which show the complexity of implementing this operation in eight countries simultaneously.

In terms of participants’ feedback regarding the program, the main suggestions included promoting the continuity of ASSIST LAC Zika ECHO sessions and the expanding the ECHO model to other conditions and topics. Participants considered the ECHO methodology as a relevant part of their work routine and did not wish for it to end. Specific suggestions included continuing patient case presentations and/or presenting updates on cases previously presented, as well as incorporating the ASSIST LAC Zika ECHO program into undergraduate and post-graduate student curricula. Many participants noted the lack of specialists in different localities and the potential the ECHO program offers to create and strengthen health teams.

Feedback from participants on the project also centered around the request for additional time to integrate multidisciplinary knowledge (which is often not present in their local area) and to discuss the patient cases. The time available during the sessions was often not enough to cover the participants’ expectations in terms of asking questions

g:


and providing input. One solution to this perceived lack of time would be to host a continuous ECHO program that runs over time, enabling a short, medium- and long-term knowledge build-up.

Providers’ indicated that participation in the ASSIST LAC Zika ECHO program was associated with increased self-efficacy, which in turn had an impact on patients and care delivery. Immediate changes observed following participation in the ASSIST LAC Zika ECHO program include the strong teams with a greater closeness to families and children. Participants reported that families acknowledged the changes in practice, and in some cases, families organized themselves to facilitate self-management of care.

The percentage of ASSIST LAC Zika ECHO program participants who rated themselves as “above average or expert” in capacity to share knowledge with other professionals increased from 29% prior to program to 91% following participation in the program.

These evaluation findings suggest that ASSIST LAC Zika ECHO participants have become an asset that should be leveraged for knowledge sharing in the LAC region.

USAID APPLYING SCIENCE TO STRENGTHEN AND IMPROVE SYSTEMS PROJECT

University Research Co., LLC

5404 Wisconsin Avenue, Suite 800

Chevy Chase, MD 20815

Tel: (301) 654-8338

Fax: (301) 941-8427

www.usaidassist.org

latin america and caribbean zika extension for community … · 2020-06-29 · latin america and...

Documents