evaluation plan (phase i) - aphl · evaluation plan (phase i) ... c. current programs..... 5 ii....

Evaluation Plan (Phase I)

for

Association of Public Health Laboratories (APHL)

2016‐2020

June 29, 2016

Association of Public Health Laboratories (APHL) Evaluation Plan (2016)

06/29/2016 (draft)

Table of Content

I. BACKGROUND............................................................................................................................................................. 3

A. ASSOCIATION BACKGROUND AND HISTORY ....................................................................................................................... 3 B. APHL’S VISION AND MISSION ....................................................................................................................................... 4 C. CURRENT PROGRAMS ................................................................................................................................................... 5

II. PREPARATION OF EVALUATION PLAN ....................................................................................................................... 6

A. EVALUATION FRAMEWORK ............................................................................................................................................ 6 B. METHODS .................................................................................................................................................................. 7 C. STAKEHOLDER EXPECTATIONS ........................................................................................................................................ 8

III. APHL EVALUATION PLAN .......................................................................................................................................... 10

A. EVALUATION OBJECTIVES ............................................................................................................................................ 10 B. EVALUATION APPROACH ............................................................................................................................................. 10 C. THEORY OF CHANGE .................................................................................................................................................. 11 D. EVALUATION QUESTIONS ............................................................................................................................................ 14 E. DESIGNS AND DATA COLLECTION METHODS ................................................................................................................... 17 F. DATA COLLECTION INSTRUMENTS ................................................................................................................................. 22 G. DATA ANALYSIS ......................................................................................................................................................... 22 H. UTILIZATION OF FINDINGS ........................................................................................................................................... 22 I. LIMITATIONS ............................................................................................................................................................ 23

IV. TIMELINES ....................................................................................................................................................... 23

V. APPENDIX .................................................................................................................................................................. 25

A. EVALUATION PLAN DEVELOPMENT INSTRUMENTS AND PRESENTATIONS ............................................................................... 25 1. Capacity Development Guide ‐ Evaluation 101 ................................................................................................ 25 2. Key Informant Interview Instrument ................................................................................................................ 30 3. Focus Group Discussion Guide ......................................................................................................................... 32 4. Facilitated Group Discussion Guide on Theory of Change ............................................................................... 33 5. Survey Instrument on Evaluation Questions .................................................................................................... 36

B. MASTER CDC LOGIC MODEL ............................................................................................................................... 38 C. APHL THEORY OF CHANGE .................................................................................................................................. 39 D. APHL SURVEY LIST .................................................................................................................................................... 40


06/29/2016 (draft)

I. Background The APHL/CDC Partnership: Strengthening Public Health Laboratories1 cooperative agreement between the Association for Public Health Laboratories (APHL) and Centers for Disease Control (CDC) calls for an evaluation plan to “help CDC and [APHL] to:

1. Monitor which activities planned were successfully completed… 2. Demonstrate how capacity building activities contribute to program outcomes… and 3. Inform decisions about future programming that drive continuous program improvement for more

efficient and effective program performance” Accordingly, in January 2016, APHL hired an evaluation manager to develop an evaluation plan to serve as a road map for evaluating the work done under the CDC cooperative agreement. The evaluation plan is the result of a few months of conversations with various stakeholders including APHL staff.

A. Association Background and History APHL is a 501(c)(3) nonprofit organization representing member state and local governmental health laboratories in the United States. Its members, known as “public health laboratories,” aim to monitor and detect health threats and conditions to protect the health and safety of Americans. APHL’s core membership is comprised of state and local governmental health laboratories in the United States, including public health, environmental, agricultural science and food safety laboratories. Representatives from federal agencies, nonprofit organizations, corporations and interested individuals also participate in the Association. Even though APHL, as it exists today, was formally founded a little over 50 years ago as a forum for state public health laboratory directors, it has a longer history spanning almost a century:

In 1921, the Southern Public Health Lab Association was formed.

In 1927, is was renamed as the State Laboratory Director's Conference (STLD) when it opened its membership to all other states.

In 1939, membership was opened up to the provincial laboratory directors in Canada.

In 1951, the organization was officially founded including all states and territories and changed its name to the Association of State and Territorial Public Health Laboratory Directors (ASTPHLD). In 1950 (a year earlier), the California Association of Public Health Lab Directors (the only state level association of public health Laboratories) was formed.

In 1998, ASTPLHD, an organization that comprised only of one individual from each state, expanded its membership base and finally changed to its current name ‐ the Association of Public Health Laboratories (APHL).

In addition to its domestic work, APHL also works internationally to develop effective national laboratory systems and expand access to quality diagnostic testing services. International participation is expanding in response to the globalization of disease and APHL’s active global health program, which has documented over 20 years' experience in 31 countries on five continents. At present, APHL has over 800 institutional and individual members.

APHL is largely funded through cooperative agreements with federal agencies including the CDC (the largest funder), Food and Drug Administration (FDA), the President's Emergency Plan for AIDS Relief (PEPFAR),

1 Funding Opportunity Number: CDC=RFA-OE15-1501


06/29/2016 (draft)

Health Resources and Services Administration (HRSA) and Environmental Protection Agency (EPA). In addition, membership dues both from traditional members and corporate members bring in some income. This evaluation plan is funded through the master cooperative agreement with CDC and will focus on activities funded through this agreement.

B. APHL’s Vision and Mission Since the late 1800s, public health laboratories (PHLs) have served on America’s front line of defense against many forms of illness. Today public health laboratorians remain relatively unrecognized despite their ongoing role in identifying potentially devastating microbial, chemical and radiological exposures, including, for instance participating in responses as against illnesses (such as Ebola), disasters (such as the West Virginia chemical spill), and terrorist attacks (such as anthrax). Public health laboratories provide scientifically‐valid data as a basis for decision making and policies. They play a crucial role in disease surveillance by providing high‐risk testing as well as specialized reference microbiology, serology and molecular typing to identify disease clusters and sources. PHLs also fill a special niche for “low frequency, high intensity" diseases such as rabies, plague, botulism and anthrax. In addition to microbiology, public health laboratories provide a wide array of other analytical services, including environmental chemistry, food & feed testing, toxicology and newborn screening. Due to their critical public health mission, PHLs must not only be capable of detecting myriad health threats, but they also must ensure the sustainability of their operations. However, PHLs face serious challenges. These challenges include rapidly emerging disease threats (such as viral hemorrhagic fevers), extensive restructuring of the nation’s healthcare system, accelerating evolution in testing and information technologies, and the ongoing need for safe laboratory practices. Additionally, PHLs often face persistent fiscal problems, workforce reductions, aging instruments that need to be maintained, and other issues that threaten to reduce capability and capacity. The nation must address these challenges so that PHLs can effectively contribute to the larger, national public health system, including the achievement of Healthy People 2020 goals and other US health strategies. Today, in addition to exemplifying scientific rigor and advancement, laboratory directors must be expert leaders and administrators, able to respond to the unknown or unusual at any moment, and be politically adept to gain agreement among constituents and partners that may not have been in existence 60 years ago. Because APHL is the only organization representing state and local PHLs, it is uniquely positioned to provide a voice to PHLs. In nearly a century of existence, APHL has grown and changed in parallel with the dramatic shift in role of the public health laboratory. Moved by its vision for a healthier world through quality laboratory systems, APHL’s mission is to “shape national and global health outcomes by promoting the value and contributions of public health laboratories and continuously improving the public health laboratory system and practice.” APHL’s vision and mission are in line with the stated purpose of the CDC cooperative agreement, which is “to improve the effectiveness, capabilities, and sustainability of public health laboratories (PHLs) — individually and as components of a national PHL system — in the United States and in other countries”Error! Bookmark not defined..


06/29/2016 (draft)

C. Current Programs APHL collaborates with laboratory and public health partners to assure effective surveillance, detection and response to health threats through the following programs:

Public Health Laboratory Strengthening and Leadership (LSL) includes the infrastructure elements APHL needs to effectively support its work, members and partners. LSL includes work in Strategic Communications, Institutional Research, Governance and Policy, as well as the coordination of efforts needed to manage and administer cooperative agreements.

Public Health Preparedness and Response (PHPR) works to improve the effectiveness, capabilities and sustainability of PHLs to prepare for, respond to and recover from all‐hazards threats. The PHPR Program works with CDC, across APHL and with multiple external partners to (1) provide guidance, tools and training; (2) promote information and technology transfer from CDC and other agencies to member laboratories; (3) expand and enhance relationships between member laboratories and other stakeholders; (4) provide support for the Laboratory Response Network; (5) shape and promote policies that support laboratory preparedness and response.

Global Health (GH) aims to facilitate to implementation of CDC and HHS’ response to global health security and the West Africa Ebola Outbreak. A key goal of the APHL Global Health Program is to develop independent country capability and capacity through policy and planning, training, technology transfer and laboratory infrastructure and system development.

Laboratory Systems and Standards (LSS) promotes the development and adoption of quality laboratory practices within the national PH system. As part of APHL’s broader Quality System program, LSS provides tools and guidance to enable PHLs and their system partners to strengthen and improve quality system management and provides an evaluation framework to measure the implementation and impact to ensure continuous quality improvement.

Informatics (Info) promotes the needs of public health in the area of information technology, and works to help all APHL members and public health partners achieve real‐time, electronic laboratory data exchange. Work in this program area encompasses a broad range of informatics related activities, from educational and collaborative activities to the investigation and piloting of viable technology architectures which can support, among other things, the implementation of CDC’s surveillance strategy, to the provision of technical assistance.

Workforce Development (WD) aims to address workforce challenges and to strengthen the public health and clinical laboratory workforce through continuing education and training, leadership development, recruitment and retention strategies, improving collaboration in the PHL system and between APHL and CDC.

Newborn Screening (NBS) works to strengthen the role of public health laboratories in genetics testing and designs strategies to address changes in the newborn screening testing field. NBS proactively develops and recommends position statements related to newborn screening and genetics to the Association. NBS also provides input to CDC’s Newborn Screening Quality Assurance Program on quality control and proficiency testing issues relevant to newborn screening laboratories across the globe. The program interacts with state, federal and Association partners in implementing national


06/29/2016 (draft)

recommendations on newborn screening and genetics testing by participating on national and federal committees.

Food Safety (FS) aims to advance laboratory practice by increasing the competence of laboratory staff, improving the capabilities of member laboratories and strengthening the credibility of results generated by food testing laboratories. The Program promotes the work of PulseNet laboratories in submitting timely, high‐quality data to the PulseNet National Databases, including efforts to help the network move to Whole Genome Sequencing.

Infectious Diseases (ID) works with public health laboratories and other partners to sustain, strengthen and continually improve infectious disease detection and surveillance activities in the US and internationally. To accomplish this goal, the Program coordinates the evaluation of new technologies, test methods and practices; provides and assures access to highly‐specialized or low‐volume testing services through a network of Reference Centers or shared‐service models; develops guidelines and recommendations on laboratory policy and practice; provides training to PHLs, clinical laboratories and other partners; provides technical assistance to PHLs to promote the adoption of more advanced test methods or practices; and promotes the role of PHLs in infectious disease testing and response activities to public health partners and the public.

Environmental Health (EH) encompasses traditional environmental public health laboratory areas such as biomonitoring, in addition to chemical and radiological preparedness. The Program raises awareness of environmental health issues, develops related policies and responds to member laboratory needs. Program staff serve as liaisons between laboratories and NCEH/ATSDR in order to promote, discuss, define and explain the unique challenges faced by environmental public health laboratories. Ultimately, these activities aim to build sound policies and increase the visibility of the environmental health laboratory community, which will in turn support the improvement of environmental health laboratory practice.

In addition to the above, during public health emergencies, APHL operates as a coordinating center for laboratory response.

II. Preparation of Evaluation Plan This section describes the approaches and processes used in developing the current APHL evaluation plan.

A. Evaluation Framework The CDC framework for program evaluation2 guided the development of APHL’s evaluation plan (see figure 1). The framework outlines six steps and four quality criteria for a good evaluation plan:

Accuracy: Will the evaluation produce valid and reliable findings?

Feasibility: Are planned activities realistic for given resources, timeline and expertise?

Propriety: Does the evaluation protect the rights of all stakeholders and participants?

Utility: Will the evaluation provide needed results?

2 Centers for Disease Control and Prevention (CDC) (1999). Framework for program evaluation in public health. Morbidity and Mortality Weekly Reports (MMWR), 48.


06/29/2016 (draft)

The framework asserts that good evaluation not only provides accurate evidence for drawing valid conclusions but also aims for results that can be used to make a difference. Use is ensured by identifying and engaging stakeholders and focusing evaluation questions to those that serve their needs. Feasibility and need for accurate results are assessed to determine appropriate in data collection activities.

B. Methods Critical stakeholders for this evaluation plan were identified to be APHL staff, CDC staff and APHL members. The following methods were used to engage these stakeholders and develop the evaluation plan between February and June 2016.

Capacity development trainings – APHL held basic evaluation trainings for staff to build consensus on the goals and approaches for the evaluation (see appendix for training materials)

Key informant interviews – Conducted 22 in‐depth key informant interviews with APHL senior staff to understand the big picture for the APHL evaluation (see appendix key informant instrument)

Focus group ‐ Held a focus group discussion with CDC senior staff to understand CDC’s needs for evaluation findings (see appendix FGD guide)

Facilitated group discussions – Led more than ten group discussions with different APHL programs as well as an APHL committee to build and refine the APHL Theory of Change (See appendix for group discussion guides).

Survey – Fielded an online survey (using Qualtrics platform) to all APHL staff to solicit relevant evaluation questions (see appendix survey instrument)

Document reviews – Relevant documents, including APHL strategic plan (2014‐2017), CoAg applications, past evaluation plan and evaluation report as well as review of PHLs and APHL related literature.

Figure 1: CDC Evaluation Framework


06/29/2016 (draft)

Key informant interviews and focus group discussions were recorded with the participants consent and transcribed. A qualitative analysis software, NVivo3, was used for initial data analysis of transcripts, survey information and its findings were used as part of the evaluation plan development.

C. Stakeholder Expectations Based on the interviews, focus groups and meetings with stakeholders, several themes emerged through the analysis. There was consensus among all interviewed individuals that an evaluation of the Association within the context of the CDC cooperative agreement is both welcomed and needed.

“As I understand it, #1, it’s something that CDC wants from us… [and #2] historically, the association has also wanted to… in essence evaluate what it does. ‘Is the association accomplishing what it sets out to with respect to its membership? …How well are we improving the public health laboratory system and the public health system through the efforts because after all, that’s why we’ve been given money by these funding agencies?”

Some themes that emerged from APHL staff as to the usefulness of evaluation of the Association were:

Help us understand our contribution to public health “What I'm hoping the evaluation will do is have us all sit together and understand a little bit more about how our work impacts health outcomes. Then be able to collectively as an organization to showcase the value of the work that we do.”

Help us do our work better “If we can learn from the evaluation and apply it to something else we're going to do away, and something that with have to do anyway, just our regular work.”

Help us tell our stories “We need to be able to constantly evaluate our projects and the work that we do to be able to show value and therefore get more money. It might not be quantifiable or it will be difficult. Somehow we still have to be able to connect those dots.”

Feel good about our impact “Get stories both from how our members are being impacted by our work and then beyond them and separate from them, how we're changing the field of public health. I think we could feel like we're having an impact, that we're not just spinning our wheels or doing things to do work, but actually having an impact.”

CDC staff agreed that evaluation of the Association’s work is important not only for demonstrating impact to funders (congress) but also for identifying areas of improvement and highlighting the value of the work to those that are not as familiar with the association and therefore may not understand the complexities of the work done by the Association.

“Be able to make more meta analytical judgments about the status of the relationship, because my sense is that, oh this great relationship, we love APHL, APHL is great, and that's probably true, but there may be really good parts about, really good things that are happening, and some things that we need to focus more on, from both sides' point of view.”

3 QSR. (2015) NVivo qualitative data analysis Software; QSR International Pty Ltd. Version 11


06/29/2016 (draft)

“Before I came into the CDC, I would not have been aware of the criticality of the need of these partner organizations like APHL … that serve as the interface node between CDC and large groups of partners, because we can't ...”

In addition, both APHL and CDC staff outlined several areas of inquiry that would be beneficial to answer (see section III.D‐ Evaluation Questions). One highlighted area was assessing the needs of the public health laboratory system to assist in moving forward into the future.

“Where are public health laboratories going? How are the evolving over the next ten years? What changes are we going to see? To inform how we need to be thinking about the workforce for the future, to inform what technology platforms we should be thinking about now, investing in. Also thinking about what the future of readiness looks like. What does that actually mean, and how will that change over time to help public health and public health laboratories think about the value of the services that they provide to the public? With these changes and trends, are there actually going to be useful services and products, or are they not? …We fail to sometimes recognize the changes in the world, as if we're not part of it, but everything we do is integrated with the rest of the world. If APHL can help us to have more of a relationship with industry, and also in other sectors, to help us think forward about public health laboratories, I think that would be really helpful.”

When asked for key attributes of a plan that would be useful, relevant and feasible, APHL staff described an evaluation process that:

Does not produce more work for staff “In order to keep it sustainable, I think it has to be something that it doesn't take too much time from the work that we do. Something that's embedded with the activities that we're doing”

Reduces current reporting burden “We do reports over and over again on every type of project, annual reports. The more of those that can be done for other things; there’s a year review for a project, we did a 12‐year review anyway. Those are the kind of things we repeatedly are trying to get the word out, so ways to better do that.”

Utilizes existing data sources “I feel like we have a program that’s pretty conducive to good evaluation because we do a lot of studies. We have things to count which makes evaluations somewhat easy.”

CDC staff, stated that they would like to see tools that would easily highlight progress on a number of fronts and reports with actionable findings.

“The challenge is to come up with evaluation reports that provide useful, actionable material for the relationship between APHL and CDC. Identifying areas where, if they're fixed, things will be better… It's not a program review that looks promotional and everything is beautiful. It actually is a concise, focused list of things ... If we did this, this, and this, and we could all get behind. “ “I would like to see a dashboard of APHL's performance in a variety of domains… it could be performance with agreed upon goals ‐ one is workforce development, which is a common issue. The other might be something … laboratory instrument placement and maintenance. Another would be the readiness domain”

06/29/2016 (draft)

2. Key Informant Interview Instrument

Interview instrument to start developing APHL’s evaluation plan Background

1. How long have you been with the association? What roles have you occupied over the years?

2. What do you think the evaluation will cover or is trying to achieve?

a. What are your worries about it?

b. What are you excited about?

3. What is the reason for putting together the evaluation plan now?

4. What was your experience with the last evaluation conducted? (Probe: process of putting

together the evaluation plan, data collection, reporting)?

a. What do you believe was achieved by it?

b. What did you get from it?

c. Why do you believe it was not continued?

(Discuss my mandate from my (evaluator’s) perspective at this point.) (May I tape?) APHL Vision/Mission

5. What is the vision that APHL is working towards? Probe for

a. The large scale changes that the organization hopes to encourage – not limited to what

your program affects… but more what APHL is trying to contribute to)

b. Refer to vision statement on APHL website – what does “a healthier world” mean? What

about “quality laboratory systems”?

c. If APHL was wildly successful, what differences do you think you will see in 5 years?

Note: Vision: A healthier world through quality laboratory systems. Mission: Shape national and global health outcomes by promoting the value and contribution of public health laboratories and continuously improving the public health laboratory system and practice.

6. What are the preconditions to this vision? (Theory of change) Probe for

What needs to happen for this large vision to be realized (intermediate outcomes) – both

within our control and outside our control?

What are assumptions that are held?

What are external influences? (All these might need to be measured.)

7. What is APHL’s mission – its contribution to the vision statement? What is APHL trying to

achieve?

a. Which of the outcomes/impacts from the above question (vision) is APHL trying to

influence?

b. In the past, have there been unexpected impacts – those not planned for? What were

those?

8. How is APHL trying to achieve this change? What are the broad strategies?

a. What is the underlying assumption in how APHL is achieving change?

06/29/2016 (draft)

b. What type of organization is APHL – primarily? (Probe: networking, capacity

development, etc.)

c. What are some of things that might prevent APHL from achieving its goal? What are

some of the outside influences that you do not have control over?

9. Who are APHL’s partners? Who is APHL trying to influence? (Thinking of boundary partners)

a. What would APHL like to see these partners doing in the future if we are to achieve the

vision?

10. How is APHL’s mission different under the new CoAg?

Program Goal

11. What is your program’s mission? How does your program mission help achieve APHL’s mission?

12. What are some of the major activities that you work on?

13. What are some of the major barriers you encounter as you do your work?

Evaluation Plan

14. As we are putting together an organization wide evaluation, what things do you think should be considered? Probe for enablers (e.g. exiting structures, culture, etc.), barriers

a. Should we be responsible for final outcomes or the increased capacity of our partners?

15. Thinking of your own program area, what are some questions or areas that you would like an

evaluation help you with?

16. (May wait till later) What are some of the data you collect routinely? What data do you collect

not so regularly?

17. Are there specific events or meetings that you think I should attend to give me a sense of the

kind of work that you do that contributes to the overall organization mission?

18. Are there any documents that you think I should review? E.g. Situational analysis, strategic plan

documents, etc…

19. When is a good time to talk to your staff together – perhaps together with other programs that

you work closely with?

06/29/2016 (draft)

3. Focus Group Discussion Guide

APHL CoAg Evaluation

Updates for CDC staff

Today’s discussion is about Phase I: 1. Who is the audience for evaluation findings (remember we are thinking of overall CoAg evaluation)?

Prioritize and focus on just 2‐3 audiences.

a. What is the reason that this audience is important with regards to the CoAg?

b. What would success look like when described to this particular audience?

c. How would you use evaluation findings with this particular audience?

d. What types of information/evidence will be convincing to the audience? (Probe for

qualitative/quantitative, good examples)

2. What are important questions to address in this evaluation? Probing questions:

a. What things would you like to know about APHL’s work under this CoAg?

b. What questions come up repeatedly, in conversations with others or in your own work,

concerning effectiveness, impact and/or success of APHL’s work under this CoAg?

3. As APHL is developing this evaluation, what things do you think should be considered? Probe for

enablers (e.g. exiting structures, culture, etc.), barriers.

Phase I (comprehensive – organization wide) – APHL is now in the process of planning this phase. Goals are

• Demonstrate how capacity building activities contribute to program outcomes (were outcomes

of interest achieved)?

– To illustrate to stakeholders how the work APHL is doing under the CoAg is contributing to

broader public health mandates (impact)

• Inform decisions about future programming that drive continuous program improvement for

more efficient and effective program performance (e.g. what and how could things be

improved)?

– To provide the Association with information they can use to improve their work under

this CoAg (learning)

Phase II (Program by program) – APHL will start planning for this once the first phase has been implemented (Year 2). This will entail designing a consistent performance and monitoring tool as the goal is to

• Monitor the extent to which activities planned were successfully completed (e.g. were activities

implemented in the expected time frame)?

• Inform decisions about future programming (program specific) that drive continuous program

improvement for more efficient and effective program performance (e.g. what and how could

things be improved)?

06/29/2016 (draft)

4. Facilitated Group Discussion Guide on Theory of Change

06/29/2016 (draft)

06/29/2016 (draft)

5. Survey Instrument on Evaluation Questions

Dear staff: As you know, we are in the process of developing an evaluation plan to be submitted to CDC by the end of this month. The goal of this evaluation plan is to

Demonstrate to key stakeholders if and how APHL’s work contributes to broader public health mandates (impact)

Inform decisions about future programming to drive continuous program improvement, i,e, to provide the Association with information we can use to improve our work (learning)

In order to ensure that we develop an evaluation plan that produces findings that is useful, we have been conducting a variety of conversations with key stakeholders within and outside the Association. As a critical stakeholder in this process, we would like to ask you to complete this brief survey in order to provide your input. Please answer whichever question you want. This is a completely confidential and anonymous survey – it is designed such that your name will not be available even to the survey team. Anonymous Survey Link Please complete this survey by noon Monday, June 20. Thanks, Sara ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐Body of Survey The goal of the current APHL evaluation plan is to

Demonstrate to key stakeholders if and how APHL’s work contributes to broader public health mandates (impact)

Inform decisions about future programming to drive continuous program improvement, i,e, to provide the Association with information we can use to improve our work (learning)

In order to ensure that we develop an evaluation plan that produces findings that is useful to your work, please answer any or all of the following questions in whatever length you prefer. This exercise is designed to develop relevant (to you) and focused evaluation questions to address goals we have listed above. Remember, a useful evaluation should answer real questions ‐ not just show others something you already know.

3. Complete the following sentence: “If I knew _______________, it would make a difference in the work I do.” (Open ended)

4. What questions seem to come up repeatedly, in conversations with others or in your own work,

concerning the effectiveness, impact or success of the Association or some of the strategies or programs? (Open ended)

06/29/2016 (draft)

5. Imagine yourself in the role of funders or the technical monitors you work with. What do you want to know about the work done by the Association? (Open ended)

6. Please rate the importance of answering the following questions (1 = low importance and 5 – high importance)

Evaluation Question Importance

a. What are the current needs/demands of members? How effective is APHL in addressing these needs?

b. How effective are key strategies _(fill in the blank)____________ (e.g. network building, training, partnerships) for bringing about needed change?

c. What are the factors and mechanisms that support needed change? What are those that hinder them?

d. What is the process by which change occurs in the public health lab system?

e. What are barriers/opportunities for more effective relationships between different partners (APHL and CDC, APHL and members, APHL and other Associations)?

7. Please tell us any other comments you may have. (Open ended)

06/29/2016 (draft)

D. APHL Survey List

Survey Name Primary Category Primary APHLProgram

Purpose Target Audience Deployment

Year(s)

Funding cuts Survey Budget Public Policy To assess level of funding cuts for member laboratories

51 state PHLs 2009, 2011

Local PHLs Profile/Core Laboratory Profiles Survey

Budget Institutional Research

To determine the local PHLs profile 42 local PHLs 2011, 2016

State PHLs Profiles / Core Laboratory Profiles Survey

Budget Institutional Research

To determine the state PHLs profile 51 state PHLs 2008, 2011, 2016

Communications Communications Communications To assess the communications outlets Entire membership 2015

Biosafety and Biosecurity Preparedness Survey

Lab practices/services

Public Health Preparedness and Response

To assess the status of biosafety and biosecurity programs and needs in public health labs (PHLs)

Biosafety officials in local, state and territorial PHLs

2016

Comprehensive Laboratory Services Survey (CLSS)


Quality Systems A biennial survey, which is the sole data source for the Healthy People 2020 Public Health Infrastructure 11 objective (PHI‐11)

51 state PHLs 2009, 2011, 2013, 2015

FDA IMS Analysts Survey Lab practices/services

Food Safety To gather information regarding the stateFDA Interstate Milk Shippers (IMS) state central milk laboratorians certification process

62 state and ag PHLs 2015

First Responders Outreach Survey



To assess PHLs’ outreach practices and challenges related to first responder training, standards, field screening and sample submission

53 PHLs 2015

HIV Diagnostic Testing Survey Lab practices/services

Infectious Disease

To capture the current HIV testing practices in state and local PHLs.

95 state and local PHLs

2015

HIV Testing practices Lab practices/services

Infectious Disease

To capture the current HIV testing practices in state and local public health laboratories


2009, 2012, 2015

NBS Incidental Findings Survey Lab practices/services

Newborn Screening

To gathered information on how state NBS programs handle different scenarios concerning incidental findings

53 state NBS programs

2015

Neisseria Meningitids and Haemophilus influenzae


Infectious Disease

To collect information on which testing methods are being used by state public health laboratories to identify Neisseria meningitides or Haemophilus influenzae in case patients


2014

Newborn Screening Timeliness Survey


Newborn Screening

To assess and identify practices, gaps and barriers related to NBS timeliness in state NBS programs

94 NBS program directors and NBS follow‐up coordinators

2010, 2012, 2014, 2015

Public Health Laboratories Systems Survey (PHLSS) ‐ Local



41 local PHLs 2015

Public Health Laboratories Systems Survey (PHLSS) ‐ State



51 state PHLs 2015

VPD Impact Survey Lab practices/services

Infectious Disease

To monitor customer satisfaction with the available services at the APHL/ CDC Vaccine Preventable Diseases (VPD) Reference Centers.

2010

Billing and Policy Guide Survey Member needs/evaluation

Quality Systems To gather information from state and local PHL directors about how they have utilized the guides and their plans for future use

95 state and local PHL directors

2015

CE Ad Awareness Survey Member needs/evaluation

Newborn Screening

To assess the effectiveness of efforts to raise awareness of the online CE activity for the 2012 CDC guidelines for newborn screening and genetic testing

2014 NBS symposium attendees

2015

Member satisfaction Member needs/evaluation

Member Services To assess member needs and satisfaction. Entire membership 2013

Vaccine Preventable Disease (VPD) Customer Satisfaction Survey

Member needs/evaluation

Infectious Disease

To monitor customer satisfaction with APHL/CDC VPD Reference centers

101 epi and lab reference POCs

2015

Affordable Care Act (ACA) Survey

Policy/Regulations APHL To assess how the ACA and other healthcare delivery system changes impact PHLs


2014, 2015

MPS‐1 Survey Policy/Regulations Newborn Screening

public health system impact assessment for MPS‐1


2014, 2015

X‐ALD Survey Policy/Regulations Newborn Screening

Utilized for a public health system impact assessment for X‐ALD


2015

All‐Hazards Laboratory Preparedness Survey

Testing capabilities


To assess, on a yearly basis, the ability of public health laboratories to respond to biological, chemical and radiological terrorism events, as well as public health threats such as pandemic influenza and food‐safety outbreaks.

54 state/territorial/local PHLs)

2007‐2015

CRE Survey Testing capabilities

Infectious Disease

To collect information on state PHL capacities, practices, and needs for CRE (Carbapenem‐Resistant Enterobacteriaceae )testing

51 state PHLs 2015

Environmental Laboratory Survey


Environmental Health

To gather information on environmental laboratories

95 state PHLs, member environmental labs and non‐member environmental labs

2011, 2015

GC/CT clinical lab practices Testing capabilities

Infectious Disease

To assess baseline data on clinical laboratories testing activities regarding GC and CT prior to releasing the guidelines


2014

Malaria Testing Survey Testing capabilities

Infectious Disease

To assess state and local PHLs malaria testing capabilities


2014, 2015

Next Generation Sequencing Testing Infectious To establish a baseline of NGS 97 state and local 2014

06/29/2016 (draft)

Survey Name Primary Category Primary APHLProgram

Purpose Target Audience Deployment

Year(s)

(NGS) Survey capabilities Disease implementation in PHLs PHLs,

PulseNet Testing capabilities

Food Safety To assess laboratory capacity and capabilities, specimen transport from clinical laboratories to public health laboratories, turnaround times and PFGE laboratory workflow


2007, 2011

Right Size Influenza Virology Surveillance


Infectious Disease

To assess PHLs use and implementation of the right size roadmap


2011, 2014

Salmonella Testing capabilities

Food Safety To assess salmonella testing algorithms in order to identify gaps within the national surveillance system.


2011

Zika Virus Testing Testing capabilities

Infectious Disease

To determine PHLs Zika testing status 58 state and localPHLs

2016

Individual Workforce survey Workforce Workforce Development

PHLs workforce enumeration Laboratory directors and staff at state, local and institutional PHLs

2011, 2016

Organizational Workforce Survey/National Laboratory Workforce Capacity

Workforce Workforce Development

PHLs workforce enumeration 95 state and local PHLs

2011, 2015

evaluation plan (phase i) - aphl · evaluation plan (phase i) ... c. current programs..... 5 ii....

Documents