Evaluation of the Process Evaluation of the Process Required to Expand the Required to Expand the National Laboratory System National Laboratory System (NLS) to All States(NLS) to All States

CDC Evaluation Using a ContractCDC Evaluation Using a Contractwith with BattelleBattelle Memorial InstituteMemorial Institute

Robert Martin, DrPHRobert Martin, DrPHEunice Rosner, EdDEunice Rosner, EdD

Patricia Bolton, PhD, Patricia Bolton, PhD, BattelleBattelle leadlead

Formative EvaluationFormative Evaluation

Formative evaluation is used to Formative evaluation is used to improve a program, especially improve a program, especially when it is still being developed when it is still being developed ((ScrivenScriven, 1991), 1991)

Our Purpose Was to Answer Our Purpose Was to Answer the Following Questions:the Following Questions:

What practices were employed by the What practices were employed by the demonstration sites?demonstration sites?How interested are How interested are SPHLDsSPHLDs and CL managers in and CL managers in collaborating with each other?collaborating with each other?What are What are CLsCLs’ perceptions of ’ perceptions of SPHLsSPHLs??–– Do Do CLsCLs in demo and nonin demo and non--demonstration sites differ?demonstration sites differ?

Do background and activities of the SPHLD affect CL Do background and activities of the SPHLD affect CL perceptions of the state lab and/or their interest in perceptions of the state lab and/or their interest in working together?working together?

MethodologyMethodology

Three major components:Three major components:Case studies of NLS demonstration sitesCase studies of NLS demonstration sitesSurvey of state public health laboratory Survey of state public health laboratory directors (directors (SPHLDsSPHLDs))Survey of a sample of clinical laboratories Survey of a sample of clinical laboratories ((CLsCLs))

Today’s Focus:

Results of the Survey of Clinical Laboratory Directors/Managers

Survey: SectionsSurvey: Sections

Background information about laboratory Background information about laboratory ––type, testing volume, etctype, testing volume, etcAmount of interaction with SPHLAmount of interaction with SPHLCommunication received from SPHLCommunication received from SPHL–– Training, PH alertsTraining, PH alerts–– Satisfaction with servicesSatisfaction with services–– How to improve communicationHow to improve communication–– Specimen transport, testing Specimen transport, testing –– Data exchangeData exchange

Sampling FrameSampling Frame

Clinical Laboratories that perform Clinical Laboratories that perform “comprehensive microbiology”“comprehensive microbiology”–– From the National Laboratory Data BaseFrom the National Laboratory Data Base–– Labs enrolled in comprehensive Labs enrolled in comprehensive

microbiology proficiency testing programmicrobiology proficiency testing program–– Population: 3,991 Population: 3,991 –– Sample selected: 738 laboratories, Sample selected: 738 laboratories,

randomly stratified to include all statesrandomly stratified to include all states

ResponsesResponses

71% response rate 71% response rate Final number of usable surveys Final number of usable surveys –– 503*503*

*Note *Note –– some laboratories selfsome laboratories self--reported as reported as HMO or PH labs and some surveys were HMO or PH labs and some surveys were incomplete so had to be eliminated from incomplete so had to be eliminated from the analysisthe analysis

Characteristics of CL Characteristics of CL RespondentsRespondents

Job Title

32%

64%

4%

Lab Directors Lab Managers Non-sup.

Laboratory Type

88%

11%1%

Hospital Independent No response

Responses from Responses from CLsCLs in demo in demo and nonand non--demo statesdemo states

Demonstration sites:Demonstration sites:N = 37 N = 37 (11 from Minnesota, 13 from Michigan,(11 from Minnesota, 13 from Michigan,and 13 from Nebraska)and 13 from Nebraska)

NonNon--demonstration sites:demonstration sites:N = 466 from all other statesN = 466 from all other states

Were you invited by your SPHL Were you invited by your SPHL to participate as a sentinel to participate as a sentinel laboratory in the LRN? laboratory in the LRN?

Total that Total that responded toresponded toquestionquestion Demo Sites NonDemo Sites Non--Demo Demo N=491 N=491 N=36N=36 N=455N=455

%Yes* %Yes* 4949 5858 4848%No 31%No 31 3131 3131%DNK 20%DNK 20 1111 2020

*Differences not significantly different in demo and non*Differences not significantly different in demo and non--demon at p

Do you have regular points Do you have regular points of contact at your SPHL?of contact at your SPHL?

DemoDemo NonNon--demodemo

YesYes 83%83% 69%69%

NoNo 17%17% 31%31%

p=.067p=.067

Public HealthPublic Health--Related Related Activities Reported* Activities Reported*

93%93% -- Notify SPHL or state EPI of results of Notify SPHL or state EPI of results of potential health threatspotential health threats

71%71% -- Hold briefings for staff about PH threats**Hold briefings for staff about PH threats**69%69% -- Have someone on staff responsible for PH Have someone on staff responsible for PH

matters**matters**66%66% -- Notify physicians about PH threats**Notify physicians about PH threats**25%25% -- Participate in CDC/SPHL epidemiology Participate in CDC/SPHL epidemiology

program such as program such as FoodnetFoodnet, Emerging , Emerging Infections Program or otherInfections Program or other

* * Number of responses varies from 465 Number of responses varies from 465 -- 492492** More likely if CL director/supervisor has some PH background** More likely if CL director/supervisor has some PH background(p

Frequency of Inquiries to Frequency of Inquiries to SPHL (n=503)SPHL (n=503)

58%16%

4%

2%

20%

none

CL Microbiology Staff CL Microbiology Staff Consultation with SPHL Consultation with SPHL 10% would consult on AST, 72% on BT issues10% would consult on AST, 72% on BT issuesReasons cited by those who did not were:Reasons cited by those who did not were:

61%61% -- Inability to quickly locate a point of contactInability to quickly locate a point of contact44%44% -- Different hours of operationDifferent hours of operation19%19% -- Not an appropriate source for some Not an appropriate source for some

informationinformation13%13% -- Lack of confidence in SPHL expertiseLack of confidence in SPHL expertise10%10% -- Concern about regulatory interventionConcern about regulatory intervention8%8% -- Concern about interference in testing Concern about interference in testing

methodsmethods

Availability of and Satisfaction Availability of and Satisfaction with SPHL Contacts*with SPHL Contacts*

*Question: If you have regular points of *Question: If you have regular points of contact, how would you rate them?contact, how would you rate them?97%97% -- Always or frequently available and Always or frequently available and respond in a timely manner (N=336)respond in a timely manner (N=336)90%90% -- Quality of responses are good or Quality of responses are good or excellent (N=391)excellent (N=391)

Which of the following would Which of the following would increase your laboratory’s reliance increase your laboratory’s reliance on the SPHL as a source of on the SPHL as a source of information?information?

85%85% –– Better understanding of the Better understanding of the SPHL’sSPHL’scapabilities and roles (N=445)capabilities and roles (N=445)

75%75% –– Better understanding of our Better understanding of our laboratory’s role in PH (N=413)laboratory’s role in PH (N=413)

67%67% –– Expanded services of SPHLExpanded services of SPHL(N= 373)(N= 373)

48%48% –– Improved responsiveness of SPHL Improved responsiveness of SPHL to our inquiries (N= 361)to our inquiries (N= 361)

How well would you say the SPHL How well would you say the SPHL understands the capabilities of your understands the capabilities of your laboratory?laboratory?

0

10

20

30

40

50

60

70

80

90

100

Demo 71 21 9Non-demo 48 32 20

Completely, pretty well

Somewhat well Not very well

%

Demo Demo vsvs nonnon--demo (p=.032)demo (p=.032)

How do you think your laboratory How do you think your laboratory would benefit from increased would benefit from increased interaction with the SPHL?interaction with the SPHL?

90%90% –– Being part of a laboratory Being part of a laboratory network or community (N=417)network or community (N=417)

89%89% –– Additional resource about PH issues Additional resource about PH issues (N=435)(N=435)

84%84% –– Better information about new tests Better information about new tests and procedures (N=432)and procedures (N=432)

78%78% –– Better access to new tests and Better access to new tests and procedures (N=422)procedures (N=422)

Perceptions of SPHL Activities Perceptions of SPHL Activities

0

20

40

60

80

100

SPHL 100 85 79 77 55 52CL 79 83 55 43 42 59CL wants more 90 86 88 51 76 83

work-shops alerts

man-uals

site visits

confer-ences

news-letters

%

Perceptions of SPHL ActivitiesPerceptions of SPHL Activities

.000.00048488282TeleconferencesTeleconferences

n.sn.s..42425252Site Site visits

.002.00253538181Manuals/Manuals/proceduresprocedures

.011.01162628484Surveys of lab Surveys of lab capacitycapacity

.022.02278789494WorkshopsWorkshops

.000.00055559797NewslettersNewsletters

p p valuevalue

NonNon--demo %demo %

Demo Demo site %site %

TopicTopic

*Note: N varies from 16-33 for demos and from 160-336 for non-demos

Timeliness of ContactsTimeliness of Contacts

Does your SPHL notify you in a timely Does your SPHL notify you in a timely manner about new or emerging public manner about new or emerging public health threats?health threats?

DemoDemo NonNon-- DemoDemo

YesYes 95%95% 66%66%N=37N=37 N=459N=459

P=.002P=.002

Modes of Communication Modes of Communication Preferred by CLs (N= 500) Preferred by CLs (N= 500)

0

10

20

30

40

50

60

70

80

90

100

59 30 4 3 3 2

Fax E-mail Tele-phoneText

MessagDon't know Other

%

Electronic Communication Electronic Communication Capability Recognized by CLCapability Recognized by CL

Q: Does SPHL have the electronic Q: Does SPHL have the electronic capability of sending test results to capability of sending test results to your laboratory? (N=494)your laboratory? (N=494)

YesYes 13%13%NoNo 54%54%DNKDNK 33% 33%

Electronic Communication Electronic Communication Capability of CLCapability of CLQ: Do you have the capability of Q: Do you have the capability of

electronically sending test results to electronically sending test results to your SPHL? (N=491)your SPHL? (N=491)

YesYes 35%35%NoNo 65%65%

Clinical and SPHL Reports Clinical and SPHL Reports of Collaborationof Collaboration

0

10

20

30

40

50

60

70

80

90

100

clinical 15 11 42 13 48 44

SPHL 51 50 65 36 96 83

Safety AST EID Regs BT agents Spec trans

%

Clinical Lab Interest in Clinical Lab Interest in Collaboration with SPHL Collaboration with SPHL

0

10

20

30

40

50

60

70

Currently do 15 11 42 13 44 48Would like to 69 68 55 42 48 59Not Interested 17 21 4 20 8 3

Lab Safety AST EID

Regs – QA/QC

Spec-imen

Trans-

BT Agents

%

Are CL perceptions of Are CL perceptions of SPHLsSPHLsdifferent in demo states and nondifferent in demo states and non--demo states?demo states?

YesYesWhat makes demo states different?What makes demo states different?–– Provide more services than averageProvide more services than average–– Have single point of contact (laboratory Have single point of contact (laboratory

program advisors)program advisors)–– Communicate frequently with clinical labsCommunicate frequently with clinical labs–– Provide more visibility for public healthProvide more visibility for public health

Suggested Activities for Suggested Activities for SPHLsSPHLs

Provide services/resources clinical Provide services/resources clinical laboratories are interested inlaboratories are interested inCommunicate Communicate CLsCLs’ role in PH’ role in PHImprove collaboration efforts Improve collaboration efforts Market role of SPHLMarket role of SPHLUse FAX, EUse FAX, E--mail for PH alertsmail for PH alertsUpgrade websites, electronic Upgrade websites, electronic capabilitiescapabilities