“Modules of Workforce Development: Pragmatic Approaches to Knowledge Acquisition, Application, and Information Dissemination”

Enterprise Access to Evidence-Based Literature: A Dynamic Digital Library for Public Health

Enterprise Access to Evidence-Based Literature: A Dynamic Digital Library for Public Health

Karen Dahlen, Project Consultant Elaine Martin, Director

New England Region NN/LM Lamar Soutter LibraryUniversity of Massachusetts Medical School

This project is currently funded by the NLM via contract through the NN/LM, New England Region

Enterprise Library Approach to Knowledge Acquisition, Application & Information Dissemination

Measure Importance of Trusted Information Access

Ensure Direct Access from Intranet-Based Site

Access (Without Name/Password Required)

Support Core Competencies/Training

Use Survey Instruments Adapted to the Location

Relate Cost Efficiencies to Enterprise Licensing/Creative Partnerships

Document Institutional Change in a Dynamic EnvironmentEvaluate Over Time (Illustrate Efficiencies)

Shifting the Culture to Evidence-Based Practice

• Culture of evidence-based practice was affected when funding for public health libraries was eliminated.

• State Libraries (in many states) were mandated to support state agencies, but had no collections to support public health.

• Streamlined access to resources through the Digital Library Platform-- along with trust in new relationships--prompted resurgence in use of resources and has facilitated this cultural shift.

• Training stimulated access to resources reviving evidence-based interest and relationship to practice.

• Project has the ability to leverage government resources (NLM, CDC, NAL) to improve evidence-based practice.

ENTERPRISE ACCESS TO EVIDENCE-BASED LITERATURE: A DYNAMIC DIGITAL LIBRARY FOR PUBLIC HEALTH

Background (Why this Project)

Project Rationale

Partnerships

Council on Linkages: Academia & Public Health Practice

Public Health Competencies

• Analytic/Assessment Skills– References sources of public health data and information.– Information technology to collect, store, retrieve data.– Utilizes data to address scientific, political, ethical, and

social public health issues.

• Policy Development/Program Planning Skills– Analyzes information relevant to specific public health

policy issues

• Public Health Sciences Skills – Conducts a comprehensive review of the scientific

evidence related to : a public health issue, concern, or intervention.

– Retrieves scientific evidence from a variety of text and electronic sources.

• Knowledge Informatics– Access trusted resources directly from the

desktop.– Describe relationships and functionality of e-

resources to core public health interests.– Introduce specific types of resources: coding

and classification tools.• Informatics Skills

– Train on interoperable features that support public health work, including connectivity to trusted sites; advanced search filters, use of topical alerts and information mgt software.

Competencies: Council on Linkages Project Tied to Competencies

Obstacles to Getting Information

Gathering and Analyzing EB Information Rapid Detection of Epidemics

Detecting an anthrax

epidemic one day earlier

would save $1-7 billion

CDC Presentation by

C. Safran citing:

http://www.cdc.gov/ncidod/eid/vol3no2/kaufman.htm

Kaufmann AF, EID, V3, N2

ENTERPRISE ACCESS TO EVIDENCE-BASED LITERATURE: A DYNAMIC DIGITAL LIBRARY FOR PUBLIC HEALTH

Library Without Signature Building

Goal of the Project

Who ParticipatesObjectives & Data Sets

Development of Digital Library

PHIA: Innovative Business ModelNot Tied to Traditional Building

Madison, WI Atlanta, GA

Goal of PHIAPublic Health Information Access

To determine what resources are core, useful, evidence-based to advance public health practice.

Full text directly available through DL

Full text available through LP

Core to public health

Unique Aspects of Digital Library

• Resides on PHD Intranet (outside a signature building) without need for id and password.

• Relies on collaboration, interoperability & funding.• Move toward analysis; less time on access• Skills & training support personalized, interactive styles tied

to continuum of learning and institutional-specific priorities and projects.

• Collection of evidence occurs in many aspects of Brownson wheel not just at the literature review stage.

• Fundamental change in the way resources are distributed and utilized.

Public Health Departments& Library Partners

State Public Heath Department Library Partners AK Alaska Division of Public Health (AKDH) Univ of Alaska (Anchorage) Medical LibraryAR Arkansas Department of Health (ADH) Univ of Alabama Medical Sciences LibraryCO Colorado Dept of Public Health & Environment Poudre Valley Health System Library

CT Connecticut Public Health Department University of Conn Maynard Stowe Library HI Hawaii Department of Health University of Hawaii Medical LibraryIN Indiana State Department of Health Ruth Lily Medical Library, Indiana UniversityKY Kentucky Department for Health University of Kentucky Medical Library

ME Maine CDC Maine Medical Center Library MA Boston Public Health Commission* University of Massachusetts Soutter Library MA Massachusetts Public Health Department Lemuel Shattuck Hospital Library MD Maryland Dept of Health & Mental Hygiene University of Maryland HS LibraryNH New Hampshire DHHS New Hampshire DHHS Library RI Rhode Island Department of Health (RIDH) Rhode Island Hospital/LifespanVT Vermont Department of Health (VDH) UVM Dana Medical Library WI Wisconsin Department of Health Services Ebling Library of the Health Sciences WV West Virginia Bureau of Public Health West Virginia University LibrariesWA Washington State Department of Health University of Washington HS Library

PHIA Evaluation ProcessTied to Data Model

To advance access to evidence-based resources in support of improved public health practice.

What resources are high use?

What technical issues arise?

What resources are missing? How can training be improved?

What project components lead to improved efficiencies?

What cost efficiencies are involved with an enterprise approach to digital library implementation, and training.

• Logic Model– Resources, activities, outputs, outcomes

(minutes, visits, trainings, people lists, IP management, levels of troubleshooting, RML reports.

• Data Tools– Overarching questions– Journal SurveyMonkey– Pre-Post Training Surveys– Just-in-time Information Checks– Formal Evaluation (interviews & focus

group sessions

Data Sets & Sources

• Resource Metrics Collected• Journal Survey Monkey Identifies

Perceived Need & Library Use.• Document IT technical issues. • Licensing Issues Related to

Journals and Databases. • Training Surveys Collected• Preliminary Meetings/Updates Held.• Trainings Held• Year End Evaluation

--Interviews & Focus Groups

View of PHIA Digital Libraries

Connecticut Digital Library Arkansas Digital Library

ENTERPRISE ACCESS TO EVIDENCE-BASED LITERATURE: A DYNAMIC DIGITAL LIBRARY FOR PUBLIC HEALTH

Resources on Digital Library

Use of Resources

Article Delivery Use & Costs

Databases & Major Publishers of Resources

PubMed , Other NLM DatabasesNAL, CDC & Other Gov’t Resources

CLSI (Clinical Laboratory Standards) Global HealthCochrane Library

– Systematic ReviewsStat!Ref—E Books (35)

-- Coding & ClassificationASABE –Health Linked to Agriculture

Identifying Core, Useful, & Evidence-Based Journals

Publisher Total Use Journal Title UseMass Medical Society (1) 14,136 New England Journal of Medicine 14,136

AAP(1) 9,888 Pediatrics 9888

Oxford University Press (20) 23,369 Clinical Infectious Diseases 3147

Springer (18) 2, 485 Maternal & Child Health 474

ASM Journals (12) 3,518 Journal of Clinical Microbiology 1729

Annual Reviews (9) 697 Annual Review Public Health 394

Sage (11) 939 The Diabetes Educator 135

BMJ (9) 2932 British Medical Journal 1832

Mary Ann Liebert (11) 1798 Breast Feeding Medicine 355

University of Chicago Press (6) 955 Infection Control & Hospital Epidemiology 749

Lippincott/Ovid (6) 122 J Public Health Mgt Practice 55

NEJM Use 2010-2013

0

1000

2000

3000

4000

5000

6000

7000

4864

1772

982554 314 527 337 311

890308

1183 1178494 422

2010=2087 2011=11,194 2012=14,022 2013=14,136

ISDHM

DPH

MDHM

H

CDPHE

Alaska DH

KYCPH

NH DHHS

WID

HSRID

HVDH

ADH

HI DPH

Maine CDC

CT PHDBPHC

298 287233

175135

94 90 89 82 78 7142 32 18 5

ASM: Journal of Clinical Microbiology

Total Use 2013=1729

Oxford University Press UseJan-Dec 2013

Total Use by Each PHDN=13,827 High Use Journal Titles

Journal Title Total UseClinical Infectious Disease 3147American Journal of Epidemiology 941Schizophrenia Bulletin 771The Journal of Infectious Diseases 627

ICES: Journal of Marine Science 474JNCI: Journal of the National Cancer Institute 434

Journal of Antimicrobial Chemotherapy 297

International Journal of Epidemiology 288Alcohol and Alcoholism 274Journal of Analytical Toxicology 262Health Promotion International 252Rheumatology 246Age and Ageing 240

PHD TotalAlaska DPH 1245Arkansas DH 1483BPHC 182CDPHE 608CT DPH 1234HI SDH 638ISDH 700KYDPH 1134Maine CDC 757MDHMH 1427MDPH 1817NH DHHS 157RIDH 620

VDH 709WI DHS 1116

STAT!Ref E-Books/Tools Provide Basic Understanding of Topic

Filters & Tools– Codes[5]

• CPT with RVUs Data File, INGENIX® (2011)[1]

• ICD-9+CM - Volumes 1, 2 & 3 (2012)– Titles By Discipline[344]

Searchable Alerts (by topic)

– Functionality with NLM– Chapters link to PubMed– TOXNET linked to Search– Related Concepts/Meta…

Functionality with CDC– MMWR, Community Guide;

STAT!Ref E- Books for Public Health

Titles Containing PH ContentE-Book Use

ACP PIER, Journal Club & AHFS DI Essentials (2013) 482Oxford Textbook of Public Health - 5th Ed. (2009) 407

Manual of Clinical Microbiology - 10th Ed. (2011) 403

AHFS Drug Information (2013) 305Red Book: 2012 Report of the Committee on Infectious Diseases - 29th Ed. (2012) 241

Emerging Infections Series (2008 - 2010) 216

Textbook of Modern Toxicology, A - 4th Ed. (2010) 209

Coding & Classification BooksE-Book Use

ICD-9-CM - Volumes 1, 2 & 3 (2013) 881

CPT with RVUs Data File, INGENIX (2013) 474

ICD-10-PCS: Procedure Coding System (2014) 12

ICD-10-CM: Clinical Modification (2014) 15

Diagnostic and Statistical Manual of Mental Disorders 4th Ed. (DSM-IV-TR, 2000) 107

CLSI (Clinical & Laboratory Standards)20 Documents Used 790 Times

April 21, 2014Doc # Document Name #M100-S24

M100-S24: Performance Standards for Antimicrobial Susceptibility Testing; Twenty-Fourth Informational Supplement

248

EP09-A3

Measurement Procedure Comparison and Bias Estimation Using Patient Samples; Approved Guideline—Third Ed.

102

EP12-A2

User Protocol for Evaluation of Qualitative Test Performance

55

MM09-A2

Nucleic Acid Sequencing Methods in Diagnostic Laboratory Medicine; Approved Guideline—Second Edition

54

Calculations Related to Enterprise Licensing

State PHD FTE 10% RuleADH 2781/1390 139AKDPH 525 52BPHC* 1100/550 55CDPHE 1200/600 60CT DPH 780 78Hawaii DPH 3000/1500

150ISDH 780 78Kentucky BPH 400 40MDHMH 8000/4000 400Maine CDC 395 39MDPH 3000/1500 150NH DHHS 279 30RIDH 400 40VDH 300 35WDPHS 400 40VW DHS 780 78Washington

1600/800 80Total PHIA FTE 10,999 1546

Enterprise Licensing is based on 10% of the Total FTE level to capture baseline information and measure use/interest of resources.

Article Delivery Costs Via Library Partnerships

May 2011 – April 2012; Cost=$18,384 May 2012 --- April 2013; Cost=$12,241

STATE ILL DD Total

Connecticut 0 83 83

Maine 560 364 924

Massachusetts (BPHC) 8 3 11

Massachusetts (MDPH) 1 1 2

New Hampshire 0 0 0

Rhode Island 47 36 83

Vermont 21 191 212

Colorado 323 156 479

Total 960 834 1794

STATE ILL DD  Total

Connecticut 2 20 22

Maine 154 90 244Massachusetts (BPHC) 9 10 19Massachusetts (MDPH) 7 6 13New Hampshire 0 0 0

Rhode Island 82 79 161

Vermont 54 243 297

Colorado 112 40 152

Arkansas 32 73 105

Kentucky 8 3 11

Wisconsin 0 0 0

Total 462 593 1055

Training Topics Embrace National and State Interests

Maryland DHMH Health Initiatives• Health Disparities(sexual minorities) • Preventive Services ACA* HIV Screening• Immigrant Health

Undocumented individuals under ACA• Drug Resistant Disease which impacts

Impact on TB, STI (plus others) • Health Care Reform and Interpersonal

Violence/Domestic Violence• Infectious Diseases; Hepatitis C Virus

HIV Infections --HIV and HCV Co-Infection

Number of Trainings/Diversity of WorkforceSTATE PHD Trainings Attendance

RI RIDH 2 30

VT VDH 3 46

MEMAINE CDC* 3

52

MA BPHC* 4 58

MA MPHD* 3 32

NH NH DHHS 2 24

CO CDPHE* 5 100

CT CT DPH* 5 101

KY KDPH 2 46

AR ADH 2 31

WI WI DPH 2 33

AK AK DPH 2 50

HI HI DPH 2 68

IN ISDH 2 60

MD MDHMH 2 38

TOTAL 15 41 769

Maryland Indiana Hawaii

Nutrition Specialist (2)

State Epidemiologist Education Coordinator (2)

Research Statistician Supervisor – Labs + (3) Asthma Coordinators(2

Policy Analyst (2) Injury Prevention Epidemiologist

Legislative Liaison

Program Mgr (2) Staff Attorney Epidemiologist

Epidemiologist Microbiologist ( 2) Research Analyst (2)

Medicare Specialist/Advisor (2)

Regional Program Director

Program Coordinator, SAPB

Policy Advisor (2) Director of Prog Evaluation

Epidemiologist (3)

Psychiatric VT Director of Prog Development

Genetic Counselor

Executive Ass’t Field Epi Director Tobacco Prevent Mgr

DeputySecretary

Access Services Mgr Informatics Analyst

Director VSA IT Supervisor Nutritionist

ENTERPRISE ACCESS TO EVIDENCE-BASED LITERATURE: A DYNAMIC DIGITAL LIBRARY FOR PUBLIC HEALTH

Enterprise EfficienciesUnique Aspects of the Business ModelEvaluation Processes

Cost Efficiencies: Article Analysis

Direct Access to Articles Alternative Article Delivery

Contract Year Total # Amount $

May 2013-March 2014 1038 10,102

May 2012-Apr 2013 1055 12,241

May 2011-Apr 2012 1794 18,385

May 2010-Apr 2011 672 8,263

Contract Year Total # Amount $

May 2013 –April 2014 17+ 285,409

May 2012-Apr 2013 15 200,352

May 2011-Apr 2012 11 156,280

May 2010-Apr 2011 9 96,252

Enterprise Licensing Efficiencies: Single TitlesContract Year May 2012-April 2013

Journal Title # Uses Cost Per Use

Replacement Cost

Cost of License

Am J Tropical Med & Hygiene 410 $1.58 $4,510 $ 650

Am J Respiratory & Critical Care Med 189 $14.28 $2,079 $2700

Health Affairs 2038 1.81 $22,418 $3690

Infection Control & Hospital Epidemiology

749 1.50 $8,239 $1125

Public Health Reports 658 4.69 $7,238 $3085

Intern’l J of Tuberculosis & Lung Diseases

192 5.83 $2112 $1120

Annual Review Series 595 12.60 $6545 $7500

Pediatrics 7846 .68 $86,306 $5300

PHIA: Innovative Business Model

• Enterprise licensing saves time, money, and effort.• Central management of IP addresses ensures quick turn-around.• Ability to understand change (System, personnel, environment).• Resources are identified via benchmarking,metrics of direct use through

Digital Library, and requests from PHDs.• Journals are identified via “Article Delivery” on Library Partner side.• Alternative delivery of resources strengthens state relationships with

immediate access (within 4-24 hours).

EVALUATION PROCESSES Levels of Data Collection

• Journal SurveyMonkey (baseline data for PHIA and PHD)• Enterprise Licensing (cost effectiveness over time)• Vendor statistical reports (validates “use of resources”) • Monthly reports from “library partners” (measures use and identify

resources to develop Digital Libraries). • Ongoing capture of suggested e-books, journals, and databases to

enhance collection• Pre-Post training links

– (data related to knowledge of resources)• Feedback from hands-on training• Interviews with leaders and workforce• Focus group sessions (after one year; subsequently) • Quarterly/Annual Reports (submitted to NLM)

Results

• Digital libraries will be in place in 17 PHDs by July.• Full-text access to more than 150 e-journals, 5

databases, reports and more.• Introductory sessions have introduced PHD leadership to

the project and explained obligations.• More resources have been added to “digital library.”• Structured training has been held in 15 PHDs.• More than 600 people have been trained with backup

models currently in test stage.

NACCHO Award Brings Recognition to Project

• March 2012 and 2013, the PHIA PHD Digital Library Project received a “promising practice award” from NACCHO.

• Presentations have been held at national and local meetings, e.g., APHA in Washington, DC in 2011 and Boston in November 2013.

• Article was published in AJPH in January 2014. Project was cited in August 2012 EID (CDPHE).

• Presentations have been made at national public health meetings, national and regional library meetings.

Testimonial

“Great recognition for this very valuable initiative. Kudos and thank you.”

Jewel Mullen, MD, MPH, MPA, Commissioner, Connecticut Department of Public Health

Discussion & OutcomeHow Do We?

• Expand PHIA to all 50 states.• Enhance partnerships & relationships.• Continuously train PH Workforce to improve

competencies (knowledge and skills) given staff turnover.

• Build a sustainable model including cost structure and project management.