Evaluation of the IAFF Hazmat and WMD Training ProgramsEvaluation of the IAFF Hazmat Evaluation of the IAFF Hazmat and WMD Training Programsand WMD Training ProgramsWETP Fall Technical Workshop, 10-18-07, Sheraton Chapel Hill, Chapel Hill, NCPatrick MorrisonAssistant to the General PresidentInternational Association of Fire Fighters

Bruce Lippy, Ph.D., CIH, CSPPresident, The Lippy Group, LLC

Introductory CommentsIntroductory CommentsPatrick Morrison

Assistant to the General PresidentIAFF

Bruce LippyBruce Lippy

President, The Lippy Group

Lippy Group Lippy Group Company Picnic 2007Company Picnic 2007

First row, left to right: Lippy, B.

Lippy Group Corporate MetricsLippy Group Corporate Metrics

Evaluation TeamEvaluation Team

Bruce Lippy, Ph.D., CIH, CSPJames Weeks, Sc.D., CIHBernard Mizula, MS, CHS-V, CET, RPIH

IAFF provided us an office and full access

Bruce Bruce Lippy’sLippy’s experience with experience with fire fightingfire fighting

Setting duckpins for the Manchester, MD Fire

Company

Evaluation approachEvaluation approach1. Interviews of board members,

headquarters staff, and students2. Evaluation of NIEHS and DHS

progress reports3. Focus groups with IAFF instructors 4. Analysis of IAFF student database5. Comparison of findings against

recommendations from guidance documents

6. Onsite evaluation of training program

7. Follow-up interviews of federal, state and county officials

Final Issues for evaluationFinal Issues for evaluation1. Methods of communication 2. Student database3. Student registration process4. Class records (storage, accuracy, etc.)5. Class sizes 6. Make-up of instructor cadre 7. Current programs 8. New programs9. Instructor resource center10. Blended learning

Evaluation timelineEvaluation timelineInitial discussions March 06

Interviews, observations and data analysis

Sep to Nov 06

Presentation to General President and Advisory Board of initial findings

Oct 06

Evaluation of presentation at Trainers Exchange

March 07

Draft reports March to May 07

Final report (v. 12) July 07

Broad Finding:Broad Finding:

NIEHSNIEHSMinimum Minimum Criteria is Criteria is excellent basis excellent basis for program for program evaluationevaluation

Is there evidence that the program is accomplishing its objectives?

IAFF ranked #2 in cost per contact hr.IAFF ranked #2 in cost per contact hr.(NIEHS Annual Report 2003)(NIEHS Annual Report 2003)

IAFF ranked 7/17 in cost per studentIAFF ranked 7/17 in cost per student(NIEHS Annual Report 2003)(NIEHS Annual Report 2003)

IAFF Number of traineesIAFF Number of traineesProjected v. actual, Projected v. actual, (FY96 (FY96 –– FY04)FY04)

Are appropriate facilities and staff available and committed to the program?

YES!“Training coordinators” are key to effectivenessStaff meetings short, focused and productive“Lead” instructor facilitates communication with the training site and with headquartersTraining at the host site with their equipment

Methods of CommunicationsMethods of CommunicationsRecommendationRecommendation

Emphasize prerequisites for courses, particularly T-t-T, and don’t rely solely on the

requesting organization

Methods of CommunicationsMethods of CommunicationsRecommendationsRecommendations

Include more actual case studies in reports to NIEHS

NIEHS official:*“Firefighters save lives every day, but their progress reports don’t reflect that sufficiently, they can be boring.”

*Ted

Student DatabaseStudent DatabaseImpressive File-Maker Pro database maintained by an outside contractor for the last 8 years38,000 records in database described as a “learning management system”

System covers shipping, System covers shipping, finance, and classesfinance, and classes

IAFF Shipping ProcessIAFF Shipping Process

Unlike other NIEHS grantees, IAFF relies upon host for equipment Student notebooks and reference materials (Pocket Guide, DOT ER book) Shipped out of headquartersRecommended positioning Guides in active locations, like AZ, rather than constantly shipping from HQ

Gender

Male91%

Female6%

NA0%

Missing3%

Student Demographics from Student Demographics from database: Genderdatabase: Gender

0.010.020.030.040.050.060.070.0

Percent

Cauca

sian

Hispan

ic

African

America

n NA

Asian/Pac

ific Is

lande

r

Native

America

n

Student Demographics: Student Demographics: EthnicityEthnicity

Age of IAFF StudentsAge of IAFF StudentsMean age is 33 yearsMean age is 33 years

Histogram: Age

-10 0 10 20 30 40 50 60 70 80 90 1000

2000

4000

6000

8000

10000

12000

14000

16000

18000

No.

of o

bs.

Schooling of IAFF StudentsSchooling of IAFF Students

Hig

h Sc

hool

Som

e C

olle

ge NA

Gra

de S

choo

l

Bach

elor

's D

egre

e

Asso

ciat

e D

egre

e

Post

Bac

helo

r's D

egre

e

Mas

ter's

Deg

ree

02000400060008000

10000120001400016000180002000022000240002600028000

No

of o

bs

0.05.0

10.015.020.025.030.035.040.045.0

Percent Completed

FRO

ERT:OpsCSO

Tech

TtT FRO

WMD TtT

zOther ID

Inc. Mgt.

Courses ProvidedCourses Provided

0102030405060708090

Percent Employed

Fire Serv

iceOthe

rNA

Private

Indu

stry/C

onsul

tant

Law Enfo

rcement

Public

Safety

Private

EMS

Indus

trial F

ire Brig

ade

Emergency

Man

agemen

t

Public

EMS

Employment of studentsEmployment of students

05

101520253035

Percent

Less

than

one y

ear

1 - 5

year

s

Over 2

0 yea

rs6 -

10 ye

ars

11 - 1

5 yea

rs16

- 20 y

ears

Not App

licable NA

Years of employment Years of employment experience of studentsexperience of students

What are the program’s strengths?

The instructor cadre. Currently 91 instructors (84 at time of survey)Headquarters staffTechnical quality of the curriculaSupport from General PresidentGuidance from Advisory BoardCoordination with Safety and Health staff

What are the program’s What are the program’s weaknesses?weaknesses?

Feedback to instructorsQuality Assessment Reports are generated by headquarters from student evaluations, but provision to instructors needs to be more consistent and timely

What are the program’s What are the program’s weaknesses?weaknesses?

Annual evaluation of instructors.A point for discussion by all Awardees:

“The annual review of instructor competency shall include, at a minimum, observation by the Training Director or his or her designee of instructional delivery, review and discussion of observations with the instructor, and an analysis of the instructor performance based upon evaluations completed by trainees during the previous year.”

Minimum Criteria guidance (9.3.3)

Frontline SafetyFrontline SafetyCurriculum development meetingCurriculum development meeting

September 21, 2006September 21, 2006

Is the evaluation tool current and appropriate for program

content?

Student evaluation form unchanged in 8 yearsExcellent and consistent efforts by Alexander Cohen, Ph.D.New efforts underway by Dr. Harold Stolovitch look quite promising

Harold Stolovitch in venturing into Kirkpatrick Level 4

evaluations

Blended LearningBlended Learning

IAFF has a talented individual leading the charge: Vilma Perez-AtwoodIAFF has created impressive web-based courses, such as the Avian Flu unitGiven the importance of skills training, not as much emphasis on web-based training as other granteesAvailability of computers and access to web still issues

IAFF NearIAFF Near--Miss ProgramMiss Program

IAFF NearIAFF Near--Miss ProgramMiss Program

An important program focused on an area that can have significant impact– OSHA insists on a near-miss program in

VPP– DOE’s Lessons Learned culture has made

a differenceReaching out to the airline industry is bold and can pay real dividends

Do you enjoy being an Do you enjoy being an instructor in the IAFF program?instructor in the IAFF program?

“EXTREMELY! Any of the problems/challenges outlined here are normal organizational stuff. They are

overshadowed by the chance to share with fellow responders in their own

environment. You really get the feeling (most of the time) that you’re making a positive difference for these students

and that’s what it’s all about .”

Questions?Questions?