Academic Program Evaluation Using BCSP Criteria

A Case Study at the University of Minnesota Duluth

Robert Feyen, Ph.D., CPEAsst. Professor/MEHS Program Director, University of

Minnesota Duluth

T.W. Loushine, Ph.D., P.E., CSP, CIHAsst. Professor, University of Wisconsin - Whitewater

UMD MEHS programBackground

• Started in 1976 with roughly 750 graduates to date• Practitioner-based program

– Designed primarily to prepare students as generalists in EHS, but allow room for some specialization

Mission statementThe Master in Environmental Health and Safety program at the University of Minnesota Duluth will produce highly-regarded and sought-after graduates who have the requisite skills and knowledge to practice environmental health and safety effectively in a diverse range of occupations and will do so in a competent, professional and ethical manner.

Curriculum: Content (2008)Core (32 credits)• SAFE 6002—Regulatory Standards and Hazard Control (3 cr)• SAFE 6011—System Safety and Loss Control Techniques (3 cr)• SAFE 6012—Risk Management and Workers’ Comp (2 cr)• SAFE 6051—Construction Safety Management (3 cr)• SAFE 6101—Principles of Industrial Hygiene (3 cr)• SAFE 6111—Industrial Noise and Ventilation Control (3 cr) • SAFE 6301—Occupational Biomechanics (2 cr)• SAFE 6302—Occupational Ergonomics (3 cr)• SAFE 6401—Environmental Safety and Legal Implications (2 cr)• SAFE 6802—Leadership, Teamwork, Behavior in EHS (3 cr) • SAFE 6821—Organization and Administration of Safety Programs (2 cr)• SAFE 6997—Cooperative Internship Electives (7 credits)• SAFE 6102—Advanced Industrial Hygiene (2 cr)• SAFE 6121—Epidemiology and Industrial Toxicology (2 cr)• SAFE 6201—Fire Prevention and Emergency Preparedness (2 cr)• SAFE 6211—Transportation Safety (2 cr)

Curriculum: Subjective Feedback

• Internal (University-level) administration– Economic constraints (e.g., coordination of resources)– Academic “consistency” with other graduate level programs

• EHS professionals (formal and informal)– Alumni– Employers– Practitioners

• Peer graduate programs in safety and health• Funding agencies (e.g., NIOSH)• External reviewers

Curriculum: Objective Assessment

• Literature– Graduate Research Council reports (e.g., student demographics)– ASSE guidelines for ABET accreditation– Academic journals (e.g., Journal of Safety Science)– ASEE and ASSE proceedings papers on curriculum assessment– BCSP outcome assessment program

• Developed an assessment approach based on Brauer (2006)– Utilized the BCSP subject matter tables (2008)– However, no other literature could be found regarding this approach

or its use

• The (the devil) details– Took a few tries, several modifications and a GRA to obtain data on

each course– Took even more tries to figure out what to do with the data…

Method• Instructors identify knowledge items from the BCSP subject

matter tables to be used in evaluating each course taught • Develop assessment matrix of courses and relevant

knowledge items• Conduct interviews with students and instructors to rank

emphasis and rate coverage of each item• Enter item ratings and rankings for each course into

spreadsheet• Analyze and interpret data

– Aggregated over all knowledge items (n=249)– For each subject area, use ratings to determine percent of items

covered within overall program and individual courses– Within subject area, identify knowledge items not covered

thoroughly or requiring additional evaluation

Method: Subject Matter (example)

Measurement and Monitoring• Knowledge (Foundation Level)

– Methods and techniques for measurement, sampling, and analysis (D2- T1)

– Uses and limitations of monitoring equipment (D2-T1) • Knowledge (Advanced Level)

– Electronic data logging and monitoring equipment (D1-T2)– Electronic data transfer methods and data storage options (D1-

T2) – Methods and techniques for measurement, sampling, and

analysis (D1-T1, D2-T1)

Methods: Measures• Objectives:

– Identify knowledge items for evaluation within a course– Rate evidence of coverage/assessment for knowledge items

Results: Initial Assessment

• 3 knowledge items not identified for rating

Results: Master Dataset

• 14 courses (horizontal)– 4 columns each

• “rated”, “primary”, “secondary”, and “tertiary”

• 249 items (vertical)– Group by subject area

• Color-coded– Green, yellow, red– Blue – no rating

Results: Aggregate by Rank & Rating

Ranking & Rating Item Count Percent of total (N=249)

Primary “>3” 114 45.78%

Secondary “>3” 150 60.24%

Tertiary “>3” 166 66.67%

Rating Item Count Percent of total“5” 160 64.26%

“4” (subtracted 5’s) 30 12.05%

“3” (subtracted 4’s & 5’s) 32 12.85%

Total “>3” 222 89.16%

Results: By Subject Areas

Subject Area

Total items

Min # Ratings

Max # Ratings

Avg # Ratings

S.D. # Ratings

% items rated “5”

% items rated >3

BMP 28 0 8 3.64 2.33 39.3% 71.4%EHFS 11 3 7 4.82 1.47 81.8% 100%EM 8 3 6 4.38 0.92 75.0% 87.5%ES 18 2 8 6.11 1.53 66.7% 94.4%ETC 23 2 11 6.30 2.20 78.3% 95.7%FS 11 1 4 2.36 0.81 90.9% 90.9%GS 8 0 6 3.25 2.19 62.5% 62.5%HRC 44 2 13 7.05 2.70 93.2% 100%HS 18 0 9 4.50 2.20 61.1% 94.4%ISP 12 2 12 5.17 3.07 66.7% 91.7%MM 5 4 9 6.40 2.07 100% 100%OBS 10 3 7 4.20 1.40 50.0% 100%RARM 18 2 11 5.61 2.79 55.6% 100%SHES 12 1 7 4.00 2.17 25.0% 58.3%SS 23 1 5 3.13 1.06 26.1% 78.3%Average 1.7 8.2 4.70 1.90 64.8% 88.3%

Results: Subject Area by CoveragePERCENT OF SUBJECT AREA RATINGS OF 3 OR GREATER 3 or Greater 5 Rating

6002 6011 6012 6051 6101 6102 6111 6201 6211 6301 6302 6401 6802 6821 Overall Overall

BMP 0.21 0.21 0.21 0.14 0.04 0.43 0.04 0.14 0.21 0.21 0.21 71.4% 39.3%

EHFS 0.09 0.27 0.18 0.64 1.00 1.00 100.0% 81.8%

EM 0.63 0.38 0.63 0.13 0.75 87.5% 75.0%

ES 0.06 0.28 0.33 0.67 0.56 0.28 0.28 0.17 0.67 94.4% 66.7%

ETC 0.65 0.26 0.26 0.57 0.39 0.26 0.65 0.13 0.39 0.04 0.43 0.30 95.7% 78.3%

FS 0.09 0.91 90.9% 90.9%

GS 0.38 0.25 0.13 0.63 0.63 62.5% 62.5%

HRC 0.68 0.16 0.02 0.82 0.50 0.50 0.50 0.36 0.36 0.27 0.32 0.18 0.07 100.0% 93.2%

HS 0.06 0.78 0.83 0.39 0.06 0.39 0.44 0.28 94.4% 61.1%

ISP 0.33 0.17 0.17 0.42 0.08 0.50 0.08 0.25 0.42 0.25 0.25 0.08 0.17 91.7% 66.7%

MM 1.00 1.00 1.00 1.00 0.80 0.40 0.40 100.0% 100.0%

OBS 0.30 0.10 0.70 0.10 0.60 0.40 100.0% 50.0%

RARM 0.39 0.61 0.50 0.22 0.17 0.22 0.06 0.56 0.50 0.44 0.22 0.06 100.0% 55.6%

SHES 0.42 0.50 0.25 0.08 0.42 0.42 0.17 0.08 58.3% 25.0%

SS 0.17 0.17 0.26 0.09 0.13 0.30 0.35 78.3% 26.1%

AVG 0.35 0.35 0.23 0.30 0.43 0.55 0.36 0.29 0.49 0.45 0.42 0.29 0.29 0.21

MAX 0.68 0.63 0.50 0.82 1.00 1.00 1.00 0.91 1.00 1.00 1.00 0.75 0.60 0.40

Blue = .80-1.00; Green = .60-.79; Yellow = .40-.59; Orange = .20-.39

Results: Subject Area by Non-CoveragePERCENT OF SUBJECT AREA RATINGS OF 2 OR LESS 3 or Greater 5 Rating

6002 6011 6012 6051 6101 6102 6111 6201 6211 6301 6302 6401 6802 6821 Overall Overall

BMP 0.07 0.04 0.04 0.25 0.11 0.04 0.18 0.50 0.36 71.4% 39.3%

EHFS 0.18 0.82 0.36 0.18 0.09 100.0% 81.8%

EM 0.13 0.13 0.38 0.50 0.25 0.50 87.5% 75.0%

ES 0.50 0.11 0.39 0.17 0.39 0.61 0.22 0.06 0.33 0.06 94.4% 66.7%

ETC 0.04 0.35 0.09 0.04 0.17 0.09 0.13 0.04 0.30 0.70 95.7% 78.3%

FS 0.55 0.09 0.55 0.18 90.9% 90.9%

GS 0.13 0.38 0.13 0.25 0.13 0.25 62.5% 62.5%

HRC 0.84 0.05 0.30 0.16 0.16 0.27 0.09 0.05 0.27 0.02 0.07 100.0% 93.2%

HS 0.22 0.11 0.17 0.11 0.06 0.22 0.33 0.06 94.4% 61.1%

ISP 0.17 0.33 0.08 0.25 0.67 0.08 0.17 0.25 91.7% 66.7%

MM 0.40 0.40 100.0% 100.0%

OBS 0.10 0.20 0.30 0.10 0.30 0.40 0.60 100.0% 50.0%

RARM 0.28 0.44 0.17 0.11 0.11 0.06 0.06 0.22 0.11 0.11 100.0% 55.6%

SHES 0.08 0.50 0.17 0.17 0.25 0.50 58.3% 25.0%

SS 0.22 0.30 0.04 0.04 0.13 0.17 0.09 0.61 0.04 78.3% 26.1%

AVG 0.08 0.29 0.22 0.00 0.22 0.11 0.26 0.28 0.20 0.13 0.10 0.27 0.23 0.33

MAX 0.08 0.84 0.44 0.00 0.39 0.17 0.82 0.67 0.55 0.30 0.25 0.61 0.50 0.70

Results: By Course Coverage• For Business Management Principles (n=28)

– 39% with rating of “5”– 71% with rating of “3 or greater”

6002 6011 6012 6051 6101 6102 6111 6201 6211 6301 6302 6401 6802 68210.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

Discussion: Cons • Requires time and effort commitment

– Process requires motivated participants– Up to two hours per course

• Faculty to provide data on the courses they teach• Subset of students to provide data on courses they have taken

– Three to four hours per course• One person to setup assessment spreadsheet, coordinate data

collection, enter and interpret data

• Terminology from BCSP blueprints is often unclear and redundant– Requires reaching consensus on definitions– May be misinterpreted

Discussion: Pros• Effective benchmarking tool

– Provide baseline measurement of topic coverage• Individual courses• Overall program

– Assess effectiveness of topic coverage• Compare faculty expectations to student perceptions of

topic coverage• Evaluate how program prepares graduates for either

the ASP or the CSP

– Evaluate impact of modifications• Course alterations• Program restructuring*

Discussion: Example of Simple Improvement

6002 6011 6012 6051 6101 6102 6111 6201 6211 6301 6302 6401 6802 68210.00

0.20

0.40

0.60

0.80

1.00

1.20

Discussion: Pros

• BCSP foundation provides objectivity and flexibility– Externally validated criteria– Provides objective assessment of curriculum– Easily adapted to any EHS program*– Other degree programs with a certification or

licensing body could apply the approach*

Summary

• The UMD MEHS program needed an objective means to assess curriculum design– Modified Brauer’s proposed approach for using

the BCSP blueprints to evaluate curriculum• Using a combined ranking and rating system

for each course in the program provided an effective multi-purpose benchmarking tool

• Questions?