Improving Evalua/ons and U/liza/on with Sta/s/cal Edge: Nested Data Designs and
Hierarchical Linear Modeling (HLM)
CES Conference -‐ June 10 2013
Marci Pernica -‐ Ministry of Community and Social Services Judith Godin – J Godin Consul7ng
• The goal of this presenta0on is to introduce the concept of HLM and explain how it can be used in program evalua0on
Introduc0on
What is Hierarchical Linear Modeling (HLM)
• HLM is a sta0s0cal technique to analyze data that is structured in hierarchies (or “nested”) • To account for the fact that people that are “clustered” or
“nested” within the same group have more in common than if they were independent random samples
Classroom 1 Classroom
2 Classroom 3
Student 4
Student 2
Student 5
Student 3 Student 8 Student 7 Student 6 Student 9 Student 10
Student 11
Student 12
Nested data designs
Student
Class
School
District
Hierarchical Structure – mul0-‐level
age I.Q
Measuring test scores (dependent variable)
Independent variables
• HLM enables a more robust analy0c approach for nested data (than regression or ANOVA) • Data in evalua0on are oZen nested
• To determine success condi*ons for the program – e.g. is the program more suitable for certain sub-‐popula0ons or more successful if delivered in a certain way
Why Use HLM in Evalua0on
Program design structure
Data structure
Evalua0on ques0ons -‐ Which par0cipant or site-‐level characteris0cs are most influen0al in
explaining the varia0on in test scores among the program par0cipants?
-‐ What program delivery characteris0cs (site level prac0ces) seem to be having the most posi0ve impact on the par0cipants’ test scores?
-‐ Are some program features more suited to certain sub-‐popula0ons (e.g. gender, age group, ethnic or cultural group)
Applying HLM in Evalua0on
Par0cipant 1
Site 1 Site 2 Site 3
Par0cipant 4
Par0cipant 2
Par0cipant5
Par0cipant 3
Par0cipant 8
Par0cipant 7
Par0cipant 6
Par0cipant 9
Par0cipant 10
Par0cipant 11
Par0cipant 12
Example of levels of a hierarchical model
Par0cipants (level 1) nested within sites (level 2)
Assessing test scores by age from site to site
Test Score
Age
Four different program sites
Although the test scores differ from site to site, the rela0on between age and test score is the same at different sites
Student 1 Student 2 Student 3
Baseline
Month 7
Month 1
Month 12
Month 6 Month 9 Month 8 Month 6 Baseline Month 2 Month 4 Month 6
Assessing change over 0me
Assessments across 0me (level 1) are nested within individuals (level 2) (i.e. repeated measures design)
Assessing improved performance over 0me
Test Score
Time
Four different study par0cipants
Although some individuals have higher test scores to start with, the rate of change (improved performance) is comparable among the par0cipants
Tradi0onal Methods
Test Score
Age
Rela0on between age and test score es0mated once for all sites together
Advantages of HLM
Test Score
Age
Four different program sites
Here, the rela0on between age and test score varies across sites.
Are there any site level variables associated with the strength of this rela0on?
Design Considera0ons for Using HLM
• Sample size – Par0cipant level – Site level – Repeated measures
• Missing Data – Can be easy or difficult to deal with
• Number of variables – Comprehensive coverage – Parsimony
Final Thoughts
• Applying HLM in evalua0ons with nested data enables more robust results and conclusions
• U0liza0on-‐focused – Iden0fy evidence-‐based success factors or condi0ons for improving the program delivery model, to ul0mately achieve beger program effec0veness
– Promo0ng the value in evalua0on (gathering the evidence to determine the ‘success factors’ for the interven0on to be effec0ve)
Ques0ons?
Marci Pernica [email protected] Ministry of Community and Social Services Judith Godin sta/s/[email protected] Independent Consultant