american educational research association annual conference new york – march 24-28, 2008 noelle...
DESCRIPTION
Section 1 WebPlay Evaluation OverviewTRANSCRIPT
American Educational Research AssociationAnnual Conference
New York – March 24-28, 2008
Noelle Griffin, Ph.D.
Evaluation of an Arts-Based Instructional Program for Primary Grades: Preliminary Findings and Lessons Learned from WebPlay
• Section 1: WebPlay Evaluation Overview• Section 2: Survey Design and Analysis• Section 3: Pre-/Post- Comparison Results
Overview
Section 1WebPlay Evaluation
Overview
WebPlay Program
• Internet-enhanced arts education project
• Integrates theater arts in school curriculum• Performing arts, literacy, social studies, and
technology
WebPlay Curriculum
• Two weekly lessons• Use of internet: Connections with partner classrooms• WebPlay curriculum involves activities in the
following four areas:Performing Arts
English Language Arts
History/Social Studies
Technology
Evaluation
• 3 year evaluation beginning 2005-06• 3rd and 5th grade classes/large urban school
district• Quantitative methods/small scale• Both summative and formative purposes
Evaluation Questions: WebPlay Program
1. Will WebPlay improve student performance?2. What is the impact of WebPlay on student skill
development?3. Do effects persist across different student/site
cohorts?4. Do different student/site characteristics interact
with treatment?
Sampling and Methodology
• Sample (2006-07 school year) 18 WebPlay schools
12 matched comparison schools (for survey)
• Student survey (pre-/post-)• Standardized state test data (English Language
Arts) – future analysis
Evaluation Roadblocks
• Selecting outcome measures• The focus on standardized data• Lack of resources• Comparison participation
Section 2Survey Design and Analysis
Key Constructs in WebPlay Curriculum
1. Theatrical knowledge2. Internet knowledge and safety3. Engagement4. Efficacy and academic esteem5. Collaboration and broadening horizons
Sample Items by ConstructAreas Items
1. Theatrical knowledge
The location or place (“setting”) of a play is where the play is being presented.
2. Internet knowledge and safety
E-mails as well as the documents sent via e-mail (“attachments”) can have viruses.
3. School engagement I think the activities I do in school are boring.
4. Efficacy and academic esteem
I would be able to do a good job if I had to write a story for school.
5. Collaboration and broadening horizons
I can learn new things from students who live in other countries.
Factor Analysis
• Pre- surveys• N = 590 (424 WebPlay, 166 comparison)• Exploratory and confirmatory factor
analysis techniques
Exploratory Factor Analysis
• Based on Likert-style items• Three higher order factors determined
Theatrical engagement/interest
General academic confidence/engagement
External connections
Confirmatory Factor Analysis
• Initial fits indices mixed (i.e., not all indices reached cut-offs)
• Initial model revised (allow for cross-loadings, correlated residuals)
• Consistent with survey theoretical design• Fit indices improved
WebPlay Student Survey: Final CFA Model
Section 3 Pre/Post Comparison
Results
Survey Pre-/Post- Analyses
• Comparing pre/post results for WebPlay and comparison students
• Based on only students with link-able pre/post data (i.e., smaller sample)
• Hierarchical linear modeling techniques used (due to nested nature of data)Two-level model (student and school)
Applicability of Hierarchical Linear Models (HLMs) to Evaluation Studies
• HLM allows analysis of outcome variable relationship to predictors at multiple data levels
• Evaluation studies often involve: Intact clusters
Nested data
Background characteristics of individuals that may vary appreciably across different clusters
Overall Results
• Two sets of outcomes analyzedKnowledge
Attitude/Engagement
• Knowledge: No significant group differences (p = .41)
• Engagement/attitude: WebPlay students increased pre-/post- relative to comparison students (p = .02)
HM Results for Knowledge Outcome
Fixed Effects Coefficient SE p Value
Intercept 7.10 0.21 <.0001
WebPlay -0.22 0.26 0.41
Pre-test 0.20 0.06 0.00
Random EffectsVariance Compone
ntSE p Value
Adjusted means 0.18 0.10 0.03
Student residual 2.12 0.17 <.0001
HM Results for Engagement/Attitude Outcome
Fixed Effects Coefficient SE p Value
Intercept 3.80 0.07 <.0001
WebPlay 0.19 0.08 0.02
Pre-test 0.57 0.05 <.0001
Random Effects
Variance Componen
tSE p Value
Adjusted mean 0.00 0.01 0.43Student residual 0.36 0.03 <.0001
Sub-scale Analyses
• Descriptive/exploratory in nature• No variation in impact between types of
Knowledge items (theater, internet)• Some apparent variation in impact on
attitude/engagement sub-itemsTheatrical engagement/interest WebPlay vs.
comparison group differences most pronounced
Attitude Outcome sub-area scores
by treatment group
3
3.2
3.4
3.6
3.8
4
4.2
4.4
4.6
1 2
Implementation (Pre and post)
Ave
rage
sco
res
Theatrical, Control
Academic,Control
Collaboration, Control
Theatrical, Treatment
Academic, Treatment
Collaboration,Treatment
Next Steps
• Integration of state standardized test data• Revision of Knowledge items (multiple
choice)• Collection and integration of teacher-level
data