analyzing ccsse data over time tair 2013, galveston, tx february 11, 2013 e. michael bohlig, ph.d....
TRANSCRIPT
Analyzing CCSSE Data Over TimeTAIR 2013, Galveston, TXFebruary 11, 2013
E. Michael Bohlig, Ph.D.Sr. Research AssociateCCCSE
2
CCSSE – A brief history of the survey
• Center established in 2001 • Survey item development and revisions through 2004
• Main survey has not changed since 2005
• Special focus items to address issues of current importance available since 2006
• Since 2011, Special item set focusing on Promising Practices
3
Source of information for evidence-based decision-making to improve student engagement and outcomes
Once decisions made and programs in place, what next?
Have your efforts have paid off?
Using CCSSE to help answer this question
4
Measuring Change – before you start Implementation– early considerations
– Goals achievable within reasonable timeframe?– Are they measurable? – Focus on things the college and staff can directly impact
Patience – Implementation dip (Fullan, 2001) – Can take up to 3 years to see positive impact
Implementation Fidelity – Can’t use CCSSE for this, but changes could be
spurious if implementation not monitored
Planning Your Longitudinal
Analysis
6
The Data – Where To Begin
At least three administrations since 2005 Benchmarks
– CCSSE website: Standardized benchmark scores are not appropriate for longitudinal analysis.
– Standardized scores re-standardized every year based on 3-year cohort
– Raw benchmark scores – available in download data file.
• Scale: 0 - 1
7
Review: How Benchmarks Are Created 1. Convert benchmark items to a 0 – 1 scale
– Items used in benchmarks do not all have same scale– Subtract 1 and divide by the number of response
categories – 1 (e.g., [4 – 1]/3).
2. Calculate the average of the 0-1 scaled variables in a benchmark – This is the raw benchmark score.
3. Using full 3-year cohort, standardized scores with a mean of 50 & standard deviation of 25, are created, for each student.
8
Standardized vs. Raw Benchmark Scores
StandardizedBenchmark Year 1 Year 2 Diff
Active and Collaborative Learning 54.63 54.03 -0.6Student Effort 50.52 49.24 -1.28Academic Challenge 50.72 49.76 -0.96Student-Faculty Interaction 54.06 51.23 -2.83Support for Learners 49.18 51.22 2.04
RawYear 1 Year 2 Diff
0.38999 0.39711 0.007120.46035 0.46406 0.003710.56512 0.57457 0.009450.41699 0.42027 0.003280.41533 0.45159 0.03626
9
Standardized vs. Raw Benchmark Scores
StandardizedBenchmark Year 1 Year 2 Diff
Active and Collaborative Learning 54.63 54.03 -0.6Student Effort 50.52 49.24 -1.28Academic Challenge 50.72 49.76 -0.96Student-Faculty Interaction 54.06 51.23 -2.83Support for Learners 49.18 51.22 2.04
RawYear 1 Year 2 Diff
0.38999 0.39711 0.007120.46035 0.46406 0.003710.56512 0.57457 0.009450.41699 0.42027 0.003280.41533 0.45159 0.03626
10
Driving Benchmark Scores - Means
Items from Support for Learners Year 1 Year 2 Diff9b Providing the support you need to help you succeed at
this college 3.04 3.10 0.06
9c Encouraging contact among students from different economic, social, and racial or ethnic backgrounds 2.31 2.46 0.15
9d Helping you cope with your non-academic responsibilities (work, family, etc.) 1.97 2.09 0.12
9e Providing the support you need to thrive socially 2.09 2.27 0.18
9f Providing the financial support you need to afford your education 2.48 2.64 0.16
13a1 Frequency: Academic advising/planning 1.71 1.78 0.07
13b1 Frequency: Career counseling 1.45 1.41 -0.04
11
Driving Benchmark Scores - FrequenciesItems from Support for Learners Year 1 Year 2 Diff9b Providing the support you need to help you succeed at this
college 23.03 22.56 -0.47
9c Encouraging contact among students from different economic, social, and racial or ethnic backgrounds 60.95 53.28 -7.67
9d Helping you cope with your non-academic responsibilities (work, family, etc.) 72.60 68.97 -3.63
9e Providing the support you need to thrive socially 71.27 62.92 -8.359f Providing the financial support you need to afford your
education 49.03 44.03 -5.00
13a1 Frequency: Academic advising/planning40.29 37.08 -3.21
13b1 Frequency: Career counseling61.62 63.62 2.00
Q9 Items: Percent reported Some or Very littleQ13 Items: Percent reported Rarely/Never
12
Longitudinal Analysis - Benchmarks
Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8
actcoll
0.3602108
0.3652025
0.3738425
0.3783556
0.3795649
0.390723
0.3865731
0.3945754
stueff
0.4879008
0.4618854
0.4694364
0.4792259
0.4696756
0.4976116
0.4815078
0.502262699999
999
ac-chall
0.5796767
0.5821051
0.5909778
0.6060924
0.5865871
0.6011233
0.6042203
0.6033747
stu-fac
0.4019986
0.4184133
0.4387495
0.4275732
0.4187017
0.4506208
0.4541376
0.4732149
support
0.4697019
0.4747556
0.513966
0.52251 0.5031957
0.4972433
0.4906533
0.4880942
0.33
0.43
0.53
0.63
Raw Benchmark Scores Over Eight Years
Raw
Sco
re
13
Longitudinal Analysis: Item-level
Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 8
4k. Email 12.15 4.39 4.32 5 7.81 3.23 2.43 1.76
4l. Facgrade 6.03 3.15 6.17 8.37 7.62 6.35 5.2 7.29
4m. Facplans 24.24 26.52 24.72 24.74 24.63 25.4 28.39 24.95
4n. Facideas 35.45 35.76 38.69 38.44 41.05 37.19 37.23 31.36
4o. Facfeed 8.39 8.22 6.28 8.05 6.75 7.48 6.17 6.88
4q. Facoth 67.35 65.89 60.97 63.88 64.71 64.78 69.75 63.18
5.00
25.00
45.00
65.00
Student-Faculty Interaction Benchmark
Pe
rce
nt
Re
sp
on
din
g
Ne
ve
r D
oin
g
14
Longitudinal Analysis: Item-level
Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7 Year 80
10
20
30
40
50
60
70
80
Academic Challenge Benchamrk
5b. Analyze5c. Synthesz5d. Evaluate5e. Applying5f. Perform9a. Envschol4p. Workhard6a. Readasgn6c. Writeany7. exams
Per
cen
t R
esp
on
din
g N
egat
ivel
y
Extending this work
16
Additional work with CCSSE data
AIR Forum 2013 – an extension of this presentation in a discussion forum.
Center Work – Initiative on Promising Practices. – Strengthening the Role of Part-Time Faculty– Improving outcomes for Men of Color – Validation of SENSE survey – Revalidation of CCSSE survey
What can you do at your colleges?
TAIR 2013Galveston, TX
E. Michael Bohlig
Sr. Research Associate
Center for Community College Student Engagement