developing an effective performance-based data system
TRANSCRIPT
Developing an Effective Performance-Based Data System
Colorado State University – Pueblo Victoria Marquesen
System Criteria
Developing System Criteria
• Allow administrators and faculty to focus on their essential mission of ensuring student learning
• Track student progress and identifies appropriate times for intervention and/or support
• Provide summative information on student proficiency on all performance-based standards
• Ensure ongoing program improvement by providing reliable and valid information on the program’s successes and weaknesses related to student performance
• Support analysis of quantitative and qualitative information
• Provide flexibility as program changes and reporting requirements change
• Include valid information
• Provide support for data analysis
TEIMS System Overview https://secure.colostate-
pueblo.edu/TEIMS/Default.aspx?ReturnUrl=%2fteims%2fHome.aspx https://estarportfolio.colostate-pueblo.edu/
• Web-based tool which can be accessed from anywhere
• Relational database driven
• Microsoft .NET application – Same look and feel as MS Office products
Click here to view TEIMS
Student Profile
Demographic info
GPA
Admission Checklist
CSU-P Central
Database Info
Employment
Reports
Advising & Support
Advising Contacts
Advising Worksheet
Long Term Plan
Substitution Request
Early Alert
Support Plan
Reports
Assessment
Field Experience
- online
Portfolio Evaluation
- online
Formal Test Scores
Reports
< Grades
Portfolio
Standards Based
Artifacts
Student Reflections
Communication
Center
Assessment System
Reports
Student
Monitoring
Program
Assessment
Student Profile
• Demographic information
• GPA
• Admission Checklist
• CSU-P Central Database Information
• Employment
Reports
Advising & Support
• Advising Contacts
• Advising Worksheet
• Long Term Plan
• Substitution Request
• Early Alert
• Support Plans
• Standard Reports
Assessment
• Field Experience
–Online Assessment
• Portfolio Evaluation
–Online Assessment
• Formal Test Scores
• Standard Reports
ePortfolio • Specific Templates and Customization Available for
Assessment and Learning • Standards Based Artifacts
– Teacher Work Sample, Lesson Plans, Assignments, Video Clips
• Student Reflections • Communication Center
– Message System (Peer + Faculty) – Review System
• Informal Review Process
• Assessment System – Available to Student – Faculty View & Assess Online
Examples of Data Analysis
0
10
20
30
40
50
60
70
80
90
100
All Min All Min All Min All Min All Min All Min
Group 1 Group 2 Group 3 Group 1 Group 2 Group 1
Year 1 Year 2 Year 3
6.4
9
11.1
1
13.8
3
4.7
6
16.2
8
3.5
7
51.9
5
55.5
6 68.0
9
57.0
0 75.3
2
85.1
9
79.2
2
77.7
8
77.6
5
95.2
4
69.7
7
78.5
7
31.1
7
33.3
3
17.0
2
33.3
3
6.4
9
0.00P
erc
en
t S
tud
en
ts
% Continuing in the Program % S Teachers
Group 1Group 2
Group 3
Removed
Counseled Out
Changed Program
Left U./Unknown
Total
14 1412
40
4 6 9
19
64
1
112
0 1 324
17
0
5
10
15
20
25
30
35
40
Nu
mb
er
of
Stu
de
nts
No
n-R
eta
ine
d
Removed
Counseled Out
Changed Program
Left U./Unknown
Total
0%
10%
20%
30%
40%
50%
60%
70%
80%
31% 24% 22%
31% 25% 25%
30% 27%
Percent of minority and non-minority students admitted to TEP, 2003-2010
TEP Non-minority TEP Minority
3.00
3.10
3.20
3.30
3.40
3.50
Art EL ED English Math Music PE Science SS Spanish ALL
Mea
n C
um
ula
tive
GP
A
Mean cumulative GPA for students from various licensure programs at admission to education, 2003-
2010
425
430
435
440
445
450
455
460
465
Art EL ED English Math Music PE Science SS Spanish ALL Norm
Me
an S
cale
d S
core
Figure SM4. Mean Overall Standard Score on the Academic Profile/MAPP for students from various licensure programs at
admission to education, 2003-2010
430
435
440
445
450
Nat
ive
Am
eri
can
His
pan
ic
Afr
ican
Am
eri
can
Asi
an
Min
ori
ty
No
n-M
ino
rity
Minority Students All Admitted Students
444.00 441.95
438.38
447.14
442.01
448.05
Me
an S
cale
d O
vera
ll Sc
ore
Figure SM12. Mean overall scaled score for students admitted to education from various ethnic groups, 2003-2010
0%
20%
40%
60%
80%
100%
Lang. Arts
Math S. Studies
Science Human. Well & PE
4% 15%
24% 16% 12% 11%
23% 14%
30% 31%
19% 24%
44% 34%
23% 33%
28% 34%
29% 36%
24% 20%
41% 31%
Pe
rce
nt
Stu
de
nts
Test Subareas
Percent of All El. Education Takers Who Scored Poor, Limited, Satisfactory, and Strong on the
PLACE Exam, 2003-2010
Strong Satisfactory Limited Poor
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2003-2004
2004-2005
2005-2006
2006-2007
2007-2008
2008-2009
2009-2010
2003-2010
Perc
ent P
assi
ng
Licensure Areas
Figure SM12. Percent of Hispanic and All First Takers passing licensure exams, 2003-2010
Hispanic All First Time Takers
1.1 1.2 1.3 1.4 1.5 ALL Mean Rating 3.64 3.62 3.61 3.70 3.67
N=553 % < Proficient (1s &2s) 2.4% 2.9% 2.4% 1.8% 1.5%
% Proficient (3s) 65% 71% 60% 51% 57%
% Advanced (4s) 32% 26% 38% 47% 41%
El Ed (N = 312) Mean Rating 3.69 3.68 3.66 3.79 3.72
T-test (EL ED & SEC) Probability <0.0001 <0.00001 <0.0002 <1E-11 <0.00007
T-test (EL ED & K-12) Probability <0.04 <0.02 <0.02 <1E-11 <0.02
Secondary (111) Mean Rating 3.52 3.48 3.49 3.49 3.55
K-12 (N=130) Mean Rating 3.63 3.60 3.58 3.68 3.65
T-test (SEC & K-12) Probability <0.02 <0.01 <0.05 <0.0003 <0.03
2004-2005 Mean Rating 3.64 3.60 3.64 3.66 3.66
2005-2006 Mean Rating 3.69 3.66 3.69 3.73 3.71
2006-2007 Mean Rating 3.69 3.69 3.64 3.74 3.70
2007-2008 Mean Rating 3.63 3.62 3.55 3.66 3.63
2008-2009 Mean Rating 3.61 3.56 3.51 3.73 3.66
2009-2010 Mean Rating 3.59 3.59 3.63 3.70 3.67
Male Mean Rating 3.57 3.56 3.57 3.63 3.62
Female Mean Rating 3.67 3.64 3.62 3.73 3.69
T-test (Gender) Probability <0.01 <0.01 <0.10 <0.01 <0.06 Minority Mean Rating 3.65 3.64 3.62 3.75 3.70
Non-Min. Mean Rating 3.64 3.61 3.60 3.69 3.66
T-test (Ethnicity) Probability <0.39 <0.25 <0.38 <0.07 <0.19
Table TS1. Information on performance on standards 1.1-1.5 for all student teachers, 2004-2010, disaggregated by teaching level, years, and by gender and ethnicity
3.00
3.25
3.50
3.75
4.00
4.25
4.50
4.75
All Secondary
4.13 4.03
4.15
4.45 4.34
4.26
4.46 4.55
Me
an R
atin
g
Figure SM4. Mean ratings by graduates after one year of teaching on standards 2.10 and 2.11
2.10 2.11
3.00
3.25
3.50
3.75
4.00
4.25
4.50
4.75
Grad Sup Grad Sup Grad Grad Sup Sup
All El Ed Sec K-12 MS HS
4.34
3.74
4.26
3.67
4.46 4.55
4.00
3.76
Me
an R
atin
g
Figure SM3. Mean ratings by program completers (Grad) and supervisors (Sup) on standard 2.11
3
3.5
4
3.1
/6.1
3.2
3.3
/6.1
3.4
/8.2
3.5
/8.2
3.6
/8.2
3.7
3.8
/6.2
3.9
/6.3
3.1
/6.5
3.1
1/6
.4
3.1
2/6
.7
3.1
3
mean
4.1
/3.2
4.2
/3.3
4.3
/3.4
4.4
4.5
/3.5
4.6
4.7
4.8
4.9
/7.4
4.1
0/6
.6
4.1
1/3
.7
4.1
2/3
.8
mean
Goal 3: Diversity Goal 4: Assessment
Standards (CSU-P Standards/CO Elements)
Mea
n S
up
erv
iso
r R
ati
ng
s
High Need Non-High Need
Table 4b. Mean percent increase in CSAP scores in Reading, Writing, and Math CSAPs for
students in classrooms with student teachers vs. mean district CSAP scores for students at the
same grade levels between 2002-2003 and 2003-2004; scores are for students scoring
proficient and advanced.
Reading Grades 3-6
Writing Grades 3-6
Math Grades 3-6
Mean % Free & Reduced Lunch
ST
Classrooms
District ST
Classrooms
District ST
Classrooms
District ST
Classrooms
District
8.09% 14.98% 3.66% -1.00% 15.89% 4.33% 61.32% 51.65%
Table 3. Percent of student teacher work samples (TWSs) that demonstrated high levels of
K-12 student achievement, Spring 2010 and Spring 201 student teachers
Evaluation Criteria
% Teacher Work
Samples
Spring
200
Spring
2011
% TWSs in which K-12 students
achieved at high levels on post-test
>90% Students 20 20
80-90% 25 55
70-79% 35 20
<70% 20 5
% TWSs in which K-12 students
improved significantly
>90% Students 15 20
80-90% 45 45
70-79% 25 25
<70% 15 10
% TWSs in which data were disaggregated for all groups 60 75
% TWSs in which disaggregated
groups achieved*
>90% Students 0 7
80-90% 13 0
70-79% 74 93
<70% 13 0
*Only those disaggregating data were included
TEIMS Lessons Learned
• Our success has been due to a Collaborative approach . . . Progressive delivery technique . . . Ability to prioritize the project as “Building Blocks”
• Input is ESSENTIAL from different stakeholders and constituencies
• Don’t rush development – think about what the system needs to do
• Make IT your friends • Consider and plan for training users