1 reading your accountability reports u-pass ayp amao judy park, assessment & accountability...

1

Reading your Accountability Reports

U-PASSAYP

AMAO Judy Park, Assessment & Accountability Director

Russell Klein, Results CoordinatorUtah State Office of Education

August 2006

2

Reading your Accountability Reports

U-PASS, AMAO, AYP

August 18 – Sevier D.O., 180 E 600 N, Richfield

August 22 – Granite D.O., 2500 S. State St., Salt Lake

August 23 – Weber D.O., 5320 S. Adams Ave. Pkwy, Ogden

August 25 – Nebo Learning Center, 570 S. Main St., Springville

3

Utah Performance Assessment System for Students

U-PASS U-PASS Accountability PlanAccountability Plan

4

U-PASS

● State Accountability

● Required by State Law

● School level report

● Based on multiple assessments

● Includes attendance & graduation rate

● 2006 – schools without a 12th grade only

● 2007 – all schools

5

U-PASSAccountability Plan

● Student Participation on Student Participation on AssessmentsAssessments

● Status ScoresStatus Scores Percentage of students proficient on the Percentage of students proficient on the

collection of assessments and other collection of assessments and other indicatorsindicators

● Progress ScoresProgress Scores Longitudinal measure based on Longitudinal measure based on

comparing the achievement levels of the comparing the achievement levels of the same student from one year to the nextsame student from one year to the next

6

Achieved State Level of Achieved State Level of PerformancePerformance

Total School Total School

& &

SubgroupSubgroup

Low Low progressprogress

Medium Medium progressprogress

High High progressprogress

Acceptable Acceptable statusstatus

Unacceptable Unacceptable statusstatus Needs

Assistance

7

U-PASSAccountability Plan

● Student Participation Student Participation 95%95%

● Acceptable Status ScoreAcceptable Status Score 80%80%

● Progress ScoreProgress Score Low 0 – 189Low 0 – 189 Medium Medium 190190 - 214 - 214 High 215 - 400High 215 - 400

8

Progress Value Table

Year 1

Level

Year 2 Level

1a 1b 2a 2b 3 4

1a 0 225 350 375 375 375

1b 0 125 225 350 375 375

2a 0 50 150 225 325 350

2b 0 0 75 175 275 325

3 0 0 0 100 200 275

4 0 0 0 0 125 225

196.0

9

Schools will be identified as:

● Achieved State Level of PerformanceAchieved State Level of Performance Participation is 95% Participation is 95% andand

Total School Total School ANDAND Subgroup Subgroup Status is Status is acceptable (80%) acceptable (80%) oror progress is medium progress is medium (190) or high (190) or high

● Needs assistanceNeeds assistance Participation is less than 95% Participation is less than 95% oror

Total SchoolTotal School or or SubgroupSubgroup Status is not Status is not acceptable acceptable andand progress is low progress is low

10

Schools without a 12th gradeStatus and Progress Score

●Language Arts (35%)ELA CRT (35%)

Or

ELA CRT (30%)

& DWA 5%

●Math (35%)Math CRT

●Science (20%)Science CRT

●Attendance (10%)

11

U-PASS Accountability Plan

● Aggregate Subgroup Accountability Every student who belongs to a subgroup other

than white Individual student proficiencies are added

together and divided by the number of students to determine the subgroup proficiency level

● Any individual subgroup not meeting proficiency will appear on the front page of the report

12

U-PASS Accountability Plan

● Subgroup Status Accountability.

Allows greater accountability. Students in a subgroup with less than 10 are now included in the

aggregate calculation.

Students are more fairly represented. Each student only counts once, regardless of the number of subgroups

in which the student qualifies.

Increased reliability. We are evaluating a larger group. More students and more schools are represented.

Increased validity. We do not overweight small populations. This lifts the burden of one subgroup being the sole determinant of

unacceptable status of the school.

13

The 2006 U-PASS Report

● Web-based report format

● An increased amount of information available on each school

● Allows for “drill-down” information

14

15

1

16

2

17

3

18

Must achieve both criteria

Final Result

19

SubgroupsNeedingAssistance

20

Status“Drill Down”

21

Click on Grade Name for

Instant Graph

22

Second Level “Drill Down”

Click on Grade Score for Further

Drill Down

23

Third Level “Drill Down” Boxes

expand on ‘mouseover’

24

Progress“Drill Down”

coming in October

25

Additional Reports

26

U-PASS

● Additional Information: NRT ACT/SAT AP Concurrent Enrollment School Summary Information Student Summary Information Percent of student reading on grade level

(reported for grades 1-10) Dropout Rate (disaggregated by ’08) Disciplinary Action Course Taking Patterns and Trends

27

28

29

30

AYP Reports

31

Schools with a 12th Grade

● Assessment Drill Downs

● Additional Information

32

Questions?

33

AYP School ReportsAYP School Reports

For 2006For 2006

34

AYP

● Federal Accountability

● Required by Federal NCLB Title I Law

● School, District & State level Reports

● Based on Language Arts CRT & Math CRT

● Includes attendance & graduation rate

35

AYP 2006AYP 2006Level 1Level 1

● Three parts:Three parts:1.1. Participation in testingParticipation in testing

2.2. Academic achievementAcademic achievement

3.3. School attendance / graduationSchool attendance / graduation

● All-or-nothing conceptAll-or-nothing concept

36

1

2

3

37

AYP 2006AYP 2006Level 1Level 1

● Did the school make AYP?Did the school make AYP?

● Math vs. Language Arts?Math vs. Language Arts?

● AMO (Annual Measurable Objectives) AMO (Annual Measurable Objectives) same as last yearsame as last year

● Consequences only for Title I schoolsConsequences only for Title I schools

38

1

2

3 3

4

39

40

AYP 2006AYP 2006Level 1Level 1

● WHICH STUDENTS ARE IN THE AYP WHICH STUDENTS ARE IN THE AYP SCHOOL REPORTS?SCHOOL REPORTS?

41

Students in AYP & U-PASS Students in AYP & U-PASS School ReportsSchool Reports

● Two large data collectionsTwo large data collections 1) End-of-Year Clearinghouse 1) End-of-Year Clearinghouse 2) Core CRT Files2) Core CRT Files

● These Two Files Need To These Two Files Need To MatchMatch To Be To Be On AYP & U-PASS School Reports!On AYP & U-PASS School Reports!

● 6 Fields for Matching: District#, 6 Fields for Matching: District#, School#, Last Name, First Name, Date School#, Last Name, First Name, Date of Birth, Genderof Birth, Gender

● As of Spring ’06 – SSID!As of Spring ’06 – SSID!

42

Full Academic YearFull Academic Year

Used for Core CRT tests to be included - Used for Core CRT tests to be included - Full Academic year 2006-Full Academic year 2006- 1. Student who was in membership on Sept. 15th 1. Student who was in membership on Sept. 15th 2. Student membership >=1602. Student membership >=1603. Student membership = School Membership + 3. Student membership = School Membership + Special Ed Membership (Self-Contained)+ Special Ed Membership (Self-Contained)+ YIC membershipYIC membership

43

Math Core CRTsMath Core CRTs

Math CRTsMath CRTs

Grade 3 – 8 Mathematics: Grade 3 – 8 Mathematics:

64% Proficient is the Status Goal (AMO) for 200664% Proficient is the Status Goal (AMO) for 2006

Grades 10-12 Algebra & Geometry:Grades 10-12 Algebra & Geometry:

Applied Math 1 & 2 – not includedApplied Math 1 & 2 – not included47% Proficient is the Status Goal – ONLY students in 47% Proficient is the Status Goal – ONLY students in Grades 10,11,12 who took the Core CRTs in either Grades 10,11,12 who took the Core CRTs in either Algebra or Geometry.Algebra or Geometry.

Schools with a 12Schools with a 12thth grade class get this AMO/Goal. grade class get this AMO/Goal.

44

Language Arts Core CRTsLanguage Arts Core CRTs

Language ArtsLanguage ArtsGrades 3 – 8 Language Arts: 71% Proficient.Grades 3 – 8 Language Arts: 71% Proficient.

This is the Status Goal (AMO) for 2006.This is the Status Goal (AMO) for 2006.

Grade 10 Language Arts: 70% Proficient.Grade 10 Language Arts: 70% Proficient.

No other language arts Core CRTs are being No other language arts Core CRTs are being

used for AYP for schools with a 12used for AYP for schools with a 12 thth grade class! grade class!

45

AYP 2006AYP 2006Level 1Level 1

● Questions?Questions?

46

AYP 2006AYP 2006Level 2Level 2

● Status vs. Safe Harbor (Improvement)Status vs. Safe Harbor (Improvement)

● Safe HarborSafe Harbor 10% reduction in % of students 10% reduction in % of students notnot proficient proficient

47

AYP 2006AYP 2006Safe Harbor (Improvement)Safe Harbor (Improvement)

ProficientNot Proficient Status

48

AYP 2006AYP 2006Safe Harbor (Improvement)Safe Harbor (Improvement)

Did not make AYP Made AYP

ProficientNot Proficient Status

49

AYP 2006AYP 2006Safe Harbor (Improvement)Safe Harbor (Improvement)

Shows Improvement

Does not show improvement

Imp

rovem

ent

Imp

rovem

ent

50

AYP 2006AYP 2006Safe Harbor (Improvement)Safe Harbor (Improvement)

Shows Improvement Made AYP

Does not show improvement

Did not make AYP

Imp

rovem

ent

Imp

rovem

ent

51

AYP 2006AYP 2006Safe Harbor (Improvement)Safe Harbor (Improvement)

ProficientNot Proficient

Status

Does not show

im

provement

Show

s Im

provement

Good

for

Status

Good for ImprovementImp

rovem

ent

Imp

rovem

ent

52

AYP 2006AYP 2006Safe Harbor (Improvement)Safe Harbor (Improvement)

ProficientNot Proficient

Status

Does not show

im

provement

Show

s Im

provement

Imp

rovem

ent

Imp

rovem

ent

Made AYP

Made AYP

Made AYP

53

Status

54

Safe Harbor

55

AYP 2006AYP 2006Level 2Level 2

● Safe HarborSafe Harbor Additional Indicator is only used when Safe Additional Indicator is only used when Safe

Harbor calculation is neededHarbor calculation is needed

56

AYP 2005AYP 2005Level 2Level 2

● Confidence Interval on StatusConfidence Interval on Status 99%99%

● Confidence Interval on Safe HarborConfidence Interval on Safe Harbor 75%75%

● Participation & Additional IndicatorParticipation & Additional Indicator N/AN/A

57

ConfidenceConfidenceIntervalInterval

● Voting – exit pollsVoting – exit polls

58

Confidence Interval on Status

59

Confidence Interval on Safe Harbor

60

AYP 2006AYP 2006Level 2Level 2

● N size – 10 vs. 40N size – 10 vs. 40

● Small N = Automatic YESSmall N = Automatic YES

61

AYP 2006AYP 2006Level 2Level 2

● N=40 participation, attendance, and N=40 participation, attendance, and graduationgraduation

● N=10 academic achievementN=10 academic achievement

62

63

64

65

AYP 2006AYP 2006

● Graduation rate – 85.7%Graduation rate – 85.7% One year delayed (have until Sep 1)One year delayed (have until Sep 1) 05 data from warehouse05 data from warehouse 04, 03 data from USOE statistical files04, 03 data from USOE statistical files

● Quality of data has improvedQuality of data has improved

● 40 ways to make (or not make) AYP40 ways to make (or not make) AYP

66

67

40 ways to make (or not make) AYP

Tim

es

10

68

“Elementary”

69

“Elementary”

70

“District”

71

Available on the secure

FTP site

72

AYP 2006AYP 2006Level 2Level 2

● Questions?Questions?

73

AYP 2006AYP 2006Level 3Level 3

● Definition of “safe harbor”Definition of “safe harbor”

● Definition of “participation”Definition of “participation”

● Definition “confidence intervals”Definition “confidence intervals”

● Definition of “graduation rate”Definition of “graduation rate”

● UAAUAA

74

AYP 2006AYP 2006Level 3Level 3

● Questions?Questions?

75

AYP 2006AYP 2006Did your school “make it”?Did your school “make it”?

● Participation – DID YOU MEET IT?Participation – DID YOU MEET IT?

● Participation – Is the N size less than 40?Participation – Is the N size less than 40?

● Did they have 95% participation?Did they have 95% participation?

● Did they average 95% participation over Did they average 95% participation over the last three (or two) years?the last three (or two) years?

76

AYP 2006AYP 2006Did your school “make it”?Did your school “make it”?

● Status – Is the subgroup size less than 10?Status – Is the subgroup size less than 10?

● Status – Is the % proficient enough?Status – Is the % proficient enough?

● Status – If the % proficient is not enough – Status – If the % proficient is not enough – does the confidence interval change it?does the confidence interval change it?

77

AYP 2006AYP 2006Did your school “make it”?Did your school “make it”?

● Safe Harbor – Is there a 10% reduction in Safe Harbor – Is there a 10% reduction in students students notnot proficient? proficient?

● Safe Harbor – Is the % reduction of Safe Harbor – Is the % reduction of students students notnot proficient within the proficient within the confidence interval?confidence interval?

78

AYP 2006AYP 2006Did your school “make it”?Did your school “make it”?

● Attendance/Graduation Rate – Is the Attendance/Graduation Rate – Is the subgroup less than 40?subgroup less than 40?

● Attendance/Graduation Rate – Did the Attendance/Graduation Rate – Did the school meet 93% attendance rate or 85.7 school meet 93% attendance rate or 85.7 graduation rategraduation rate

● Attendance/Graduation Rate – has there Attendance/Graduation Rate – has there been an increase from the previous year?been an increase from the previous year?

79

Level Three ClarificationsLevel Three Clarifications

80

AYP 2006AYP 2006Level 3Level 3

● Participation Category – Students enrolled Participation Category – Students enrolled on test date.on test date.

● Participation within tests – Students you’ve Participation within tests – Students you’ve had for the full academic year. (subset of had for the full academic year. (subset of “category” set)“category” set)

● Participation – grad/attend. Is ALL Participation – grad/attend. Is ALL students…students…

81

95% Participation95% Participation

For 2004 - 2006,For 2004 - 2006, Missing tests will be handled Missing tests will be handled by the testing process. Same as 2003 by the testing process. Same as 2003 with the with the exceptionexception that participation percentage can be that participation percentage can be averagedaveraged over the previous two (or three) over the previous two (or three) years, if necessary. years, if necessary. IfIf this does not help, the this does not help, the current year participation percentage will be current year participation percentage will be usedused

82

LEP Subgroup DefinitionLEP Subgroup Definition

2004 - 2006, LEP students will count in 2004 - 2006, LEP students will count in the LEP subgroup for two years past their the LEP subgroup for two years past their exit of the ELL program. This is exit of the ELL program. This is determined by the lep_exit_date in the determined by the lep_exit_date in the student_lep table of the warehouse.student_lep table of the warehouse.

83

Additional Indicator AYPAdditional Indicator AYP

Attendance 93%Attendance 93%

The goal for attendance will be set at 93%, consistent with the The goal for attendance will be set at 93%, consistent with the additional indicator goal for elementary and middle schools. additional indicator goal for elementary and middle schools. Schools must meet this Schools must meet this goal or improve from the previous goal or improve from the previous year. Improvement is defined as year. Improvement is defined as any increaseany increase in this area. in this area.

The attendance rate is the sum of days in attendance divided The attendance rate is the sum of days in attendance divided by the sum of days in membership for all students in the unit of by the sum of days in membership for all students in the unit of accountability who were in membership for at least eleven (11) accountability who were in membership for at least eleven (11) days and whose membership is equal to or greater than days and whose membership is equal to or greater than attendance (180 days).attendance (180 days). Membership includes self-contained SCRAM and YIC systems.

84

Additional Indicator AYPAdditional Indicator AYP

Graduation Rate 85.7%Graduation Rate 85.7%

Grad Rate is Yes, if Grad Rate is >= 85.7Grad Rate is Yes, if Grad Rate is >= 85.7The cohort graduation rates by school according to the ED approved The cohort graduation rates by school according to the ED approved

algorithm in our AYP plan. Here is the algorithm:algorithm in our AYP plan. Here is the algorithm:

=IF(G2<40,"too few",IF(L2>0.857,"pass",IF(L2>W2,"pass","fail")))=IF(G2<40,"too few",IF(L2>0.857,"pass",IF(L2>W2,"pass","fail")))

In English:In English:

If the cohort [G] contains less then 40 students, there are too few students If the cohort [G] contains less then 40 students, there are too few students to make a reliable decision about the graduation rate for the unit of to make a reliable decision about the graduation rate for the unit of analysis (school or district). = NAanalysis (school or district). = NA

If the cohort is equal to or greater than 40 students and the 2002 If the cohort is equal to or greater than 40 students and the 2002 graduation rate [L] is greater than (the constant) 85.7%, the unit graduation rate [L] is greater than (the constant) 85.7%, the unit passes. = Yespasses. = Yes

If the 2002 graduation rate is less than 85.7% but greater than the 2001 If the 2002 graduation rate is less than 85.7% but greater than the 2001 graduation rate [W], the unit passes. = Yesgraduation rate [W], the unit passes. = Yes

In all other cases, the unit fails. = NoIn all other cases, the unit fails. = No

85

UAA TestingUAA TestingSpecial Ed Alternate AssessmentSpecial Ed Alternate Assessment

● Grade level is derived based on birthdate.Grade level is derived based on birthdate.● Core Test Id and grade level will be used to Core Test Id and grade level will be used to

determine which Actual Test ID to assign a determine which Actual Test ID to assign a student to. LA and grade level will correspond student to. LA and grade level will correspond to an Actual Language (Reading) class for that to an Actual Language (Reading) class for that student. MA and grade level will correspond to a student. MA and grade level will correspond to a math course.math course.

● UAA students whose birthdate is in 9-12th UAA students whose birthdate is in 9-12th grades will be added to GE0812R Geometry. grades will be added to GE0812R Geometry. They are part of High School Math AYP!They are part of High School Math AYP!

● UAA in 7-8th grade will be added to 7th grade UAA in 7-8th grade will be added to 7th grade math.math.

86

Out-of-Level TestingOut-of-Level Testing

● Not allowedNot allowed No for participation No for academic achievement

● Math Levels:Math Levels: 66thth – Math 6; 7 – Math 6; 7thth – Math 7; 8 – Math 7; 8thth – Pre-Alg. Anything – Pre-Alg. Anything

lower for the grades is Out-of-Level and considered Not lower for the grades is Out-of-Level and considered Not Proficient.Proficient.

87

District AYP Reports

● Previous District AYP Report

Grades 3 – 8 only

88

District AYP Reports● 2006 District AYP Report

Grades 3 – 8 Grades 10 – 12

Business RuleAn LEA will receive a report based on both 3-8th and 10-12th grade AMO's.  An LEA must fail at both the 3-8 AND the 10-12 grade levels to fail AYP.  (Either subject)

89

District AYP Reports● 2006 District AYP Report

Grades 3 – 8 Grades 10 – 12

● Program Improvement

Business RuleAn LEA is identified for improvement only when it misses AYP in the same subject and in all grade spans for two consecutive years, or the other academic indicator in all grade spans for two consecutive years.

(It is possible for an LEA to not pass AYP and yet not be put into 'Improvement' because 'Improvement' considers 'subject' in addition to grade levels whereas 'not making AYP' does not. 

90

AMAO Reports

91

AMAO

● Federal Accountability

● Required by NCLB Title III Law

● District & State Level Reports

● Based on IPT progress IPT proficiency AYP for ELL

92

93

94

● Enrollment Information

● AMAO #1 – Annual increases in the number or percentage of children making progress in learning English

● AMAO #2 – Annual increases in the number or percentage of children attaining English proficiency

● AMAO #3 – AYP for ELL

95

ELL Enrollment

● Grade Span

● Language Proficiency Category

● 2005 ELL Enrollment

● 2006 ELL Enrollment

96

ELL Enrollment

Grade Span

The grade span used for AMAO targets is grades K-6 and 7-12.

97

ELL EnrollmentLanguage Proficiency Category● A - Non-English proficient in speaking,

reading and writing

● B - Limited English proficient (student is non in at least one of the language modalities of speaking, reading and writing or at least limited in two of the language modalities of speaking, reading and writing)

● C - Fluent in two of the language modalities and limited in one

98

ELL EnrollmentLanguage Proficiency Category● D - Monitored student for minimum of two

years for English proficiency (student is fluent in all language modalities).

● E - Exited, Former LEP student who is fully proficient in English speaking, reading and writing and has been exited from an alternative language services program and fully assimilated into the mainstream.

● O - Identified as LEP but opted out of services by parent.

99

ELL Enrollment

2005 ELL Enrollment

The number of ELL students enrolled in 2005 for the full academic year. For E students, only those students who have exited within the last 3 years are included.

100

ELL Enrollment

2006 ELL Enrollment

The number of ELL students enrolled in 2006 for the full academic year. For E students, only those students who have exited within the last 3 years are included.

101

AMAO Requirement #1

● Grade Span

● % of 2005 students making progress learning English in 2006 (AMAO 80%)

● District made the AMAO targets for making progress learning English

102

AMAO Requirement #1

Grade Span

The grade span used for AMAO targets is grades K-6 and 7-12.

103

AMAO Requirement #1

% of 2005 Students making progress

learning English in 2006

The percentage of ELL students who made progress in one or more domains (speaking, reading, writing). Progress is defined as moving from non speaking to limited or limited to fluent.

104

AMAO Requirement #1

District made the AMAO targets for

making progress learning English

This column is Yes if the district made the AMAO target of 80% of students showing progress in one or more domains (speaking, reading, writing).

105

AMAO Requirement #2

● % of 2005 students Attaining English Proficiency in 2006 K-6, 16.2% 7-12, 21.6%

● District made the AMAO target for Attaining English Proficiency

106

AMAO Requirement #2

% of 2005 students Attaining English

Proficiency in 2006 (K-6, 16.2%) (7-12,

21.6%)

Percentage of students enrolled for both the 2005 and 2006 full academic year, with a language proficiency of D or E in 2006.

107

AMAO Requirement #2District made the AMAO target forAttaining English Proficiency

The column is Yes if the district made the AMAO target.

For grade ranges K-6, 16.2% of the students are at the D or E level.

For grade ranges 7-12, 21.6% of the students are at the D or E level.

108

AMAO Requirement #3● Grade Span

● % Participation of ELL for AYP Language Arts

● District made AYP for the ELL subgroup in Language Arts

● % participation of ELL for AYP Math

● District made AYP for the ELL subgroup in Math

109

AMAO Requirement #3Grade Span

The grade span used for AYP targets is grades 3-8 and grades 10-12.

% Participation of ELL for AYP LanguageArts

The percentage of ELL students who participated in the language arts CRT.

110

AMAO Requirement #3District made AYP for the ELL subgroup

in Language Arts

The ELL subgroup attained 95% participation and for the 3-8 grade span attained 71% of the students proficient on the language arts CRT; and for the 10-12 grade span attained 70% of the students proficient on the language arts CRT.

111

AMAO Requirement #3% Participation of ELL for AYP Math

The percentage of ELL students who participated in the Math CRT.

112

AMAO Requirement #3District Made AYP for the ELL subgroup

in Math

The ELL subgroup attained 95% participation and for the 3-8 grade span attained 64% of the students proficient on the Math CRT; and for the 10-12 grade span attained 47% of the students proficient on the Math CRT.

113

District Made AMAOs

● The District is “Yes” if there is a “Yes” in every column. One “No” in any column will result in a “No” for District Made AMAOs.

114

Appeals1.An error in the aggregating and merging of

data and/or a calculation error, or other errors that directly affects school or district status.

2.Extreme circumstances that affected data and/or test administration that had a significant impact on the school (e.g., flood, fire, natural disaster).

115

Appeal Process● September 11

Deadline for District Superintendent to submit a written appeal request for U-PASS, school AYP, District AYP, or District AMAO stating the reason for the appeal An E-mail response from USOE will confirm receipt of the appeal.

Data and documentation that directly influences the appeal needs to be stored at the district.

116

● September 12-15 USOE reviews appeals and makes changes

to the U-PASS, AYP & AMAO school or district designation.

● September 16 Districts release U-PASS, school AYP, district

AYP & district AMAO to the public.

● September 17 U-PASS, AYP & AMAO reports posted on the

web. Reports for granted appeals, changed from a No to a Yes (by appeal).