data informed decision making

1

Data Informed Decision Making

Dr. Robert Rodosky

Dr. Marco Muñoz

Jefferson County Public Schools (Kentucky)

Presented at the National Evaluation Institute

October 4-6, 2007

2

Presentation Overview

• Wallace Foundation• Leadership Interest Groups (LIG) • Data-Driven Decision Making • Georgia, Kentucky, Michigan, New

Mexico, and Ohio• In the fall of 2006, all principals in

Kentucky were surveyed regarding data access to support their roles as instructional leaders.

3

Legislative Pressures• Demands of No Child Left Behind (NCLB)

legislation—meeting AYPs.• Demands of KERA—meeting goals.• Assessment and accountability that

require knowledge and skills in measurement and data use for school improvement.

• Principals must use data in order to inform their decision-making as instructional leaders.

• Understandings of how data analysis can inform instruction and ultimately improve student achievement.

4

Data Informed Decision Making

Project Purposes

What are the data needed to support/make key decisions that leaders need to make to improve student achievement?

What are the best practices of how leaders need the data analyzed to make it useful/actionable?

Main FocusNon-negotiable decisions that can increase student achievement.

Objectives

1. Identify the sets of data questions asked by school, district, and state leaders.

2. To classify the data questions in ways that help us understand how to make data actionable.

3. To identify best practices in answer to these data questions.

4. To develop tools and training related to these data questions.

5

Survey Development: Helping Principals Make Data Actionable

Modules Examples of StatementsKinds of

Response Formats

Key Sources of Data

AYP math and reading results for the different groups

Importance

Availability

Steps in the Improvement

Process

Help my teachers analyze data to understand achievement gaps

Preparation

Frequency of use

Data-Informed Interventions

Instructional strategies when not achieving academic standards

Importance

Support

Constraints Principals

Face

The data I need are available in a timely fashion

Rate agreement

Level of Accountability

Literacy

I can communicate data effectively in different settings

Rate agreement

6

Data Collection System

State Format Participants

GA2 paper-based surveys: Part 1 and Part 2 given to 2 groups at 4 different times.

249 principals in 27 districts

KY 1 web-based survey 481 principals state wide

MIInterviews & 1 web-based survey

16 principals interviewed; 258 principals responded to the survey

NM 1 web-based survey 79 principals in principal support network in 28 districts

OH 3 web-based surveys 186 principals state wide

Total Participants 1285

7

Survey Domains• Introductory Section: Demographic• Section 1:

• Student demographic data • Student achievement data• Teacher and program data• Climate/perception data• Financial resource data

• Section 2: Principal preparation and use• Section 3: Importance and support • Section 4: Data access and quality

8

Important Data for Principals

State or District

Student Demographic

Data

Student Achievement

Data

Teacher & Program Data

Climate & Perception Data

Financial Resource Data

GeorgiaAbsentee rates; retention rates.

AYP results by subgroup.

Instructional Practice. Student Satisfaction.Level of funding For

my school.

KentuckyStudent needs or

disability information.

Individual proficiency results from state

assessments.

Instruction effectiveness as measured by student

growth scores.

School safety indicators.

Level of funding for my school;

Adequacy of funding for my school.

OhioWhat are the special

needs/disability profiles in my school?

What are the specific achievement gap

profiles for the different groups of students in

my school?

How does our instruction align with the

assessments used for accountability?; What

instructional practices/programs are

my teachers using?

How safe do my students feel in my

school?

How adequate are the technology, facilities, and instructional

materials for my school?

MichiganSpecial

needs/impairment profiles of students.

Student proficiency results on MEAP

broken down by State Standards.

Teachers' instructional effectiveness.

School culture.

Level of funding for my school;

Adequacy of funding for my school.

9

Difficult Questions that Principals Face after Data Analysis

Top 3 Killer Questions

Georgia Kentucky Ohio New Mexico

1

What instructional strategies do I need to implement when students are not

achieving academic standards?

What can I do to help my teachers

become more effective?

What instructional strategies do I need to implement when

my students are not achieving academic

standards?



standards?

2What can I do to help my teachers become

more effective?



What can I do when teachers are not

making reasonable progress?

What instructional strategies will work best with students

with special needs?

3How do I find more financial resources

for my school?

How do I increase the level of my staff's satisfaction with my

school?

What instructional strategies and/or

programs will work best with students

with special needs?

What can I do to help my teachers become

more effective?

10

Support for Data-Informed Decisions

Georgia (n=235)

Kentucky (n=360)

Ohio (n=28)New Mexico

(n=64)

Number 1 Killer Question



What can I do to help my teachers

become more effective?



standards?



standards?

% who answered "Less support than I need or no support" in response to "How

Much Support Do You Get To Answer

This Question?"

22% 24% 36% 40%

11

District and State Supportwhen Using Data

86%

65%

41%

47%

62%60%

10%

0

0.2

0.4

0.6

0.8

1

Georgia (n=248) Kentucky (n=363) Ohio (n=40) New Mexico (n=63)Most principals receive adequate support from the school district in using dataMost principals receive adequate support from the state in understanding what to do with data

Note: Georgia did not include a question on district support. Kentucky asked “I receive adequate support from the district.” and “I receive adequate support from the state.”

12

I have the assistance I need to use data effectively

81%

66%

72%

28%

0%

20%

40%

60%

80%

100%

Georgia (n=249) Kentucky (n=361) Ohio (n=40) New Mexico (n=73)

Completely or Somewhat Agree

13

I would like more support in understanding what to do with teacher effectiveness and program data

78%

86%82%

0%

20%

40%

60%

80%

100%

Kentucky (n=352) Ohio (n=42) New Mexico (n=63)


14

What are the steps in a continuous improvement process that Principals use most frequently?

Note: Number in parentheses are the percent of respondents who stated that they used this step very frequently or frequently during the school year.

3 Most Fequently Used Steps


1

Prioritize which issues are most important to my

school (94%)

Help my staff make decisions that will improve student

achievement (98%)

Develop hypothesis about which issues

are most important to my school (86%)

Prioritize which issues are most important to my

school (94%)

2Gather data in an effective manner

(91%)

Effectively monitor teachers’

implementation of instructional plans

(96%)


achievement (87%)

Help my teachers understand the goals

and standards for student achievement

(93%

3


achievement (89%)

Gather data in an effective manner

(96%)

Help my teachers select interventions

for different subgroups of students (85%)

Gather data in an effective manner

(93%)

15

How prepared were first-year Principals to use these steps in

the Continuous Improvement Process?


Most Frequently Used Step

Prioritize which issues are most important to my school (94%)

Help my staff make decisions that will improve

student achievement

(98%)

Develop hypothesis about which issues are most important to my school (86%)

Prioritize which issues are most important to my school (94%)

% who reported that they were not too prepared or

not at all prepared to use this step

41% 23% 9% 36%

16

What Are The Most Common Constraints That Principals Face In Using Data

Top 3 Constraints


1I have the time

necessary to use data effectively (45%)

I have the time necessary to use

data effectively (58%)

I have the time necessary to use data

effectively (50%)

I have the time necessary to use data

effectively (76%)

2

I have the resouces to act on the needs

revealed by the data (28%)

I have the resouces to act on the needs revealed by the data

(45%)



The data I need are available in a timely

fashion (75%)

3

I have the tools and the training I need to use data effectively

(23%)

The data I need are available in a timely

fashion (30%)

I have the tools and the training I need to use data effectively

(22%)



Note: Number in parentheses are the percent of respondents who somewhat disagreed or disagreed with the statements.

17

The data I need are available in a timely fashion

90%

69%

92%

25%

0%

20%

40%

60%

80%

100%

Georgia (249) Kentucky (n=363) Ohio (n=40) New Mexico (n=63)


18

My Preparation/Certification Program prepared me to deal effectively with data

43%

54%

38%

28%

0%

20%

40%

60%

80%

100%



19

I can communicate data effectively in Political, Professional, and Community Settings

89%92%

89%

74%

0%

20%

40%

60%

80%

100%



20

JCPS StudyElementary, Middle, and High School Principals

21

JCPS Survey Procedures

• Web-based survey

• Population: JCPS principals

(N = 133)

• Response rate: 55%

(N = 73)

• Respondents by school level:• 39 elementary schools• 19 middle schools• 15 high schools

22

Section 1: Demographic Data

• Importance vs. Availability of Data:

• Race (100% vs. 100%)

• Absentee rates (100% vs. 100%)

• ECE information (100% vs. 95.4%)

23

Section 1: Student Achievement Data


• Individual proficiency results from state assessment (98.4% vs. 98.4%)

• AYP results by subgroup (96.8% vs. 95.0%)

• Student results on other district administered tests (93.7% vs. 93.3%)

24

Section 1: Teacher and Program Data


• Instructional practices (100% vs. 91.9%)

• Instructional effectiveness as measured by student growth scores (100% vs. 83.9%)

• Professional development effectiveness (95.2% vs. 72.6%)

25

Section 1: School Climate Data


• Teacher satisfaction with school (100% vs. 95.2%)

• Student satisfaction with school (100% vs. 96.8%)

• School safety indicators (100% vs. 96.8%)

26

Section 1: Financial Resource Data


• Level of funding for school (100% vs. 91.9%)

• Adequacy of funding for school (100% vs. 77.4%)

• Availability of discretionary funds (100% vs. 75.4%)

27

Section 2: Preparation vs. Frequency of Use

• Principal Preparation:• Rate how prepared you were –as a first

year principal- for using each of these data steps to make decisions

• Frequency of Use:• Rate how frequently you use each of these

data steps in your current work as principal

28

Principal Preparation vs. Use:Strengths and Opportunities

• Help teachers analyze data to understand where our achievement gaps are (90% vs. 96%)

• Help teachers select interventions for different subgroups of students (72% vs. 96%)

• Guide teachers’ use of student achievement data to plan effective instruction (86% vs. 94%)

29

Section 3: Importance, District, and State Support

• Importance:• Rate how important these questions are to

you in your work as a principal

• (District and State) Support:• Rate how much support you get to answer

these questions which are revealed by data

30

Importance Compared with Availability of District and State Support

• Instructional strategies that need to be implemented when students are not achieving academic standards (100%, 82%, 58%)

• Helping teachers become more effective (100%, 78%, and 46%)

• Finding more financial resources for school (100%, 57%, and 27%)

31

Section 4: Data Access & QualityStrengths and Opportunities (%

Agreement)• Data needed are accurate and reliable

(87%)

• Data needed are available in a timely fashion (79%)

• I have the time necessary to use data effectively (49%)

32

Data Use – Strengths and Opportunities (Agreement %)

• I have the authority to act on the needs revealed by the data (79%)

• My preparation/certification program prepared me to deal effectively with data (68%)

• I have the resources to act on the needs revealed by the data (63%)

33

More Data Issues – Strengths and Opportunities (Agreement %)

• I receive adequate support from my school district in using data (91%)

• Most experienced principals are well prepared to use accountability data effectively (88%)

• Most first-year principals are well prepared to use accountability data effectively (54%)

34

EffectiveSchoolLeadershipUses Data About: Student Demographics Student AchievementTeachers & ProgramsClimate & PerceptionFinancial & Human Resources

To Answer The Difficult Questions

Improved Teaching

& Student

Achievement

Conclusion: Data Informed Decision Making in a Leadership System

Standards Training Conditions

Set clear expectations about what leaders should be able to do with data to improve instruction & learning

Collaborate with higher education to provide quality preparation in the use of data

Provide continuing PD & support to leaders in using a wide variety of data

Ensure that leaders have the time, authority, and resources to make use of data

Data

Data

Data

35

How is the DID LIG Impacting the work of the State and District?

State Standards Training Conditions

JCPS

JCPS Human Resources will require data-informed decision-making components in all principal preparation programs.

IDEAS, Principals for Tomorrow, and Internship Program will include accountability data tools and their effective use.

JCPS Principal candidates will use real CATS data reports in demonstrating mastery of data knowledge and skills.

KY

The Education Professional Standards Board will require data-informed decision-making components in all principal preparation programs.

Statewide Principal Academy piloted in four districts will include accountability data tools and their effective use as part of the program.

Principal candidates will use real data reports in demonstrating mastery of data and to meet standards for licensure.

36

For More Information

Dr. Robert Rodosky [email protected]

Dr. Marco Muñoz [email protected]

data informed decision making

Education

data use

availability of data

data note

data access

data analysis

important data

program data importance

demographic data importance