© ccsr 5essentials: background and methodology rachel levenstein senior manager for survey research...

© C

CS

R

5Essentials: Background and methodology

Rachel LevensteinSenior Manager for Survey Researchrmlev@uchicago.edu

© C

CS

R

20 years of research in Chicago Public Schools (CPS)- Conversations with practitioners- Wide scan of the literature- Surveys

Organizing Schools for Improvement

Origin of the 5Essentials

© C

CS

R

Surveys of CPS

1999

2001

2003

2005

2007

2009

2011

2012

2013

2014

2015

0

0.2

0.4

0.6

0.8

1

Student and teacher response rates

Students Teachers

© C

CS

R

Initial measure development process conducted in CPS

• Prior research findings indicate a link between construct and student outcomes1. Search existing literature for survey questions

2. Focus groups Draft questions

3. Cognitive interviews Revise questions

4. Pilot survey questionsCheck reliability, revise questions

• Validation study

© C

CS

R

Development of survey questions: Focus groups

• Gather 5-10 practitioners or students together to talk about the concept.

• Moderated by a researcher, using a protocol.

• Identify themes and common nomenclature.

Get ideas for how to write survey questions

© C

CS

R

Development of survey questions: Cognitive Interviews• One-on-one interviews with practitioners,

students• “Tell me how you came to your answer”• “What kinds of programs were you thinking

about?”

Identify problems understanding or interpreting the questions.

© C

CS

R

Development of survey questions: Pilot testing

• Small- or large-scale pilot• Same conditions as the “regular” survey

– May be conducted as part of the “regular” survey

• Analyze responses and calculate reliability.

Identify if the measure holds together (or not).

© C

CS

R

Development of survey questions:Validation study

Program Coherence example: Newman et al., 2001

– Qualitative data collection: (School improvement plans, classroom observations, staff meeting observations, assignment samples)

• Reports highly correlated with survey measure

– Schools that improved on Program Coherence between 1994 and 1997 showed more improvement on ITBS than schools that did not improve—by 1/5 of a school year!

© C

CS

R

Practices change Understanding of constructs change General revisions for clarity Clients outside of Chicago inform some changes

May add, delete, or modify questions every few years.

Changes to content

© C

CS

R

To what extent do you agree or disagree with each of the following statements?

- You can see real continuity from one program to another in this school.

- Many special programs come and go in this school.- Once we start a new program, we follow up to

make sure that it’s working.- We have so many different programs in this school

that I can’t keep track of them all.- (Strongly disagree, disagree, agree, strongly agree)

Changes to content: Program Coherence

© C

CS

R

To what extent do you agree or disagree with each of the following statements?

- You can see real continuity from one program to another in this school.

- Once we start a new program, we follow up to make sure that it’s working.

- Many special programs come and go in this school.- Once we start a new program, we follow up to make sure

that it’s working.- We have so many different programs in this school that I

can’t keep track of them all.- (Strongly disagree, disagree, agree, strongly agree)

Changes to content: Program Coherence

© C

CS

R

How much influence do teachers have over school policy in each of the areas below?- Hiring new professional personnel.- Planning how discretionary school funds should be used.- Determining books and other instructional materials used

in classrooms.- Establishing the curriculum and instructional programs.- Setting standards for student behavior.(Not at all, A little, Some, To a great extent)

Changes to content: Teacher Influence

© C

CS

R

How much influence do teachers have over school policy in each of the areas below?- Hiring new professional personnel.- Planning how discretionary school funds should be used.- Determining books and other instructional materials used in

classrooms.- Establishing the curriculum and instructional programs.- Setting standards for student behavior.

(Not at all, A little, Some, To a great extent)

(No influence, A little influence, Some influence, A great deal of influence)

Changes to content: Teacher Influence

© C

CS

R

Putting a measure into the 5Essentials

• Two years of data before it becomes an Essential

• Fit within 5Essentials framework– Factor analysis