validation of a school reform rubric

7/28/2019 Validation of a School Reform Rubric

1/15

Validation of a SchoolReform Rubric

Susan Gracia

RIC Faculty Research

Conference

April 27, 2007


2/15

CSR

The objective of the federal Comprehensive SchoolReform (CSR) Program is to improve studentachievement by supporting the implementation ofcomprehensive school reforms based on scientifically

based research and effective practices so that allchildren, especially those in low-performing, high-povertyschools, can meet challenging state content andacademic achievement standards (CSR ProgramGuidance, July 26, 2002).

Specifically, comprehensive school reform is a means toimprove student achievement through reorganizing andrevitalizing the entire school, rather than implementingisolated programs.


3/15

CSR Components

CSR schools must plan a comprehensive program, asdefined by these 11 eleven components:

Effective, research-based, replicable methods and strategies

Comprehensive design with aligned components

Professional development Measurable goals and benchmarks

Support within the school

Support for principals and staff

Parental and community involvement

External technical support and assistance Evaluation strategies

Coordination of resources

Scientifically-based evidence of improved student achievement


4/15

CSR Evaluation

Implementation/progress of each component

needs to be documented

A number of instruments exist Rubrics, surveys, etc. purport to be able to

identify stages of implementation of each

component

E.g., CSR Implementation Continuum


5/15

Present Study

CSR Implementation Continuum was

modified into a series of survey scales

153 teachers in CSR schools rated theirperceptions of CSR implementation in their

schools along the 11 CSR components


6/15

Research Questions

Do survey scales fit the Rasch model?

If so:

What is the nature of the continuum of CSRimplementation (for each component)?

Does the continuum of CSR implementation

match the CSR Implementation Continuum

rubric? How can measurement of CSR implementation be

improved?


7/15

Rasch model

People and items form a hierarchy of less than/more

than on a continuum (variable)

People are more likely to agree with easy (to

endorse) items than difficult (to endorse) items Respondents perceiving high CSR implementation

are more likely to endorse all items than those

perceiving low implementation

Probability of providing responses defines an order

of respondents and items


8/15

Rasch model

Parameters are neither sample nor test

dependent

Total scores have interpretable meaning

Ordinal data is converted to interval data

Missing data is not problematic


9/15

Model (PD Scale)

CSR implementation data fit the model

Fit statistics were well within defined bounds

Item reliability = .97

How precisely items can be located on the latent variable Person reliability = .86

The Rasch equivalent of the KR-20 or Cronbach Alpha"test reliability" statistic

Item and person separation indices are 5.63 and2.48 Items have sufficient breadth and persons are well-

discriminated.


10/15

Variable Map

A yardstickresulting from data

Reveals the operational definition of what the

CSR PD scale is measuring.

The variable map allows one to see how

items spread out over the continuum of CSR

implementation and how respondents

distribute on the CSR implementationvariable.


11/15

Respondents MAP OF Items

|

6 XXXX +

|

|

|

XX |

|

5 +

|

|

XXXXX T|

X |

|

4 XXXX +

||

XXXXX |

XXXXXX |

S|

3 XXXXX +

|

XXX |

XXXXXXXXXXX |

XXX |

|

2 XXXX +

XXXX | MANY PD OPPS AVAILABLE TO PARENTS/COMMUNITY

XXXXXXX M|

XXXXXXXXXX |T

XXXXXXX |

XXXXXXXXX |

1 XXXXXXXXXX + STAFF RECEIVE FEEDBACK ON STRATEGY IMPLEMENTATION

XXXX |SXXXX |

XXX |

XXXX | PD PLAN RELATED TO SCHOOL GOALS/STAFF INPUT

STAFF REGULARLY ATTENDS PD & WORKGROUPS

XX S| PD CONTINUOUS, DIRECTED TOWARD COMMON GOALS

SCHOOL DEVELOPS INTERNAL CAPACITY FOR PD

0 XXXXXXX +M

XXX | PD IS ONGOING PROCESS BASED ON STAFF NEEDS

XXX | PD OPPS FOCUS ON FACULTY NEEDS RE: STRATEGIES

SCHOOL HAS GENL PLAN FOR PD

SCHOOL PROVIDES ONSITE PD & TIME FOR REFLECTION

XX |

| SOME PD OPPS AVAILABLE ONSITE

XX |S

-1 XX +

| PD OPPS RELATED TO GOALS

STAFF GENERALLY ATTENDS PD

X T|

|T

|

|

-2 +

|

|

|

X |

|

-3 +

|

Most

difficultitems to

endorse

People reporting

highest

implementation

Easiest items

to endorse

People reporting

lowest

implementation


12/15

Correspondence with original CSR

Implementation Continuum

If expectations of what is being measured are

based on good theory, the results shown in

the empirical map should correspond to the

conceptual one


13/15

Respondents MAP OF Items

|

6 XXXX +

|

|

|

XX |

|

5 +

|

|

XXXXX T|

X |

|4 XXXX +

|

|

XXXXX |

XXXXXX |

S|

3 XXXXX +

|

XXX |

XXXXXXXXXXX |

XXX |

|

2 XXXX +

XXXX | Level 5XXXXXXX M|

XXXXXXXXXX |T

XXXXXXX |

XXXXXXXXX |

1 XXXXXXXXXX + Level 3

XXXX |S

XXXX |

XXX |

XXXX | Level 2 Level 4

XX S| Level 5 Level 5

0 XXXXXXX +M

XXX | Level 4

XXX | Level 1 Level 3 Level 3

XX || Level 2

XX |S

-1 XX +

| Level 1 Level 3

X T|

|T

|

|

-2 +

|

|

|

X |

|

-3 +

|


14/15

Results

Implementation as laid out in CSR

Implementation Continuum rubric does not

match implementation continuum as revealed

through survey data

Similar results for all 11 scales/CSR

components

CSR Implementation Continuum should berevised to reflect empirical data


15/15

REVISED CSR PROFESSIONAL DEVELOPMENT IMPLEMENTATION RUBRIC

1 2 3 4 5Professional development

opportunities at my school

are related to school,district, or state goals.

Staff, in general, attends

professional development

activities.

Some professional

development opportunities

are available onsite.

The school provides some

onsite professional

development opportunities

as well as time for

reflective practice.

The school has a general

plan for professional

development.

Professional development

opportunities focus on the

needs of faculty to

implement improvement

strategies.


is an ongoing process

based on staff needs toimplement improvement

strategies.


is continuous and self-

directed towards theachievement of

collectively developed

school wide improvement

goals.

The school develops

internal capacity to provide

onsite professional

development to its faculty

and sponsors small gradelevel or subject matter

work groups of teachers

for reflective practice.

Staff regularly participates

in professional

development activities and

in small grade level or

subject matter work groups

to reflect on practice.

The professional

development plan is

related to school wide

improvement goals and is

the result of staff input.

Instructional staff receive

feedback on the

implementation ofimprovement strategies.

Many of these professional

development services are

available to parents andcommunity members.

validation of a school reform rubric

Documents