A new scoring approach for ECERS-R

Richard Clifford, PhDJohn Sideris, PhD

Jennifer Neitzel, PhDBeatriz Abuchaim, MSc

University of North CarolinaFPG Child Development Institute

Chapel Hill, April 2012

Previous studies

• Several studies have found that there is an underlying factor structure in the ECERS-R, beyond the subscale level (e.g., Clifford & Rossbach, 2005; Early, et al., 2005; Sakai, Whitebook, Wishard, & Howes, 2003)

• Most commonly, two factors have emerged: (1) Teaching and Interaction and (2) Provisions for Learning)

Previous studies (cont.)

• Studies have showed that results of the ECERS-R are related to a variety of children outcomes, but this relationship is only modest (Aboud, 2006; Burchinal et al., 2000; McCartney, Scarr, Phillips, & Grajek, 1985; Phillips, McCartney, & Scarr, 1987; Buchinal et al., 2011)

• Concern has been raised about factor scores and category disordering that may help explain the very modest relations to child outcomes, particularly in higher quality classrooms (Gordon, et al., in press)

• Our hypothesis is that a new scoring system, using the indicator information, can improve the predictive power of the ECERS-R

Goals

•To develop a new scoring system, using the indicator level information

•To test the predictive power of this new system

Sample

•8500 cases, from 6 different studies, in which all the indicators were scored

•States: California, Iowa, Minnesota, Nebraska, North Carolina, Georgia, Illinois, Kentucky, New York, Ohio, Massachusetts, New Jersey, Texas, Washington and Wisconsin

• Issue with skewed distribution, few low scoring programs

Procedures

•Step 1 – Hypothesize a new set of factors for the ECERS-R

•Step 2 – Conduct Factor Analyses

•Step 3 – Conduct Confirmatory Analyses

•Step 4 - Test of these new factors to check their predictive power.

Hypothesized New Subscales or Factors

• Use of time

• Special Needs

• Physical Environment

• Individualization

• Diversity

• Access to Materials

• Creativity

• Grouping

• Fine Motor

• Gross Motor

• Independence

• Social/Emotional

• Engagement

• Routines

• Teaching

• Science/Math/Reasoning

• Literacy/Language/Concepts

• Health

• Safety

• Families

• Staff

• Supervision

Characterizing each indicator

ITEM INDICATOR PRIMARY SECONDARY TERTIARY

17. Using language to

develop reasoning

skills

17.5.2 Children encouraged to talk through or explain their reasoning when solving problems (Ex. why they sorted objects into different groups; in what way are two pictures the same or different).

Literacy Language Concepts

Social Emotional

Engagement

17.3.1 Staff sometimes talk about logical relationships or concepts (Ex. explain that outside time comes after snacks, point out differences in sizes of blocks child used).

Social Emotional

LiteracyLanguage Concepts

Teaching

17.7.2 Concepts are introduced in response to children’s interests or needs to solve problems (Ex. talk children through balancing a tall block building; help children figure out how many spoons are needed to set table).

EngagementSocial

Emotional

LiteracyLanguage Concepts

Factor Analysis

•All indicators for the Parents and Staff Subscale were dropped.

•Multiple Factor Analyses were carried out to test the newly hypothesized factors.

•Some models included multiple factors

Confirmatory model

•Models were confirmed on remaining half of the sample

•Example: Education factor: Teaching, Literacy and Math/Science

Model Fit

•Across most models, fit was good

•Chi-square were all significant, unsurprising given sample size

•RMSEA all .04 or less

•CFI ranged between .80 and .97

Problematic Indicators

• In all of these models, some indicators presented estimation problems and were eliminated.

•Extremely low variance

•Correlated at one with other indicators in the model.

•Empty cells in the 2 X 2 crosstabs of pairs of indicators

Problematic Indicators, Example

•6.1.2 Child related display: Inappropriate materials for predominant age group.

•99.84% of our sample passed this indicator.

•Lack of variance may be due to non-random sample

Problematic Models

•Special Needs – one factor solution required the elimination of the majority of indicators

•Use of time and routines – two factor solution not replicated in second half of the sample

Exploratory Model for Health

•A set of health and safety indicators were selected.

•Less certain that they would represent a single factor

•First analysis indicated three factors, but third factor included only about 8 of 40 indicators, all cross-loaded on first two factors

Exploratory Model for Health

•Factor One – General Health & Safety

• 10.1.3 Sanitary conditions not usually maintained

• 11.1.2 Nap/Rest provisions unsanitary

•Factor Two – Supervision to Promote Health and Safety

• 31.1.2 Discipline is so lax that there is little order or control

• 14.3.2 Adequate supervision to protect children’s safety indoors and outdoors

Expected OrderingDescription

7.7.3 Space has convenient features (Ex. close to toilets and drinking water, accessible storage for equipment; class has direct access to outdoors).

23.7.2 Different activities done with sand and water (Ex. bubbles added to water, material in sand table changed, i.e. rice substituted for sand).

24.7.1 Materials rotated for a variety of themes (Ex. prop boxes for work, fantasy, and leisure themes).

31.7.3 Staff seek advice from other professionals concerning behavior problems.

30.5.3 Staff show awareness of the whole group even when working with one child or a small group (Ex. staff frequently scan room when working with one child, make sure area not visible is supervised by other staff).

3.5.2 Space for privacy can be easily supervised by staff.6.5.2 Most of the display is work done by the children.29.5.1 Staff act to prevent dangerous situations before they occur (Ex. remove broken toys or

other dangers prior to children’s use; stop rough play before children get hurt).

27.3.3 Time children allowed to use TV/video or computer is limited (Ex. TV/videos limited to one hour daily in full-day program; computer turns limited to 20 minutes daily).

4.3.2 Visual supervision of play area is not difficult.

Actual Ordering

IndicatorThresho

ld7.7.3 0.154

6.5.2 -0.45023.7.2 -0.73227.3.3 -0.7953.5.2 -0.829

24.7.1 -0.8314.3.2 -0.852

31.7.3 -0.89530.5.3 -0.94929.5.1 -0.974

Inter-Factor Correlations

Inter-Factor Correlations

Correlations with Traditional Scored

ECERS-R

Conflict of Interest

Disclosure

• Richard Clifford has a financial conflict of interest as a result of receiving royalty and consulting payments in connection with use of the ECERS-R. His work on this effort is conducted under IRB approval from the University of North Carolina at Chapel Hill which includes a management plan for dealing with the conflict of interest noted here.