implementing and validating a revised tqris: a north carolina update
DESCRIPTION
Implementing and Validating a Revised TQRIS: A North Carolina Update. Edna Neal Collins Jani Kozlowski Smart Start National Conference May 1, 2013. QRIS Advisory Committee. Met September 2009 – June 2012 Diverse group of stakeholders - PowerPoint PPT PresentationTRANSCRIPT
Implementing and Validating a Revised TQRIS: A North
Carolina Update
Edna Neal Collins
Jani Kozlowski
Smart Start National Conference
May 1, 2013
QRIS Advisory Committee
• Met September 2009 – June 2012
• Diverse group of stakeholders• For-profit, non-profit, and government sponsored ECE providers
• State and local early education partners
• Analyzed the current Quality Rating & Improvement System (QRIS) with the goal of creating recommendations that will guide the evolution of the system over the next 10 years.
• Drafted short-term and long-term recommendations reflecting a statewide vision for quality early care and education systems and programs.
• Recommendations directed to DCDEE, the Child Care Commission and other key system partners.
Considerations
All components of a QRISStandardsAccountability and assessmentsProgram and provider outreach & supportFinancial incentives linked to compliance with standardsConsumer education
Specific areas of emphasisCultural competenceInclusion of children with disabilitiesAlignment with Foundations
Systems that support the QRIS and programs
Advisory Committee Work Groups
Ad hoc group on QRIS Structure
Education and Professional Development
Evaluation of the Overall System
Finance
Infant-Toddler
Program Assessment
Program Standards
Recommendation Process
HomeworkStudied North CarolinaLooked at research and national reportsLooked at other statesListened to resource persons
Extensive review of recommendationsWork group ideasAnalysis (who affected, resources, timeframe, RTT,
etc.)Reviews by ChairsReviews in whole group
QRIS Structural Considerations
• Raise the floor and raise the ceiling.• Encourage continuous improvement.• Distinguish different levels and logical
“stair steps”.• Allow for child care provider choice.• Make it as simple as possible.• Integrate work group recommendations.• Provide a starting point for discussion.
Conceptual Framework for QRIS
Program & Environment
Ratios &Group Size
Education & Professional
Development
POLICIESAdmin Practices,Family Engagement,Cultural Responsiveness,Inclusion, PD Plans
SPACE,SAFETY,
COMPLIANCE, etc.
Block Requirements (increase with levels)
Core Requirements (maintained at all levels)
1st Level(Basic Licensing Requirements)
Conceptual Framework for QRIS
Program & Environment
Ratios &Group
Size
Education &
PD
POLICIESAdmin Practices, Family Engagement,Cultural Responsiveness, Inclusion, PD Plans
SPACE, SAFETY,
COMPLIANCE, etc.
1st
Level
Block Requirements (increase with levels)
Core Requirements (maintained at all levels)
Basic Licensing Requirements
2nd
Level
Specialization
Could Include CSEFEL
Infant Toddler
School Age
STEM The Arts ETC.
All requirements of Level 1, plus points that lead toward the Level 3 standards. Points must be earned in at least 2 of these 4 categories
(Program & Environment; Ratios & Group Size; Education & PD; Specialization)
Conceptual Framework for QRIS
Program & Environment
Ratios &Group
Size
Education &
PD
POLICIESAdmin Practices, Family Engagement,Cultural Responsiveness, Inclusion, PD Plans
SPACE, SAFETY,
COMPLIANCE, etc.
1st
LevelBlock Requirements (increase with levels)
Core Requirements (maintained at all levels)
Basic Licensing Requirements
2nd
Level
Specialization
Could Include CSEFEL
Infant Toddler
School Age
STEM The Arts ETC.
All requirements of Level 1, plus points that lead toward the Level 3 standards. Points must be earned in at least 2 of these 4 categories
(Program & Environment; Ratios & Group Size; Education & PD; Specialization)
3rd
Level*
Program &
Environment
Ratios &
Group Size
Ed/ PD
Requirements in P&E, Ratios/Group Size, Ed/PD, PLUS Specialization
* ERS or other quality measure required at this level. Also, it is expected that program accreditation will be added to the model during the pilot phase, most likely at levels 3-5.
Conceptual Framework for QRIS
Program & Environment
Ratios &Group
Size
Education &
PD
POLICIESAdmin Practices, Family Engagement,Cultural Responsiveness, Inclusion, PD Plans
SPACE, SAFETY,
COMPLIANCE, etc.
1st
Level
Block Requirements (increase with levels)
Core Requirements (maintained at all levels)
Basic Licensing Requirements
2nd
Level
Specialization
Could Include CSEFEL
Infant Toddler
School Age
STEM The Arts ETC.
All requirements of Level 1, plus points that lead toward the Level 3 standards. Points must be earned in at least 2 of these 4 categories
(Program & Environment; Ratios & Group Size; Education & PD; Specialization)
3rd
Level*
Program & Environment
Ratios &Group Size
Ed/ PD
4th
Level*
All requirements of Level 3, plus points that lead toward the Level 5 standards.
Points must be earned in at least 2 of these 4 categories (Program & Environment; Ratios & Group Size; Education & PD; Specialization)
Potential Programs of Distinction starting at 4th Level
Requirements in P&E, Ratios/Group Size, Ed/PD, PLUS Specialization
* ERS or other quality measure required at this level. Also, it is expected that program accreditation will be added to the model during the pilot phase, most likely at levels 3-5.
Conceptual Framework for QRIS
Program & Environment
Ratios &
Group Size
Education &
PD
POLICIESAdmin Practices, Family Engagement, Cultural Responsiveness, Inclusion, PD Plans
SPACE, SAFETY, COMPLIANCE,
etc.
1st
Level
Block Requirements (increase with levels) Core Requirements
(maintained at all levels)
Basic Licensing Requirements
2nd
Level
Specialization
Could Include CSEFEL
Infant Toddler
School Age
STEM The Arts ETC.
All requirements of Level 1, plus points that lead toward the Level 3 standards. Points must be earned in at least 2 of these 4 categories
(Program & Environment; Ratios & Group Size; Education & PD; Specialization)
3rd
Level*
Program & Environment
Ratios &Group Size
Ed/ PD
4th
Level* All requirements of Level 3, plus points that lead toward the Level 5 standards.
Points must be earned in at least 2 of these 4 categories (Program & Environment; Ratios & Group Size; Education & PD; Specialization)
Potential Programs of Distinction at the 4th and 5th Levels
5th
Level*
Program &
Environment
Ratios &
Group Size
Ed/ PD
Requirements in P&E, Ratios/Group Size, Ed/PD, PLUS Specialization
Requirements in P&E, Ratios/Group Size, Ed/PD, PLUS Specialization
* ERS or other quality measure required at this level. Also, it is expected that program accreditation will be added to the model during the pilot phase, most likely at levels 3-5.
Executive Summary
Process Recommendations
• Consider data about how programs meet current requirements.
• Map current and proposed requirements onto the proposed conceptual framework.
• Collect a lot of input from stakeholders about both the requirements of the new system and how it should be implemented.
• “Test” the proposed revisions using existing data and targeted data collection.
• Project how programs would be distributed across a proposed new structure or structures before finalizing the revised TQRIS.
Implementation Plan
Executive Summary available at http://ncchildcare.dhhs.state.nc.us/PDF_forms/QRIS-Advisory-Committee-Executive-Summary.pdf
Spreadsheet of specific recommendations provided to DCDEE
DCDEE staff coordinate feedback, validation, and implementation process. Integrate recommendations to create a coherent
system. Refer some specific recommendations directly to
Child Care Commission Refer system-level recommendations to system
partners.
Race to the TopEarly Learning
Challenge
Building on the Recommendations
RttT-ELC
• U.S. Department of Education and U.S. Department of Health & Human Services
• Total federal funding of $500 million
• Award Period: January 2012 – December 2015
• 9 winners: NC, MA, WA, DE, OH, MD, MN, RI, CA
• NC Award: $69,991,121
• Primary focus: School readiness for children at risk
RttT-ELC and the QRIS
• QRIS as a key component of a system to promote school readiness for all children.
• QRIS-related activities in NC grant• Validating and implementing a revised TQRIS
• Developing a new program quality measure for use in a TQRIS
• Strengthening TA and PD related to possible QRIS quality indicators
• Strengthening the workforce
• Expanding the QRIS & moving programs to higher levels
What is TQRIS Validation?
Validation of a QRIS is a multi-step process that assesses the degree to which design decisions about program quality standards and measurement strategies are resulting in accurate and meaningful ratings.
Zellman, G. L. & Fiene, R. (2012). Validation of Quality Rating and Improvement Systems for Early Care and Education and School-Age Care, Research-to-Policy, Research-to-Practice Brief OPRE 2012-29. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.
TQRIS Validation
(Zellman & Fiene, 2012)
1. Examine the validity of underlying constructs.• Expert and empirical support• “Stakeholder validity”
2. Examine the measurement strategies and psychometric properties of measures.
• Do they perform as expected?• Do the “cut scores” differentiate levels appropriately?
3. Assess the outputs of the rating process.
4. Relate the ratings to children’s developmental outcomes (and other important outcomes).
Pilot Study and Validation Study
• Study the impact of recommended changes.
• Use results to develop blocks, points, weighting, and standards for each level.
QRIS Advisory Committee
Recommendation for Pilot Study
• Identify quality features that distinguish between programs at higher levels.
• Relate program features and quality ratings to children’s progress, especially children with High Needs.
Race to the Top Early Learning
Challenge - TQRIS Validation
Study
Pilot/Validation Study
Revised TQRIS Pilot and Validation Study (funded by RttT-ELC)
• Study the impact of recommended changes.
• Use results to develop blocks, points, weighting, and standards for each level.
• Identify quality features that distinguish between programs at higher levels.
• Relate program features and quality ratings to children’s progress.
• Study the impact of.
Validation Study Phase 1
• Gather information from stakeholders to inform the creation of alternative TQRIS models for testing.
• Generate 3-4 alternative models for testing, consistent with the Conceptual Framework, including Core Requirements, Blocks, Points, and Specializations.
• Test 3-4 alternative models using existing data, targeted data collection, and additional stakeholder input.
• Use results of this process to choose one model for further testing and validation in Phase II.
Validation Study Phase 2 (subject to approval)
• Validate the proposed new Block levels.• Examine WHICH quality indicators do the best job of
distinguishing among programs at the higher levels of quality.
• Test reliability and validity of a few new components.• Conduct a pilot study, including assessment of
relationship between quality ratings in the revised TQRIS and child outcomes.
• Generate data needed to make adjustments to the revised TQRIS before implementation.
Recent Work
• DCDEE Staff mapped individual recommendations onto conceptual framework.
• QRIS Advisory Committee Work Group Chairs gave feedback.
• Contract for Phase I of Validation Study awarded to FPG.
• Stakeholder interviews completed.
• Provider survey launched.
• Focus groups in progress.
Challenges to Discuss
• Provider buy-in
• Balancing consistency and flexibility
• Supporting continuous quality improvement
• Supporting and recognizing the highest quality
• Weaving in all aspects of quality
• The role of accreditation