the program review process: ncate and the state of indiana
DESCRIPTION
The Program Review Process: NCATE and the State of Indiana. Richard Frisbie and T. J. Oakes March 8, 2007 (source:NCATE, February 2007). Programs for which NCATE does not have standards. Art, music, dance, or drama education Business, speech, and vocational education - PowerPoint PPT PresentationTRANSCRIPT
The Program Review Process: NCATE and the State of Indiana
Richard Frisbie and T. J. OakesMarch 8, 2007(source:NCATE, February 2007)
Programs for which NCATE does not have standards
Art, music, dance, or drama education Business, speech, and vocational
education Some advanced teacher education
programs (e.g., Curriculum & Instruction School counselorThese programs undergo state review or
are accredited by other accrediting agencies
NCATE Program Review-Specialized Professional Associations (SPAs)
The national organizations that represent teachers, professional education faculty, and other school personnel who teach a specific subject matter, teach students at a specific developmental level, teach students with special needs, administer schools, or provide services to students.
State of Indiana Program Review
The state review is similar to the NCATE process. For details on the state review, see http://www.doe.state.in.us/dps/teacherprep/program_review.html
Challenges for Institutions
Programs are having surprising difficulty demonstrating how assessments align with SPA standards
Programs have had difficulty selecting appropriate assessments
6-8 Assessments: The Rules
Institution must submit a minimum of six assessments, unless the SPA specifies more than six required assessments
Institution may submit additional assessments when SPA does not specify all eight assessments
Five specific types of assessments are required by all SPAs
Required Assessments in National Program Review
1. State licensure exam for content2. Assessment of content knowledge3. Assessment of Planning (e.g., lesson
plan or unit plan)4. Student teaching/internship assessment5. Assessment of candidate impact on
student learning or providing a supporting learning environment
6. Other assessment
NCATE Unit Standard #1 Candidates preparing to work in
schools as teachers or other school personnel know the content of their fields, demonstrate professional and pedagogical knowledge, skills, and dispositions and apply them so that students learn. Assessments indicate that candidates meet professional, state, and institutional standards.
Required Program Assessments
Content:1. State licensure exam for program
area (if available—otherwise another content based assessment)
2. Content AssessmentProfessional and Pedagogical
Knowledge Skills and Dispositions:3. Assessment of Planning (e.g., unit
plan)
4. Student teaching/internship assessment
P – 12 Student Learning:5. Assessment of candidate impact on
student learning or providing a supporting learning environment
Program is required to have at least one more assessment, can have a maximum of 8 to demonstrate candidate mastery of SPA standards
Some SPAs have additional required types
Important things learnedfrom the pilots:
Unit-wide assessments can be used as key program assessments, but faculty must demonstrate alignment with specific SPA standards
Three effective strategies
SPA-specific addendum SPA-specific rubrics Coding assessment
AND including information in narrative about context
Important things learned:
It is critical to document alignment between assessments, scoring guides and SPA standards
Most Important Learning from Pilots: All Parts of Assessments Must Interrelate!
Scoring Guide
Data & Findings
Assessment
STANDARDS
What does that mean?
Parts of the assessment are aligned with the SPA standards
These standards can be seen in the elements of the scoring guide
The data is broken down by the elements in the scoring guide
Developing Appropriate Assessments: What are the Criteria?
Working definitions of assessment
The term “assessment” can refer to different aspects of the program review process
Any of the 5 required types of assessment
A specific element of an assessment Assessment (assignment summary) Scoring guide Data & findings
The five key principles by which reviewers evaluate your reports
Are assessments aligned with standards? Do assessments meet the cognitive
demands and skill levels of the standards? Are assessments free from bias? Do scoring guides clearly describe the
different categories? Do data provide evidence for meeting
standards?
Distributing assessments across standards
Do not take a “this assessment addresses all standards” approach
Think of assessments as “primary” vs. “supportive” in meeting standards
Typically, one to three assessments, taken as a whole, meet a single standard
The assessment instrument
Align to standards! Preferably with references to standards in the assessment itself.
If alignment is not embedded in the assessment, provide alignment chart in your discussion of the assessment, and a strong argument for how the assessment covers the standard. Be specific.
Characteristics of a strong assessment
The objectives candidates must meet correspond to the depth and breadth required by SPA standards
Assessments are scored primarily on standards-based criteria.
The required activities candidates must perform are clearly described.
Assessment shows no evidence of bias.
Scoring Guides
must be consistent with the assessment instrument
acceptable scores must target the level of competence required by a standard
must clearly describe differences among scored categories
Putting it all together (with data)
The data table is how you “tell your story”
Data should be collected and used in ways that help your program analyze its strengths and weaknesses.
Data should mirror the scoring categories for the assessment (e.g. if grades are used, data should be reported as grades).
Speaking of grades…
Grades in required courses are usually acceptable for Assessment #2.
Provide course descriptions and align courses to standards.
Course content must intensively and comprehensively address standards.
Show range of grades received by candidates, not just GPAs.
Other comments on data
Remove identifying #s and names.Make sure the number of candidates
in data sets are consistent across tables in reports, or explain inconsistencies that might puzzle reviewers.
Note and explain any conversions of data (e.g. numerical scores converted to grades in a data table).
Analyze your data
Make the most of the two pages you have per assessment to analyze and “defend” your assessments and data (Section IV). Analyze your data according to the SPA standards. Compare data across standards, cohort groups, time periods, etc. Tell what you learn from it.
In Section V, make sure to describe what you are doing or will do to address areas where data are poor or below average.
How Much Data?
Fall 2008 and spring 2009, two
years of data Three years of data are optimal –
not necessary to provide more Remember: Data are only as good
as the assessment/scoring guide
Resources on NCATE web site www.ncate.org, click on Institutions,
click on Program Reviews Program Report Forms, Guidelines,
Instructions, Examples of Assessments, Examples of Report Sections, Examples of Reports
www.ncate.org, click on Program Reviewers All training materials for program
reviewers
Who to Contact at SPAs
For questions related to the standards and what are appropriate assessments for a particular discipline:
SPA Program Review Coordinators http://www.ncate.org/programreview/spacontact.asp?ch=88