![Page 1: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/1.jpg)
The Program Review Process: NCATE and the State of Indiana
Richard Frisbie and T. J. OakesMarch 8, 2007(source:NCATE, February 2007)
![Page 2: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/2.jpg)
Programs for which NCATE does not have standards
Art, music, dance, or drama education Business, speech, and vocational
education Some advanced teacher education
programs (e.g., Curriculum & Instruction School counselorThese programs undergo state review or
are accredited by other accrediting agencies
![Page 3: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/3.jpg)
NCATE Program Review-Specialized Professional Associations (SPAs)
The national organizations that represent teachers, professional education faculty, and other school personnel who teach a specific subject matter, teach students at a specific developmental level, teach students with special needs, administer schools, or provide services to students.
![Page 4: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/4.jpg)
State of Indiana Program Review
The state review is similar to the NCATE process. For details on the state review, see http://www.doe.state.in.us/dps/teacherprep/program_review.html
![Page 5: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/5.jpg)
Challenges for Institutions
Programs are having surprising difficulty demonstrating how assessments align with SPA standards
Programs have had difficulty selecting appropriate assessments
![Page 6: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/6.jpg)
6-8 Assessments: The Rules
Institution must submit a minimum of six assessments, unless the SPA specifies more than six required assessments
Institution may submit additional assessments when SPA does not specify all eight assessments
Five specific types of assessments are required by all SPAs
![Page 7: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/7.jpg)
Required Assessments in National Program Review
1. State licensure exam for content2. Assessment of content knowledge3. Assessment of Planning (e.g., lesson
plan or unit plan)4. Student teaching/internship assessment5. Assessment of candidate impact on
student learning or providing a supporting learning environment
6. Other assessment
![Page 8: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/8.jpg)
NCATE Unit Standard #1 Candidates preparing to work in
schools as teachers or other school personnel know the content of their fields, demonstrate professional and pedagogical knowledge, skills, and dispositions and apply them so that students learn. Assessments indicate that candidates meet professional, state, and institutional standards.
![Page 9: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/9.jpg)
Required Program Assessments
Content:1. State licensure exam for program
area (if available—otherwise another content based assessment)
2. Content AssessmentProfessional and Pedagogical
Knowledge Skills and Dispositions:3. Assessment of Planning (e.g., unit
plan)
4. Student teaching/internship assessment
![Page 10: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/10.jpg)
P – 12 Student Learning:5. Assessment of candidate impact on
student learning or providing a supporting learning environment
Program is required to have at least one more assessment, can have a maximum of 8 to demonstrate candidate mastery of SPA standards
Some SPAs have additional required types
![Page 11: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/11.jpg)
Important things learnedfrom the pilots:
Unit-wide assessments can be used as key program assessments, but faculty must demonstrate alignment with specific SPA standards
![Page 12: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/12.jpg)
Three effective strategies
SPA-specific addendum SPA-specific rubrics Coding assessment
AND including information in narrative about context
![Page 13: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/13.jpg)
Important things learned:
It is critical to document alignment between assessments, scoring guides and SPA standards
![Page 14: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/14.jpg)
Most Important Learning from Pilots: All Parts of Assessments Must Interrelate!
Scoring Guide
Data & Findings
Assessment
STANDARDS
![Page 15: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/15.jpg)
What does that mean?
Parts of the assessment are aligned with the SPA standards
These standards can be seen in the elements of the scoring guide
The data is broken down by the elements in the scoring guide
![Page 16: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/16.jpg)
Developing Appropriate Assessments: What are the Criteria?
![Page 17: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/17.jpg)
Working definitions of assessment
The term “assessment” can refer to different aspects of the program review process
Any of the 5 required types of assessment
A specific element of an assessment Assessment (assignment summary) Scoring guide Data & findings
![Page 18: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/18.jpg)
The five key principles by which reviewers evaluate your reports
Are assessments aligned with standards? Do assessments meet the cognitive
demands and skill levels of the standards? Are assessments free from bias? Do scoring guides clearly describe the
different categories? Do data provide evidence for meeting
standards?
![Page 19: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/19.jpg)
Distributing assessments across standards
Do not take a “this assessment addresses all standards” approach
Think of assessments as “primary” vs. “supportive” in meeting standards
Typically, one to three assessments, taken as a whole, meet a single standard
![Page 20: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/20.jpg)
The assessment instrument
Align to standards! Preferably with references to standards in the assessment itself.
If alignment is not embedded in the assessment, provide alignment chart in your discussion of the assessment, and a strong argument for how the assessment covers the standard. Be specific.
![Page 21: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/21.jpg)
Characteristics of a strong assessment
The objectives candidates must meet correspond to the depth and breadth required by SPA standards
Assessments are scored primarily on standards-based criteria.
The required activities candidates must perform are clearly described.
Assessment shows no evidence of bias.
![Page 22: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/22.jpg)
Scoring Guides
must be consistent with the assessment instrument
acceptable scores must target the level of competence required by a standard
must clearly describe differences among scored categories
![Page 23: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/23.jpg)
Putting it all together (with data)
The data table is how you “tell your story”
Data should be collected and used in ways that help your program analyze its strengths and weaknesses.
Data should mirror the scoring categories for the assessment (e.g. if grades are used, data should be reported as grades).
![Page 24: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/24.jpg)
Speaking of grades…
Grades in required courses are usually acceptable for Assessment #2.
Provide course descriptions and align courses to standards.
Course content must intensively and comprehensively address standards.
Show range of grades received by candidates, not just GPAs.
![Page 25: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/25.jpg)
Other comments on data
Remove identifying #s and names.Make sure the number of candidates
in data sets are consistent across tables in reports, or explain inconsistencies that might puzzle reviewers.
Note and explain any conversions of data (e.g. numerical scores converted to grades in a data table).
![Page 26: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/26.jpg)
Analyze your data
Make the most of the two pages you have per assessment to analyze and “defend” your assessments and data (Section IV). Analyze your data according to the SPA standards. Compare data across standards, cohort groups, time periods, etc. Tell what you learn from it.
In Section V, make sure to describe what you are doing or will do to address areas where data are poor or below average.
![Page 27: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/27.jpg)
How Much Data?
Fall 2008 and spring 2009, two
years of data Three years of data are optimal –
not necessary to provide more Remember: Data are only as good
as the assessment/scoring guide
![Page 28: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/28.jpg)
Resources on NCATE web site www.ncate.org, click on Institutions,
click on Program Reviews Program Report Forms, Guidelines,
Instructions, Examples of Assessments, Examples of Report Sections, Examples of Reports
www.ncate.org, click on Program Reviewers All training materials for program
reviewers
![Page 29: The Program Review Process: NCATE and the State of Indiana](https://reader036.vdocument.in/reader036/viewer/2022062518/568146e0550346895db41716/html5/thumbnails/29.jpg)
Who to Contact at SPAs
For questions related to the standards and what are appropriate assessments for a particular discipline:
SPA Program Review Coordinators http://www.ncate.org/programreview/spacontact.asp?ch=88