what should you consider when talking about technical assessments? john m. townsend tennessee board...
TRANSCRIPT
What Should You Consider What Should You Consider When Talking About When Talking About
Technical Assessments?Technical Assessments?
John M. TownsendJohn M. TownsendTennessee Board of RegentsTennessee Board of Regents
NACTEI 28NACTEI 28thth Annual Conference Annual Conference
Boise, IdahoBoise, Idaho
Why are you assessing?Why are you assessing?
Most important question
– For Perkins IV?
– For Program Improvement?
– For student attainment?
If For Student AttainmentIf For Student Attainment
At what level is appropriate?
– Entry level
– Mastery level
– Industry certification level
When do you assess?When do you assess?
End-of-program
End-of-course
End of Pathway
Cluster related
Norm- or Competency-referencedNorm- or Competency-referenced
• Norm-referenced
– Comparison among individuals
– Demonstrates changes of students as a group
• Competency-referenced– Attainment of an
individual– Measurable learning
outcomes based upon industry standards
Buy or Develop?Buy or Develop?
1. Standardized instruments
2. Centralized state-developed instruments
3. Local assessments recognized by state
Performance or CognatePerformance or Cognate
• Performance– What the student
should be able to do
– Checklist– Limited number of
areas can be assessed in specified time
• Cognate– What the student
should know
– Test-items– Many areas can be
assessed in a specified time
Building an AssessmentBuilding an Assessment
• Adopt – seek items or assessments that you
can adopt as is
• Adapt – items or assessments that can be
modified to fit your standards
• Build – items or assessments when others
cannot be found
Table of Specifications (blueprint)Table of Specifications (blueprint)
• Determine competencies to be assessed
• Determine competency groups (3 – 10)
• Determine percentage of each cognitive typology per group
Development QuestionsDevelopment Questions
• Is cost and time worth benefit of test?
• Who should develop the assessment?
• Test security issues?
Multiple-choice elements:Multiple-choice elements:
Stem
Correct response
Foils or distractors
Fixed-form Fixed-form vv Variable-form Variable-form
Fixed-form
• High reliability• Needs high security
• Limited number of test items needed
Variable-form
• Good reliability• Lesser security
needed• Requires a large
number of test items
Assessment AdministrationAssessment Administration
• Decide the media for administration
• Identify who should conduct the assessment
• Determine when the assessment activities should occur
Reporting issues:Reporting issues:
1. Grading of the assessment
2. Assessment improvement needs
3. Reports needed
Assessment ReviewAssessment Review
• Validity – the degree to which certain inferences can be made from test scores.
• Reliability – the degree of consistency between two or more measures of the same thing.
Item & Assessment AnalysisItem & Assessment Analysis
Find a good statistician, psychometrician or educational psychologist !
Ethical Issues in TestingEthical Issues in Testing
Code of Fair Testing Practices in Education
by theJoint Committee on Testing Practices,
American Educational Research Association, American Psychological Association &
National Council on Measurement in Education
Legal IssuesLegal Issues
• Accommodations for members of special populations
• Access to the test-items by parents and others
• Utilization of assessments in relation to passing or graduation
Postsecondary Issues:Postsecondary Issues:
“Each eligible agency shall identify in the State plan core indicators of performance for career and technical education students at the postsecondary level that are valid and reliable, and that include, at a minimum measures of each of the following:”
Postsecondary Issues:Postsecondary Issues:
“(i) Student attainment of challenging career and technical skill proficiencies, including student achievement on technical assessments, that are aligned with industry-recognized standards, if available and appropriate.”
Postsecondary Issues: OVAEPostsecondary Issues: OVAE
• OVAE has denied the utilization of GPAs as an indicator
• OVAE preference is a third-party developed and administered assessment (e.g. NOCTI or centralized state assessment unit)
• OVAE has not defined what they will accept; burden of proof is on State
Postsecondary Issues: SystemicPostsecondary Issues: Systemic
• Local colleges seek to serve local business/industry – diversity of curriculum
• Cannot require anything beyond what the state requires for graduation – in TN most certification assessments are post-program
• Accreditation processes often evaluates the assessment process of a program of study
Postsecondary Issues: Challenge Postsecondary Issues: Challenge to OVAEto OVAE
• Goes against Perkins IV Sec. 8 “establishment” clause
• An unfunded mandate – many postsecondary receive little Perkins IV funds
• In pedagogy and andragogy, a single assessment is not sound
Postsecondary Issues: ReferencePostsecondary Issues: Reference
Wiseley, Chuck. (August 31, 2007) California Community Colleges NSWG response to the: Proposed DRAFT Guidelines for State Phase-in of Perkins IV Core Indicators 2S1 and 1P1 Technical Skills Assessments.
Contact Information:Contact Information:
Dr. John M. Townsend
Executive Director, Workforce Development
Office of Academic AffairsTennessee Board of Regents
1415 Murfreesboro Pike, Suite 350Nashville, TN 37217
615-366-4428