ASSIGNMENT DESIGN, MEASURING VALIDITY, AND CONTINUOUS IMPROVEMENT

LEAP Texas 2017 Annual ForumDr. Chris Duke, San Jacinto Collegechris.duke@sjcd.edu

Full-Scale Implementation

DECISIONS

• Recommended by faculty evaluators during the pilot year.• Implemented 2012-2013

Each core curriculum course has an assignment that is:• developed by full-time faculty teaching the course• assigned and assessed in each section of the course• aligned in Blackboard to the relevant outcomes

Signature Assignments

• Recommended by faculty evaluators during the pilot year.• Assignment details presented to evaluators by Bb Outcomes

Evaluator Access to Assignments

Outcome Selected Rubric

Critical Thinking Critical ThinkingQuantitative Literacy (3)Problem Solving (4)

Teamwork Teamwork

Empirical & Quantitative Reasoning Quantitative Literacy (3)Problem Solving (2)Inquiry and Analysis

Communication Written Communication(custom) Oral Communication

Social Responsibility Integrative Learning

Personal Responsibility Lifelong Learning

Selected LEAP VALUE Rubrics

CHALLENGES

Unusable Samples

.wps

Unusable Samples

Unusable Samples

Unusable Samples

Confounding Student Skill

• Signature assignments not implemented or used. • Signature assignments not designed within context of rubric.

Signature Assignment Challenges

SOLUTIONS

“Mailing it in”

Usability of Sample

VALUE ADDED

Identifies Sample Attrition

Identifies Confounding Skills

Estimates Adoption of Sig. Assign.

MEASURES VALIDITY

Outcome Selected Rubric

Critical Thinking Critical ThinkingQuantitative Literacy (3)Problem Solving (4)

Teamwork Teamwork

Empirical & Quantitative Reasoning Quantitative Literacy (3)Problem Solving (2)Inquiry and Analysis

Communication Written Communication(custom) Oral Communication

Social Responsibility Integrative Learning

Personal Responsibility Lifelong Learning

Selected LEAP VALUE Rubrics

(1) Revise the rubric

or

(2) Revise the assignment

Improving Validity

All criteria relevant to Personal Responsibility and Social Responsibility.Faculty selected criteria most appropriate to assess the outcome based on local and state definition.

Closed Card Sort

Outcome Selected Rubric

Critical Thinking Critical Thinking

Teamwork Teamwork

Empirical & Quantitative Reasoning Quantitative Literacy (modified)

Communication Written Communication w/ VisualOral Communication w/ Visual

Social Responsibility LEAP VALUE Criteria

Personal Responsibility LEAP VALUE Criteria

Selected Rubrics (AY 2017)

Validity Worksheet for Assignment Design

Usability of Sample Criterion

Chris Duke, Ph.D.Assistant Vice Chancellor for Accreditation and AssessmentSan Jacinto College chris.duke@sjcd.edu

“Usability of Sample” Criterion San Jacinto College developed the “Usability of Sample” criterion and added it to each AAC&U LEAP VALUE rubric the College has implemented for general education outcomes assessment. The “Usability of Sample” is dependent upon evaluators having access to the assignment to which a student responded. When evaluators are assessing samples of student work, the criterion provides a method for indicating methodological issues or concerns that may confound the observed results.

The levels of the criterion are separate and distinct categories; it is not a scale in which 4 is better than 3 is better than 2 and so on. For statistician and assessment types, it is a nominal scale not an ordinal scale like the rest of the rubric. The categories being presented as “Level #” is only an artifact of the criterion being wedged into the existing LEAP VALUE rubric structure.

Level 4, No Confounding Issues No issues encountered. Level 3, Other Confounding Student Skill The student responded in a manner that interfered with reliable assessment of this outcome, e.g. student wrote much less than assignment required; or, poor writing skills interfere with assessing a non-writing outcome. Level 2, Validity Concern (signature assignment not used) The assignment was not the signature assignment expected for the course, e.g. entirely different assignment, excessively modified common assignment, mistaken alignment, or presented as not part of regular course grade. Level 1, Validity Concern (signature assignment not aligned to rubric) There was a disconnect between the rubric and the signature assignment, e.g. student may or may not have performed well, but the signature assignment did not ask the student to perform in a manner expected by the rubric. Level 0, Document Unusable This document was not access-able or not assessable: e.g. wrong file format, unable to open the file, illegible/unreadable, unexpected teamwork, or instance of plagiarism.

Title: Assignment Design, Measuring Validity, and Continuous Improvement

Type: Concurrent

Track: Designing Effective Assignments and Signature Work Successfully Implementing the VALUE Rubrics

Abstract:

San Jacinto College believes a locally designed “Usability of Sample” criterion added to each rubric is essential to the continued effectiveness of the full-scale implementation of LEAP VALUE-based assessment. The “Usability of Sample” criterion embeds within the juried assessment process by faculty evaluators a method for measuring the validity of the assessment process. The implementation of the criterion has yielded indispensable data that has formed the foundation for continuous improvements focused on assignment design and rubric selection. This session will discuss specific examples of observed data and the resulting continuous improvements. A copy of the criterion will be provided, and recommendations for implementation based on lessons learned will be offered. Description:

San Jacinto College has implemented LEAP VALUE rubric-based assessment of general education outcomes at full scale across the entire core curriculum. A team of faculty teaching each core curriculum course developed a signature assignment that is assigned to students in every section of the course. Given the scope of implementation across 90+ courses, faculty evaluators encountered instances in which the signature assignment did not require students to perform in all manners expected by the rubric; in short, the assignments have not always been a valid assessment of the aligned outcome as defined by the rubric used to assess the outcome. To address that challenge, the college created and added to each rubric a “Usability of Sample” criterion. That criterion allows each evaluator to indicate for each sample assessed (a) the signature assignment for the sample was a valid assessment of the outcome, (b) the signature assignment for the sample was not aligned with the rubric, (c) a student skill other than that being assessed confounded the assessment, (d) an assignment other than the expected signature assignment was used, or (e) the student document could not be assessed. The “Usability of Sample” criterion has provided indispensable data that has indicated the need for and supported critical continuous improvements to the college’s assessment process. Based on faculty interpretation of observed “Usability of Sample” data, the college has (1) reviewed and revised rubrics selected to assess personal responsibility, social responsibility, critical thinking, and empirical and quantitative skills, (2) implemented as part of the annual assessment process an opportunity for faculty to review and to revise signature assignments, and (3) developed a “Validity Worksheet” to support that design and revision process. A copy of the criterion will be provided, and recommendations for implementation based on lessons learned will be offered. Brief biographical sketch(es) of the presenter(s):

Dr. Chris Duke is the Assistant Vice Chancellor for Accreditation and Assessment at San Jacinto College; he currently serves as a LEAP Texas Assessment Fellow and as a member of the LEAP Texas Board of Directors. Chris has facilitated at San Jacinto a full-scale, core-curriculum wide implementation of the LEAP VALUE rubrics to assess the Texas core objectives, and he serves as a LEAP Texas Assessment Fellow in which he co-facilitates the first LEAP Texas inter-institutional assessment project. Chris holds a Ph.D. in educational psychology from Texas A&M University.