using value rubric results for learning improvement...

Assessment that Empowers Faculty to Take Risks with Pedagogical Innovation

LEAP TexasTerrel L. Rhodes

Association of American Colleges and UniversitiesMarch 25, 2018

Using VALUE Rubric Results for Learning Improvement, Professional Development and Equity:

Course Level Teaching,

Learning & Assessment

Teaching &

Learning

Policy & Quality

Assurance

The Traditional Assessment & Accountability Landscape

Program Assessment

Disciplinary and/or

Professional Accreditation

General Education/

Core Curriculum Assessment

Institutional Assessment

Regional Accreditation

System and/or State

Level Accountability

Federal Level Accountability

Other Accountability Mechanisms

(e.g., VSA)

Movement from course-embedded and program-level assessment to more global, institutional assessment = increase cost, harder to tie results directly to improving teaching & learning at the local level; quality assurance mechanisms evidence generated requires valid and reliable measures that transcend local conditions in order to set effective policy. The traditional measure preferred at the policy level – commercially available standardized tests – lack transparency in design and the ability to disaggregate data below the institutional level to make changes to improve teaching and learning at the course and program level.

Course-LevelRecognize and promote student agency and faculty development and expertise in order

to improve teaching and through the adoption of active learning pedagogies and

enhanced assignment design

Institutional LevelCreate guided learning pathways – including successful 2- to 4-year transfer - to promote retention and completion for all students,

while addressing quality assurance and accountability requirements through

general education and beyond

Program Level Design curricula that leverage high-Impact practices within and across degree areas that respect disciplinary paradigms and

professional standards while promoting the attainment of higher order necessary

abilities to thrive in work, citizenship, and life for all students

Policy LevelTo create a common language of evidence

that facilitates collaboration across the triad – system/state, federal, and regional

accreditation – and enables the development of sound public policy to

promote individual student success and educational attainment for the

common good

The VALUE Model -Evidence of quality student learning to:

The Key Elements for a Compelling Quality Framework Already Are in

Hand

Consensus Aims and Outcomes

Practices that Foster Achievement AND Completion

Evidence on “What Works” for Underserved Students

Assessments That Raise – and Reveal – the Level of Learning

The VALUE Rubric Approach to Assessing Student Learning

Valid Assessment of Learning in Undergraduate Educationwww.aacu.org/value

http://www.aacu.org/value

VALUE Rubric Approach - Assumptions

Learning is a process that occurs over time Student work is most robust representation of student

motivated learning Focus on what student does in terms of key

dimensions of learning outcomes Faculty and educator expert judgment Results are useful and actionable for improvement of

learning

VALUE Embraces Imperfection as Part of the Learning Process

“Never Let the Perfect Get in the Way of the Good”

VALUE embraces the variables that other assessment approaches control or eliminate in their consideration of

student learning, including: Individual, faculty-designed assignments taken straight off

the syllabus and out of the classroom. There are no required common prompts.

An approach to sampling that is designed to raise up, not wash out, the inherent diversity—from race, ethnicity, and socioeconomic status to the diversity of courses, credit-levels, and disciplinary backgrounds—found on campuses.

Scorer training sessions that are equal parts calibration to reach a consensus score and a rich faculty development opportunity, and that are open to all faculty whether they are contingent or tenure-track, two-year or four-year, curricular or co-curricular.

www.aacu.org/OnSolidGroundVALUE

http://www.aacu.org/OnSolidGroundVALUE

VALUE Project map: The Multi-State, Minnesota, and Great Lakes Colleges Association Collaboratives

Multi-state Collaborative

Multi-state and Minnesota Collaboratives

VALUE Initiative to Date:

92 institutions submitted 29,000 student work products for assessment by 400 faculty using VALUE rubrics.

Faculty & staff saw the VALUE rubrics as valid.

Percent of scorers who reported Strongly Agree or Agree with each aspect of rubric use

75%

80%

83%

86%

89%

Encompassed meaning of outcome

Descriptors were relevant

Descriptors were understandable

Scoring levels provided sufficient range

Useful for evaluating student work

These results are not generalizable across participating states or the nation in any way. Please use appropriately.

Criteria

The Anatomy of a VALUE Rubric

Levels

Performance Descriptors

Profile of Scorers by Discipline and/or Institutional Role

Arts and Humanities

Natural and Applied/Formal Sciences

Professions

Social Sciences

Administrative

MSC Profile of VALUE Scorers


Potential to disaggregate by demographic characteristics

1.0

1.5

2.0

2.5

Asian Black Hispanic White

Critical Thinking scores by race

2 year 4 year


Asian Black Hispanic White

Critical Thinking scores by Pell eligibility

1.0

1.5

2.0

2.5


2 year 4 year

Not Eligible Pell Eligible Not Eligible Pell Eligible

Faculty & staff saw the VALUE rubrics as valid

Percent of scorers who reported Strongly Agree or Agree with each aspect of rubric use

75%

80%

83%

86%

89%

Encompassed meaning of outcome

Descriptors were relevant

Descriptors were understandable

Scoring levels provided sufficient range

Useful for evaluating student work


Interrater reliability was moderate to strong.

ACCOUNTABILITY AND ACCREDITATION

Campus Benefits and Uses

The reflective essay prepared by Southern Connecticut State University outlined the institution’s “significant” national and international assessment initiatives as well as the “major internal assessment initiatives” undertaken to assess general education and provide support for internal program reviews and specialized accreditation reviews. We are pleased to learn that results gleaned from analyses of student work conducted as part of SCSU’s participation in the Multi-State Collaborative have been used to inform the restructuring of the University’s access programs, developmental math curriculum, liberal education program, and writing across the curriculum program. We are also gratified to learn of SCSU’s use of the results of a longitudinal cohort study of retention rates to determine the “most important predictors of academic success and student retention” and to develop programming to foster the “habits of mind” that are predictive of success. The essay provided evidence that SCSU graduates are successful in their chosen fields, as measured by success in clinical placements. licensure passage rates, and employer evaluations and satisfaction rates.

CAPACITY BUILDING


“It’s a professional development opportunity for all of us,”

CTL

IR

FacultyStudent Affairs

Committee Chairs

“..to see how the decisions made by an assessment committee affect how institutional research is able to collect or analyze the data…”

“…How...data are likely to be used in conversations [with faculty] about curriculum and development…”

“…gather and talk about something as important as learning outcomes.”

“…a rare opportunity for all these different players…”

The Hamline Plan is not a set number of courses. Instead, it focuses on certain skills that you can learn in a variety of

subjects.

• “I’ll also say that we are greener than [St. Olaf College] and when we began, their faculty director of assessment was very useful for us to think about how they had built this robust assessment structure that moved from pretty effective program-level assessment to a cross-program general education system…”

“ Participation in VALUE has diversified and expanded how we understand learning. “

Hamline University

one-hour assessment charrette and offered it as a companion to an “assessment salon” – feedback on assignments and discussion of assessment.

• one whole department talked about how they are collecting data . . . and working alongside career services, counseling, and advising to figure out if students are on the right pathway.”

Math faculty

French faculty

the faculty really enjoyed. . . sharing

what they were doing

they would like to do something like this again…grow a stronger culture

of assessment at the college.

PROGRAM AND COURSE IMPROVEMENT


Applying lessons learned to our local initiatives on campus

• MSC provided a means for better general education assessment.• Potential:

– Sustainable model to assess general education– Less work for individual faculty member (than current model)– Broader faculty engagement across campus– Assess what a student knows at 90+ credits –use major and gen

ed courses.– Use results for VSA/College Portrait???

• Mini MSC- Assessment Retreat –CCSU faculty scoring same artifacts– Compare home scores to MSC scores – Very manageable, especially with use of Taskstream for

uploading and scoring of artifacts.

Learning Outcome & Faculty Faculty (N)

Number of Artifacts

First Year Soph Junior Senior

CCSU Total

MSC Total

Critical Thinking (33 Majors)

12 16 21 58 130 225 119

Quantitative Reasoning (19 majors)

5 0 6 29 82 117 78

Written Communication (28 Majors)

13 13 19 62 97 191 87

Grand Total(45 majors – 75% of majors)(27 faculty – 45% of dept)

27 29 46 140 318 533 283

Reflections on the Pilot Year

“…data…collected over the last eight or nine years to look at programs “more holistically” and

evaluate staffing, course sequencing, or program-wide curricula…”

• Math, Statistics, and Computer Science department…started using a statistical software manual “because they realized that students didn’t have quite the competence level … as they wanted [them] to have when they graduated…”

Institutional Data

Criterion by Criterion

Focus

Sources and Evidence

De-Identified

Project and Institution

Took Apart the Rubrics

Action

Project • Project as a Whole

Institutional • Local Scoring

Course • Individual Faculty

Research highlights importance of faculty and

student success and equity

33

Assignment Difficulty

INTRODUCE Assignment designed

to introduce the outcome

PRACTICE Assignment designed

to afford student practice with the

outcome

REINFORCE Assignment designed

to reinforce previously practiced

outcome

MASTERY Assignment designed

for students to demonstrate level of

mastery of the outcome

1 2 3 4 5 6 7 8

Critically Important to Collect

INTRODUCE

Assignment designed to introduce the outcome

PRACTICE

Assignment designed to afford student practice with the outcome

REINFORCE

Assignment designed to reinforce previously practiced outcome

MASTERY

Assignment designed for students to demonstrate level of mastery of the outcome

1

2

3

4

5

6

7

8

Analysis of student work assessed using the Critical Thinking and Written Communication VALUE Rubrics, seniors’ work was significantly more likely to be scored at the “Capstone” level—the highest level of performance—when the assignment was designed to produce work at the Capstone level.

When the assignment was “easier” seniors’ performance, on average, went down to meet the lower expectations of the assignment.

When asked for less, they produced less.

non-white and lower income students gain access to high quality and

demanding assignments less often

Inclusive excellence is an affordable, feasible goal if the highest impact high-impact practice is

high quality and appropriately demanding assignments.

Achieving it would transform the learning outcomes of American higher

education.

Our notion of high-impact practices consists

primarily of things like student-faculty research,

overseas study, and participating in learning communities—practices

where increasing the supply is costly in higher

education

But insisting that all faculty give assignments that are both demanding

and intentional about higher-order learning goals not just content learning goals is not

financially costly, only politically and

managerially difficult.

We need to begin thinking of high quality and demanding assignments as perhaps the highest

impact and lowest cost high-impact practice.

Lessons Learned from VALUE/MSC

• Context or landscape is important• Local data are critical• Data need deconstruction/disaggregation at local

level• Interdisciplinary/integrative experience is required to

attain high quality levels associated with graduation• What faculty/educators do is foundational to achieve

quality student learning

Questionsor

Comments?

[email protected]

Assessment that Empowers Faculty to �Take Risks with Pedagogical Innovation�Slide Number 2Slide Number 3The Key Elements for a Compelling Quality Framework Already Are in HandThe VALUE Rubric Approach to Assessing Student LearningVALUE Rubric Approach - AssumptionsSlide Number 7VALUE embraces the variables that other assessment approaches control or eliminate in their consideration of student learning, including:�Slide Number 9Slide Number 10VALUE Initiative to Date:Faculty & staff saw the VALUE rubrics as valid.Slide Number 13Slide Number 14Slide Number 15Slide Number 16Slide Number 17Faculty & staff saw the VALUE rubrics as validSlide Number 19Accountability and Accreditation�Slide Number 21Capacity Building�Slide Number 23�The Hamline Plan is not a set number of courses. Instead, it focuses on certain skills that you can learn in a variety of subjects.Slide Number 25one-hour assessment charrette and offered it as a companion to an “assessment salon” – feedback on assignments and discussion of assessment.�Program and Course improvement�Slide Number 28Reflections on the Pilot Year“…data…collected over the last eight or nine years to look at programs “more holistically” and evaluate staffing, course sequencing, or program-wide curricula…”Slide Number 31Slide Number 32Research highlights importance of faculty and student success and equityAssignment DifficultySlide Number 35Slide Number 36Slide Number 37Lessons Learned from VALUE/MSCQuestions�or�Comments?

using value rubric results for learning improvement...

Documents