terry rhodes: show me the learning: valid assessment of learning in undergraduate education (value)
Post on 15-Dec-2014
3.037 Views
Preview:
DESCRIPTION
TRANSCRIPT
Show Me the Learning: Valid Assessment of Learning in Undergraduate Education
(VALUE)
Terrel L. RhodesAssociation of American Colleges and
UniversitiesWASC Resource Fair
January 2012
Survey Results Are Consistent
Everyone – business, policy makers, faculty, students - wants better learning AND
better information, compelling evidence on what students know and are able to do…
whether for personal development, program assessment, accreditation or
hiring a new employee.
Major VALUE Project Activities
• National advisory board [12 members]• Rubric collection and creation of 15
metarubrics by teams [over 100 individual faculty and others]
• Piloting and refining metarubrics through three cycles of leadership campus use (using e-portfolios of student work [over 100 campuses, including 12 leadership campuses]
• Final reliability and ease-of-use check with national panel of 40 academics, employers, teachers, community members
Outcomes for the development of
metarubrics:• Inquiry and analysis • Critical thinking • Creative thinking • Written communication • Oral communication • Quantitative literacy • Information literacy• Reading • Teamwork • Problem solving • Civic knowledge and engagement—local and
global • Intercultural knowledge and competence • Ethical reasoning and action • Foundations and skills for lifelong learning • Integrative learning
Commonalities among rubrics
Motivated by: Need for among-campus communicationMobile students, transferBelief that, in spite of uniqueness, core outcomes are shared
Rubrics Basics
Criteria
VALUE Rubrics & AssessmentVALUE Rubrics & Assessment
Rubrics Basics
Levels
VALUE Rubrics & AssessmentVALUE Rubrics & Assessment
Rubrics Basics
Performance Descriptors
VALUE Rubrics & AssessmentVALUE Rubrics & Assessment
Validity and Usability• Over 3000 distinct institutions have downloaded one
or more of the VALUE rubrics for use since fall 2010• Over 11,000 distinct individuals have downloaded one
or more of the VALUE rubrics for use• 3 major consortia are using VALUE rubrics for cross
institutional collaboration – Connect2Learning – LaGuardia College/AAEEBL (FIPSE) – 23 campuses; Integrative Portfolio Process – Michigan (FIPSE) – 6 campuses; RAILS – Syracuse (Institute for Museum and Library Studies ACRL) – 10 campuses
Reliability Study• 40 Faculty• 4 Traditional Disciplinary Divisions –
Humanities, Social Sciences, STEM, Professions
• Three VALUE rubrics – Critical Thinking, Civic Engagement, Integrative Learning
• Common set of student portfolio work• Agreement = .66 without norming; .8
normed• Another set of 5 campuses, using same set
of rubrics with 500 samples of student work – still analyzing
Building the Evidentiary Base
• University of Kansas – Representing Results
Per
cen
t o
f R
atin
gs
Critical Thinking: Issues, Analysis, and Conclusions
Inter-rater reliability = >.8
Building the Evidentiary Base
• University of Kansas – Representing Results
Per
cen
t o
f R
atin
gs
Critical Thinking: Evaluation of Sources and Evidence
Building the Evidentiary Base - Kansas
Per
cen
t o
f R
atin
gs
“VALUE added” for 4 years - writing
Building the Evidentiary Base
• University of Kansas – • “analysis of the data from the AACU VALUE rubrics
affirmed that a team approach to course design can yield larger improvement in some forms of student writing and thinking”
• “We also saw that the rubrics work best when there is close alignment between the nature of the assignment and the dimensions of intellectual skill described in the rubric”
• “Finally, at a practical level we are very encouraged that this process is manageable and sustainable”
Roanoke College
Writing
Reliability = >.8
Using the Results“…we excluded the scores for those instructors and ran
frequencies and descriptive statistics on the categories again. We found the means for the rubric categories of Focus and Thesis and Organization remained close to 2.5 with the scores of 2 and 3 occurring most often (approx. 2.4) while most scores for the categories of Evidence and Reasoning were 2s. The category of Style and Mechanics was in the middle with a mean of 2.3. Our adjusted results support what most faculty believe about the writing of our first-year students, they can learn to develop a thesis and organize a paper more effectively than they can provide convincing evidence and strong reasoning to support the thesis.”
Building the Evidentiary Base
University of North Carolina - Wilmington
Dimension % of Work Products Scored 2 or higher
% of Work Products Scored 3 or Higher
IL1 Determine Information Needed 87.2% 46.2%
IL2 Access Needed Information 89.6% 46.8%
IL3 Evaluate Information and Sources 88.5% 39.7%
IL4 Use Information Effectively 85.9% 43.6%
IL5 Access and Use Information Ethically 93.6% 59.0%
Table 1. Information Literacy Results
Table 1 Information Literacy ResultsTable 1 Information Literacy ResultsTable 1 Information Literacy Results
Inter-rater reliability = >.8
Alverno CollegeVideo of oral presentations
First year and last year
Carleton CollegeSophomore portfolio
In the disciplines
Dissemination• Dozens of presentations at accreditation and assessment conferences (US and abroad) and workshops• Two issues (winter 2009, winter 2011/12) of Peer Review on VALUE rubrics = 9,200 each, 1200 institutions• Two Publications: 1) Rubrics = 6500+; 2)e-Portfolios = 2800+ since early 2010 (about to be a book from Jossey-Bass)• Virtually all vendors of e-portfolio software products have adopted VALUE rubrics• Degree Qualifications Profile [DQP] (Lumina)• Being used worldwide, e.g. Japan, Hong Kong, Australia, United Arab Emirates, Korea• Major state university systems are using VALUE rubrics
Contacts
Terrel L Rhodes, AAC&U Vice Presidentrhodes@aacu.org
http://www.aacu.org/value
top related