life after vsa: an update oklahoma association of institutional research and planning fall 2009...
TRANSCRIPT
Life After VSA: Life After VSA: An UpdateAn Update
Oklahoma Association of Institutional Research and Planning
Fall 2009 Conference
October 23, 2009
Voluntary System of Voluntary System of Accountability (VSA)Accountability (VSA)
Initiative for public universities to supply basic, comparable information on the undergraduate student experience to important constituencies through a common web report – the College Portrait.
Sponsored by APLU & AASCU. Funded by the Lumina Foundation for Education.
• Provide a useful tool for students during the college search process
• Assemble information that is transparent, comparable, and understandable
• Demonstrate accountability and stewardship to public
• Measure educational outcomes to identify and enhance effective educational practices
VSA GoalsVSA Goals
• Prospective students and families• High school, community college counselors
• Policymakers and elected officials
• Campus faculty and staff
• Accrediting associations, institution and state boards
• Taxpayers and public-at-large
Multiple AudiencesMultiple Audiences
VSA ParticipationVSA Participation
• 329 institutions from 49 states and Puerto Rico, including 7 from Oklahoma (10/05/09)
• 63% of 520 APLU & AASCU institutions
• 250+ College Portraits posted on www.collegeportraits.org
VSA ContextVSA Context
• Perceived lack of useful and transparent data preventing institutions from demonstrating accountability and contribution to public good (Spellings Commission)
• Policymakers and employers demand evidence of educational outcomes – particularly broad transferable skills
A Community EffortA Community Effort
• Concept outlined in APLU white papers, refined through feedback from members
• Strengthened by AASCU endorsement
• Structure developed by task forces - presidents, provosts, student affairs, IR – 82 members from 70 institutions
– Input from associations and larger HE community
Result: College PortraitResult: College Portrait
• Student & Family Information• Student Experiences & Perceptions• Student Learning Outcomes
Free web report that communicates user-friendly, consistent information about college campuses.
www.collegeportraits.orgwww.collegeportraits.org
Student & Family InformationStudent & Family Information
• Institution description, highlights• Student characteristics• Cost of attendance, net cost calculator • Success & progress rate• Degree programs, class sizes• Post graduation plans
Campus Defined Buttons
Introduction to Campus Community
Menu links take users to each section and data element
Developed as alternative to IPEDS graduation rate
Focuses on student success in higher education system
Success & Progress Rate
Tracks student progress across 2-year and 4-year institutions using NSC data
New cohort query available to all Student Tracker users
Student Experiences & PerceptionsStudent Experiences & Perceptions
– group learning
– interaction with faculty
– institutional commitment
– active learning
– diversity experiences
– student satisfaction
• Selected senior responses from one of 4 surveys: NSSE, CSS, CSEQ, UCUES
• Snapshot of student experiences on campus within 6 common areas
6 Common Constructs
Selected responses
from Seniors
Reported in a common format using CAAP, CLA or MAPP
Student Learning Outcomes
Learning gains in critical thinking,
analytic reasoning, written
communication measured at
institution level
Student Learning Outcomes
Test Validity Study
Funded by the Fund for the Improvement of Post-Secondary Education (FIPSE).
Background• VSA measures value-added for student
learning outcomes of critical thinking and written communications.
• VSA offers a choice of three tests of student learning outcomes: CAAP, CLA and MAPP.
• Testing companies agreed to adopt the same value-added methodology and reporting format, controlling for entering student quality.
• Validity study needed to ensure that the three tests produced consistent scores.
Methodology• Students take each of the three tests.• 13 tests administered to 1,100+ students at 13
universities
• 13 tests components of CLA, CAAP, MAPP– 4 tests of critical thinking
– 2 tests of reading
– 2 tests of mathematics
– 4 tests of writing
– 1 test of science
Participating Universities
• Alabama A&M• Arizona State• Boise State• California State-
Northridge• Florida State• MIT• Trinity College• University of Colorado-
Denver
• University of Michigan, Ann Arbor
• University of Minnesota, Twin Cities
• University of Texas, El Paso
• University of Vermont• University of Wisconsin,
Stout
Conclusions• Validity and reliability are high across tests and
should not affect choice of measure
• Factors affecting adoption:– Acceptance by students, faculty, administrators
or other policy makers– Trade-offs in cost, easy of administration, etc– Utility of the test for other purposes such as
supporting campus activities and services or providing guidance on improving learning
• Thus VSA will continue to permit universities to choose to use either CLA, CAAP or MAPP.
Final Reports
Two reports are available on the VSA website:
http://www.voluntarysystem.org/index.cfm
1. The complete (and technically complex) Test Validity Study full report
2. An interpretative report for VSA participating schools can be found at
HARTRESEARCH
P e t e r D
A S S O T E SC I A
How Should Colleges Assess & Improve College Learning?
Employers’ Views on the Accountability Challenge
Key findings from survey among 301 employersConducted November 8 – December 12, 2007
for
2424
Employers Find College Transcripts Of Employers Find College Transcripts Of Limited Use In Evaluating PotentialLimited Use In Evaluating Potential
16%
34%
33%
4% 13%
How useful do you find the college transcript in helping you evaluate job applicants’ potential to succeed at your company?
Very useful
Fairly useful
Just somewhat useful
Not useful
Not sure
2525
7%
33%
35%
46%
69%
Supervised/evaluated internship/community-based project where students apply college learning in real-world setting
83%
79%
60%
56%
32%
Advanced comprehensive senior project, such as thesis, demonstrating student’s depth of knowledge in major & problem-solving, writing, and analytic reasoning skills
Essay tests to evaluate level of problem-solving, writing, and analytical-thinking skills
Electronic portfolio of student’s college work, including accomplishments in key skill areas and faculty assessments
Multiple-choice tests of general content knowledge
Assessments’ Effectiveness In Ensuring College Assessments’ Effectiveness In Ensuring College Graduates Have Skills/KnowledgeGraduates Have Skills/Knowledge
Very effective Fairly effective
2626
Assessments’ Usefulness In Helping Employers Assessments’ Usefulness In Helping Employers Evaluate College Graduates’ PotentialEvaluate College Graduates’ Potential
6%
14%
23%
30%
30%
40%
Faculty supervisor’s assessment of applicant’s student internship/ community-based project applying college learning in real-world setting
67%
61%
56%
54%
36%
Sample of applicant’s student senior project and overview of faculty assessment of the project
Applicant’s score on essay test to evaluate level of problem-solving, writing, and analytical-thinking skills
Electronic portfolio of applicant’s college work, including accomplishments in key skill areas and faculty assessments
Applicant’s score on multiple-choice test of general content knowledge
Very useful Fairly useful
Applicant college’s score showing how the college compares to others in advancing students’ critical-thinking skills
29%
An obvious place to begin:
Tell The Students What They Are Expected to Accomplish
The Power of Rubrics for Assessing Learning
• Help guide students around learning expectations
• Help guide faculty • Reinforces assessment for learning• Places individual faculty judgment within
national shared experience• Can build up from course level to institutional
reporting needs AND down from general to specific program/course context
Openedpractices.org
• A website open to everyone in higher education – Free to join, easy to sign up– openedpractices.org
• A library of sorts…a site where a collection of rubrics (and other best practices) can be:– Stored– Shared– Tagged
• Built with open source software to share knowledge and best practices openly
VALUE Process
• Develop criteria from rubric collection
• Each rubric development team member wrote a row of performance descriptors for one criteria
• Draft rubric then discussed and debated, edited and transformed
• Testing and feedback – e-Portfolios
Voluntary Framework of AccountabilityVoluntary Framework of Accountability
• Initiative to report comparable information on community college
• Use measures better suited to community colleges
• Funded by the Bill & Melinda Gates Foundation and the Lumina Foundation for Education
• Sponsored by American Association of Community Colleges (AACC), College Board and Association of community College Trustees
OklahomaOklahoma
• Encourage participation in VSA
• Participate in Achieving the Dream
• Revise Assessment Policy
• Use measures in Brain Gain funding
Dr. Debra L. Stuart
Vice Chancellor for Educational Partnerships
405-225-9168