foundational perspectives on student development: implications for accountability and assessment...

Foundational Foundational Perspectives on Student Perspectives on Student

Development: Development: Implications for Implications for

Accountability and Accountability and AssessmentAssessment

George D. KuhGeorge D. Kuh

Council on University Planning and Analysis

Guelph, OntarioGuelph, OntarioJune 23, 2009June 23, 2009

ContextContext

Global Competitiveness in Degree Global Competitiveness in Degree AttainmentAttainment

The New Majority and Demographic The New Majority and Demographic GapsGaps

Questionable Levels of Student Questionable Levels of Student PerformancePerformance

ContextContext

Global Competitiveness in Degree Attainment

The New Majority and Demographic Gaps

Questionable Levels of Student Performance

In an Environment of Increasing Fiscal In an Environment of Increasing Fiscal Strain…Strain…

We Need Higher Levels of Student We Need Higher Levels of Student AchievementAchievement

Inquiring Minds Want to Know…Inquiring Minds Want to Know…

What frameworks and perspectives What frameworks and perspectives about student learning, development, about student learning, development, and success can be instructive in and success can be instructive in guiding assessment of essential guiding assessment of essential learning outcomes?learning outcomes?

How do they fit in with the current How do they fit in with the current press on institutional accountability? press on institutional accountability?

OverviewOverview

The context for accountability and assessment

Theoretical and empirical perspectives on student development and success

Implications

Accountability 2009Accountability 2009

Greater emphasis on student learning outcomes and evidence that student performance measures up

Demands for comparative measuresAssessment “technology” has made

great strides, but is insufficient to document learning outcomes most institutions claim

Increased calls for transparency ---public disclosure of student and institutional performance

Continuous Continuous ImprovementImprovement

AccountabilityAccountability

Strategic dimensions

Purpose Formative (improvement) Summative (judgment)

Stance Internal External

Predominant ethos Engagement Compliance

Implementation

Instrumentation Multiple/triangulation Standardized

Nature of evidence Quantitative and qualitative Quantitative

Reference points Over time, comparative, established goal

Comparative or fixed standard

Communication of results

Multiple internal channels Public communication, media

Use of results Multiple feedback loops Reporting

Two Paradigms of AssessmentTwo Paradigms of Assessment

Ewell, Peter T. (2007). Assessment and Accountability in America Today: Background and Context. In Assessing and Accounting for Student Learning: Beyond the Spellings Commission. Victor M. H. Borden and Gary R. Pike, Eds. Jossey-Bass: San Francisco.

Purposes of AssessmentPurposes of Assessment

EvidenceEvidence -- Demonstrate value and return on investment for accountability purposes

ImprovementImprovement – Determine what is working and what is not to improve programs, courses, departmental outcomes.

TransparencyTransparency -- Provide information about educational effectiveness and what students know and can do.

FeedbackFeedback -- Enable students to see their progress and learn how to improve.

What’s HappeningWhat’s Happening

Over past 5 years institutions report…Over past 5 years institutions report…

More assessment activities; greater attention to accreditation requirements

Emphasis on clearer statements of student learning outcomes, but little student understanding of outcomes

More emphasis on engaged learning practices including learning communities, service-learning courses, internships, senior capstones, and undergraduate research

Little transparency, even less documented improvement

(Ewell, 2008; Maki, 2004; Trends in Learning Outcomes, General Education, and Assessment, AAC&U, 2009)

TransparencyTransparency

AssessmentAssessment

ImprovementImprovement

AAccountability Triangleccountability Triangle

Student DevelopmentStudent Development

Outcomes: Host of Outcomes: Host of desirable skills, desirable skills, knowledge, beliefs and knowledge, beliefs and attitudes, persistence, attitudes, persistence, educational attainmenteducational attainment

Process: Unfolding of Process: Unfolding of human potential toward human potential toward increasingly complicated, increasingly complicated, refined functioningrefined functioning

Outcome DomainsOutcome Domains

Cognitive complexityCognitive complexity

Knowledge acquisition & Knowledge acquisition & applicationapplication

HumanitarianismHumanitarianism

Interpersonal and intrapersonal Interpersonal and intrapersonal competencecompetence

Practical competencePractical competence

Source: Kuh, 1993Source: Kuh, 1993

Cognitive complexityCognitive complexity

Cognitive skills including reflective Cognitive skills including reflective thought, critical thinking (e.g., thought, critical thinking (e.g., ability to summarize information ability to summarize information accurately and perceiving logical accurately and perceiving logical coherences and discernable coherences and discernable themes and patterns across themes and patterns across different sources of information), different sources of information), quantitative reasoning, and quantitative reasoning, and intellectual flexibility (i.e., openness intellectual flexibility (i.e., openness to new ideas and different points of to new ideas and different points of view).view).

Knowledge acquisition & applicationKnowledge acquisition & application

Understanding knowledge from a Understanding knowledge from a range of disciplines and physical, range of disciplines and physical, geographic, economic, political, geographic, economic, political, religious, and cultural realities, religious, and cultural realities, and the ability to relate knowledge and the ability to relate knowledge to daily life including using to daily life including using information presented in one class information presented in one class in other classes or other areas of in other classes or other areas of life.life.

HumanitarianismHumanitarianism

Understanding and appreciation Understanding and appreciation of human differences, including of human differences, including an increased sensitivity to the an increased sensitivity to the needs of others and needs of others and contributions to common good.contributions to common good.

Interpersonal & Interpersonal & intrapersonal competenceintrapersonal competence

A coherent, integrated A coherent, integrated constellation of personal attributes constellation of personal attributes (e.g., identity, self-esteem, (e.g., identity, self-esteem, confidence, integrity, appreciation confidence, integrity, appreciation for the aesthetic and spiritual for the aesthetic and spiritual qualities of life and the natural qualities of life and the natural world, sense of civic responsibility) world, sense of civic responsibility) and skills (e.g., how to work with and skills (e.g., how to work with people different from oneself).people different from oneself).

Practical competencePractical competence

Skills reflecting an enhanced Skills reflecting an enhanced capacity to manage one’s capacity to manage one’s personal affairs (e.g., time personal affairs (e.g., time management, decision making), management, decision making), to be economically self-sufficient, to be economically self-sufficient, and to be vocationally competent.and to be vocationally competent.

Association of American Colleges and Universities

Narrow Learning is Not Enough:Narrow Learning is Not Enough:The Essential Learning OutcomesThe Essential Learning Outcomes

Knowledge of Human Cultures Knowledge of Human Cultures and the and the Physical & Natural WorldPhysical & Natural World

Intellectual and Practical SkillsIntellectual and Practical Skills

Personal and Social ResponsibilityPersonal and Social Responsibility

““Deep” Integrative LearningDeep” Integrative Learning

Deep, Integrative LearningDeep, Integrative Learning

Attend to the underlying meaning of Attend to the underlying meaning of information as well as contentinformation as well as content

Integrate and synthesize different Integrate and synthesize different ideas, sources of informationideas, sources of information

Discern patterns in evidence or Discern patterns in evidence or phenomenaphenomena

Apply knowledge in different Apply knowledge in different situationssituations

View issues from multiple View issues from multiple perspectivesperspectives

Theoretical Perspectives: Theoretical Perspectives: Student DevelopmentStudent Development

Psychosocial (Chickering, 1968; Chickering & Reisser, 1993; Erikson, 1968; Cross, 1995)

Cognitive-structuralCognitive-structural (Perry, 1970; Kitchner & King, 1981, 1990; Baxter-Magolda, 1992; 2001; Kohlberg, 1969, 1984; Gilligan, 1977, 1986)

TypologyTypology (Cross, 1976; Kolb, 1976, 1984; Holland, 1968, 1997; Kuh et al., 2000)

Person-environment interaction Person-environment interaction (Astin, 1968, 1993; Barker, 1968; Horowitz, 1987; Pace & Stern, 1958, Moos, 1979

Theoretical Perspectives: Theoretical Perspectives: Persistence & Educational AttainmentPersistence & Educational Attainment

Sociological Sociological (Astin, 1977, 1993; Tinto, (Astin, 1977, 1993; Tinto, 1975, 1987, 1993; Marsden, 2004; Pascarella & Terenzini, 1991, 2005)

Organizational Organizational (Bean, 1983; Berger & Braxton, 1998)

Psychological Psychological ((Bandura, 1982; Bean & Eaton, 2000; Dweck, 2000)

Cultural Cultural ((Bourdieu & Passeron, 1977; Kuh & Love, 2000; Rendon et al, 2000; Tierney, 1992)

Economic Economic (Becker, 1964; Braxton, 2003))

Pascarella’s (1985) causal model: Effects of college student development

Student Student developmentdevelopment

Pascarella’s (1985) causal model: Effects of college student development

Student Student developmentdevelopment

StudentsStudents

•AptitudeAptitude

•AchievementAchievement

•PersonalityPersonality

•AspirationAspiration

•EthnicityEthnicity

Pascarella’s (1985) causal model

Student Student developmentdevelopment

InstitutionInstitution

•EnrollmentEnrollment

•Faculty-student Faculty-student ratioratio

•SelectivitySelectivity

•% Residential% Residential

StudentsStudents

•AptitudeAptitude

•AchievementAchievement

•PersonalityPersonality

•AspirationAspiration

•EthnicityEthnicity

Pascarella’s (1985) causal model

InstitutionInstitution

•EnrollmentEnrollment

•Faculty-student Faculty-student ratioratio

•SelectivitySelectivity

•% Residential% Residential

InteractionsInteractions with with

facultyfaculty

•peerspeers

Student Student developmentdevelopment

StudentsStudents

•AptitudeAptitude

•AchievementAchievement

•PersonalityPersonality

•AspirationAspiration

•EthnicityEthnicity

Pascarella’s (1985) causal modelPascarella’s (1985) causal model

InstitutionInstitution

•EnrollmentEnrollment

•Faculty-student Faculty-student ratioratio

•SelectivitySelectivity

•% Residential% Residential

InteractionsInteractions with with

facultyfaculty

•peerspeers

Student Student developmentdevelopment

Quality of Quality of student effortstudent effort

EngagementEngagement

Institutional Institutional EnvironmentEnvironment

StudentsStudents

•AptitudeAptitude

•AchievementAchievement

•PersonalityPersonality

•AspirationAspiration

•EthnicityEthnicity

Major Syntheses of Major Syntheses of Postsecondary Impact StudiesPostsecondary Impact Studies

Jacobs (1956)Jacobs (1956) Sanford (1962)Sanford (1962) Feldman & Newcomb (1969)Feldman & Newcomb (1969) Astin (1977) Astin (1977) Bowen (1977)Bowen (1977) Pace (1979) Pace (1979) Pascarella & Terenzini (1991)Pascarella & Terenzini (1991) Astin (1993)Astin (1993) Pascarella & Terenzini (2005)Pascarella & Terenzini (2005)

What Matters to Student SuccessWhat Matters to Student Success

Kuh, G.D., Kinzie, J., Buckley, J.A., Bridges, B.K., & Hayek, J.C. (2007). Piecing together the student success puzzle: Research, propositions, and recommendations. ASHE Higher Education Report, 32(5). San Francisco: Jossey-Bass.

Commissioned papers at:

http://nces.ed.gov/npec/papers.asphttp://nces.ed.gov/npec/papers.asp

• Study Habits

• Peer Involvement

• Interaction with Faculty

• Time on Task

• Motivation

• Other

• First Year Experience

• Academic Support

• Campus Environment

• Time on Task

• Peer Support

• Teaching & Learning

Approaches

• Other

Pre-college

experienc

es

Pre-college Characteristics Pre-college Characteristics Associated with Student SuccessAssociated with Student Success

Academic preparationAcademic preparation

Pre-college Characteristics Pre-college Characteristics Associated with Student SuccessAssociated with Student Success

Academic preparationAcademic preparationAbility and college-level skillsAbility and college-level skillsFinancial wherewithal Financial wherewithal Family education and supportFamily education and support

Early College Indicators of Early College Indicators of Persistence and SuccessPersistence and Success

Goal realizationGoal realization Psycho-social fitPsycho-social fit Credit hours completedCredit hours completed Academic and social supportAcademic and social support Involvement in the “right” kinds Involvement in the “right” kinds

of activitiesof activities

What Really Matters in University: Student EngagementStudent Engagement

Because iBecause individual effort and ndividual effort and involvement are the critical involvement are the critical determinants of impact, determinants of impact, institutions should focus on the institutions should focus on the ways they can shape their ways they can shape their academic, interpersonal, and academic, interpersonal, and extracurricular offerings to extracurricular offerings to encourage encourage student engagementstudent engagement. .

Pascarella & Terenzini, 2005, p. 602Pascarella & Terenzini, 2005, p. 602

Foundations of Student EngagementFoundations of Student Engagement

Time on task Time on task (Tyler, 1930s)(Tyler, 1930s)

Quality of effort Quality of effort (Pace, 1960-70s)(Pace, 1960-70s)

Student involvement Student involvement (Astin, 1984)(Astin, 1984)

Social, academic integration Social, academic integration (Tinto,1987, 1993)(Tinto,1987, 1993)

Good practices in Good practices in undergraduate education undergraduate education (Chickering & Gamson, 1987)(Chickering & Gamson, 1987)

College impact College impact (Pascarella, 1985)(Pascarella, 1985)

Student engagement Student engagement (Kuh, 1991, (Kuh, 1991, 2005)2005)

Student Engagement PropositionsStudent Engagement Propositions

What students What students dodo -- time and energy -- time and energy devoted to educationally purposeful devoted to educationally purposeful activitiesactivities

What institutions What institutions dodo -- using -- using effective educational practices to effective educational practices to induce students to do the right induce students to do the right thingsthings

Educationally effective institutions Educationally effective institutions channel student energy toward channel student energy toward the the right activitiesright activities

Good Practices in Good Practices in Undergraduate EducationUndergraduate Education

(Chickering & Gamson, 1987; (Chickering & Gamson, 1987; Pascarella & Terenzini, 2005)Pascarella & Terenzini, 2005)

Student-faculty contactStudent-faculty contact Active learningActive learning Prompt feedbackPrompt feedback Time on taskTime on task High expectationsHigh expectations Respect for diverse learning stylesRespect for diverse learning styles Cooperation among studentsCooperation among students

National Survey of National Survey of Student EngagementStudent Engagement(pronounced “nessie”)

Community College Community College Survey of Student Survey of Student EngagementEngagement(pronounced “cessie”)

College student surveys that assess the extent to which students engage in educational practices associated with high levels of learning and development

Student Engagement InitiativesStudent Engagement Initiatives

2,500,000+ students from 1,400 different schools

80+% of 4-yr U.S. undergrad FTE

50 states, Puerto Rico

64 Canadian IHEs

Australasian Survey of Student Engagement (AUSSE)

South African Survey of Student Engagement (SASSE)

Grades, persistence, student Grades, persistence, student satisfaction, and engagement satisfaction, and engagement go hand in handgo hand in hand

It’s more complicated than this…It’s more complicated than this…

Many of the effects of college are “conditional”

Some are compensatory

NSSE: NSSE: Who’s more engaged?Who’s more engaged?

WomenWomen Full-time studentsFull-time students Students who live on campusStudents who live on campus Students with diversity Students with diversity

experiencesexperiences Students who start and stay at Students who start and stay at

the same schoolthe same school Students in learning communitiesStudents in learning communities

Next StepsNext Steps

Develop additional indicators of Develop additional indicators of success for different types of success for different types of students.students.

Additional IndicatorsAdditional Indicators

course retentioncourse retentiontransfer student success transfer student success success in subsequent coursessuccess in subsequent coursesdegree/certificate completiondegree/certificate completiongraduate schoolgraduate schoolemploymentemploymentcapacity for lifelong learningcapacity for lifelong learningfocus on gaps in successfocus on gaps in success

High Impact ActivitiesHigh Impact Activities

First-Year Seminars and Experiences First-Year Seminars and Experiences  Common Intellectual ExperiencesCommon Intellectual Experiences Learning CommunitiesLearning Communities Writing-Intensive CoursesWriting-Intensive Courses Collaborative Assignments and ProjectsCollaborative Assignments and Projects “ “Science as Science Is Done”; Science as Science Is Done”;

Undergraduate Research Undergraduate Research Diversity/Global LearningDiversity/Global Learning Service Learning, Community-Based Service Learning, Community-Based

Learning Learning InternshipsInternships Capstone Courses and ProjectsCapstone Courses and Projects

Integrating ideas or information from various sources

Included diverse perspectives in class discussions/writing

Put together ideas from different courses

Discussed ideas with faculty members outside of class

Discussed ideas with others outside of class

Analyzing the basic elements of an idea, experience, or theory

Essential Learning Outcome: NSSE Deep/Integrative Learning

Synthesizing & organizing ideas, info., or experiences

Making judgments about the value of information

Applying theories to practical problems or in new situations

Examined the strengths and weaknesses of your own views

Tried to better understand someone else's views

Learned something that changed how you understand an issue

Effects of Participating in High-Impact ActivitiesEffects of Participating in High-Impact Activitieson Deep/Integrative Learning and Gainson Deep/Integrative Learning and Gains

Deep

Learning Gains

General Gains

Personal Gains

Practical

First-Year

Learning Communities +++ ++ ++ ++ Service Learning +++ ++ +++ ++

Senior

Study Abroad ++ + ++ Student-Faculty Research +++ ++ ++ ++ Internship ++ ++ ++ ++

Service Learning +++ ++ +++ ++ Culminating Experience ++ ++ ++ ++ + p < .001, ++ p < .001 & Unstd B > .10, +++ p < .001 & Unstd B > .30

Effects of Participating in High-Impact ActivitiesEffects of Participating in High-Impact Activitieson Student Engagementon Student Engagement

Level of Academic Challenge

Active and Collab.

Learning

Student-Faculty

Interaction

Supportive Campus

Env.

First-Year Learning Communities ++ +++ +++ ++ Service Learning ++ +++ +++ ++

Senior Study Abroad ++ ++ ++ + Student-Faculty Research +++ +++ +++ ++ Internship ++ +++ +++ ++

Service Learning ++ +++ +++ ++ Culminating Experience ++ ++ +++ ++ + p < .001, ++ p < .001 & Unstd B > .10, +++ p < .001 & Unstd B > .30

High Impact Activities High Impact Activities Increase Odds Students Will:Increase Odds Students Will:

Invest time and effort Invest time and effort Interact with faculty and peers Interact with faculty and peers

about substantive mattersabout substantive matters Experience diversityExperience diversity Get more frequent feedbackGet more frequent feedback Reflect & integrate learningReflect & integrate learning Discover relevance of learning Discover relevance of learning

through real-world applicationsthrough real-world applications

High-Impact Practices and the High-Impact Practices and the Disparities Within…Disparities Within…

Fewer 1Fewer 1stst gen students gen students Fewer students of colorFewer students of color Fewer transfer studentsFewer transfer students Fewer part-time studentsFewer part-time students Fewer older studentsFewer older students

Assessing Student Engagement inAssessing Student Engagement in High-Impact PracticesHigh-Impact Practices

To what extent does your institution provide these experiences? [√ = have on campus; √ = required; estimate the % of various

student populations in these activities]

www.aacu.org

Next StepsNext Steps

Develop additional indicators of Develop additional indicators of success for different types of success for different types of students.students.

Verify the institutional policies and Verify the institutional policies and practices that work best with different practices that work best with different groups of students at different types groups of students at different types of institutions. of institutions.

Further develop, validate, and refine Further develop, validate, and refine assessment tools.assessment tools.

Sample Data SourcesSample Data Sources• Locally-developed measures • National instruments

– National Survey of Student Engagement (NSSE)– Beginning College Survey of Student

Engagement (BCSSE)– Faculty Survey of Student Engagement (FSSE)– Cooperative Institutional Research Program

(CIRP)– Your First College Year (YFCY)– College Student Experiences Questionnaire

(CSEQ)– Noel Levitz Student Satisfaction Inventory– ETS MAPP and Major Field Tests– ACT Collegiate Assessment of Academic

Proficiency – Collegiate Learning Assessment (CLA)

• Institutional data -- GPA, financial aid, transcripts, retention, certification tests, alumni surveys, satisfaction surveys…

• Electronic portfolios

Next StepsNext Steps

Determine appropriate ways to Determine appropriate ways to measure, report, and use student measure, report, and use student success indicators for accountability success indicators for accountability and improvement (e.g., common and improvement (e.g., common template)template)

Student engagement varies Student engagement varies more more withinwithin than between than between institutions.institutions.

Variation With-in

Quality is not uniform within institutions. The lion’s share of the variation is among students, within institutions

Next StepsNext Steps

Determine appropriate ways to Determine appropriate ways to measure, report, and use student measure, report, and use student success indicators for accountability success indicators for accountability and improvement (e.g., common and improvement (e.g., common template)template)

Determine what postsecondary Determine what postsecondary institutions can realistically do at what institutions can realistically do at what cost to help academically cost to help academically underprepared students overcome the underprepared students overcome the deficiencies they bring with them to deficiencies they bring with them to college. college.

Let’s Do Four ThingsLet’s Do Four Things

1.1. Measure things that matter.

We value what we measure, so wise decisions are needed decisions are needed about about whatwhat to measure in the to measure in the context of campus mission, context of campus mission, values, and desired outcomes. values, and desired outcomes.

Employers On Accountability Challenge – December 2007 – Hart Research forEmployers On Accountability Challenge – December 2007 – Hart Research for

7%

33%

35%

46%

69%

Supervised internship/community-based projectSupervised internship/community-based project83%83%

79%79%

60%60%

56%56%

32%32%

Senior project (e.g., thesis, project)Senior project (e.g., thesis, project)

Essay testsEssay tests

Electronic portfolio & faculty assessmentsElectronic portfolio & faculty assessments

Multiple-choice testsMultiple-choice tests

Evidence of College Graduates Skills/Knowledge

Very effectiveVery effective Fairly effectiveFairly effective

Focus on mission-relevant, Focus on mission-relevant, educationally meaningful educationally meaningful indicators linked to student indicators linked to student success that the institution can success that the institution can do something aboutdo something about

Let’s Do Four ThingsLet’s Do Four Things

1.1. Measure things that matter.

2. 2. Use theory and research to Use theory and research to inform assessment effortsinform assessment efforts

3. 3. Adopt and adapt best practices Adopt and adapt best practices in assessment, public reporting, in assessment, public reporting, and improvementand improvement

4. 4. Resolve the perceived tensions Resolve the perceived tensions between improvement and between improvement and accountabilityaccountability

Continuous Continuous ImprovementImprovement

AccountabilityAccountability

Strategic dimensions

Purpose Formative (improvement) Summative (judgment)

Stance Internal External

Predominant ethos Engagement Compliance

Implementation

Instrumentation Multiple/triangulation Standardized

Nature of evidence Quantitative and qualitative Quantitative

Reference points Over time, comparative, established goal

Comparative or fixed standard

Communication of results

Multiple internal channels Public communication, media

Use of results Multiple feedback loops Reporting

Two Paradigms of AssessmentTwo Paradigms of Assessment

Ewell, Peter T. (2007). Assessment and Accountability in America Today: Background and Context. In Assessing and Accounting for Student Learning: Beyond the Spellings Commission. Victor M. H. Borden and Gary R. Pike, Eds. Jossey-Bass: San Francisco.

What’s Next for Student Learning What’s Next for Student Learning Outcomes Assessment? Outcomes Assessment?

Multiple measuresDiagnostic, milestone, and culminating

assessments all necessaryWide range of assessment practices –

basic to more complex assessments of learning environment, student work, direct learning outcomes, “value-added”

Measures of college-level learning by different majors, fields, levels

Demonstrate action on assessment resultsQuality improvement – measuring change,

real improvements in student learning

AAC&U VALUE RubricsAAC&U VALUE Rubrics

Inquiry and analysis Critical thinking Creative thinking Written communication Oral communication Quantitative literacy Information literacy Teamwork Problem solving Civic knowledge and engagement Intercultural knowledge and competence Ethical reasoning and action Foundations and skills for lifelong learning Integrative learning

AAC&U VALUE project – 12 RubricsAAC&U VALUE project – 12 Rubrics

Documenting what students learn and can do is of growing interest both on campus and with its many constituencies.

Know far too little about what actually happens in assessment on campuses around the country.

NILOA is conducting a survey and other activities to help fill that void.

NILOA’s mission is to identify best practices in assessment that institutions find useful.

www.learningoutcomesassessment.org

Questions Questions & &

DiscussionDiscussion