lessons learned from industry: achieving diversity & efficacy in college success ets - college...

Lessons Learned from Industry: Achieving Diversity & Efficacy in

College Success

ETS - College Board Invitational Conference

Washington, DC

Wayne Camara & Krista Mattern

September 8, 2008

2

Job Analysis• Organizations use job analysis to determine what

work outcomes are desired.

• Sample individual outcomes (productivity, job performance, retention) and organizational outcomes (efficiency, quality, innovation, team work)

• Identify performance components (pc)pc = {Declarative knowledge x Procedural knowledge x

Motivation}

Knowledge x Cognitive skills x Level of effort Goals Interpersonal “ Persistence “

Ability, Interests, Education, Experience Importance & Prob. Of Outcomes

Predictors

3

Job Analysis – College Success1. Identify desired performance outcomes

for individuals and organizations (college success) (GPA, return, graduate, life after college – grad school, certification)

Each outcome likely has different predictors

2. Identify performance tasks associated with outcomes (persistence, academic ability, health, engagement)

3. Identify or develop performance measures (GPA, advisory ratings, self report data, dB of student engagement)

4

Predicting Performance

Goldstein, Zedeck, & Schneider (1993)

5

Group differences are not unique to tests: They are present across most educational measures

6

Large Mean Differences Persist on Cognitive Ability Tests by

Race/Ethnicity Remain

SA

T

CR

+M

7 Source: U.S. Census Bureau

College-Going Rates of High School Graduates Aged 18 to 24 by Ethnic Group,

1999-2006

20%

25%

30%

35%

40%

45%

50%

55%

60%

65%

70%

05-0604-0503-0402-0301-0200-0199-0098-99

Black

White

Hispanic

Asian

8

9

Disparities Exist in HS Graduation, HS Drop Out and College Ready

Source: Manhattan Institute, Public HS Graduation and College-Readiness Rates: 1991-2002, http://Manhattan-institute.org/html/ewp_08.htm;* Condition of Education, 2007 Table 23-2

10

Graduation Rates in 2004 by ethnicity

Published 3/7/2007 Title Awards conferred by Title IV institutions, by race/ethnicity, level of award, and gender: United States, academic year 2004–05 (recalculated to eliminate students who with other or no ethnicity reported). http://nces.ed.gov/ipeds/factsheets/pdf/fct_awards_conferred_03072007_5.pdf; Public HS graduation rates: WICHE 3/2008, http://www.wiche.edu/policy/knocking/1992-2022/index.asp

11

Rationale for looking beyond Grades and Tests

What is college success? Is it more than grades and GPA? (Camara & Kimmel)

Develop measures that predict your goal or desired outcome.

Employers test multiple measures:

openness, conscientiousness, extraversion, agreeableness, neuroticism

Military use today (GED).

Can do does not equal will do.

12

Predictors of College Success

Tests MeasureColleges Collect in some form (applications, transcripts)

Not Collected in Standardized form

College Skills Content Knowledge

Achievement Non-Cognitive

Personal Qualities/ Experiences/

Characteristics

School Performance/

Context

Interests - Vocations

Verbal Reasoning Math Motivation Letters Grades Career Interests

Math Reasoning Language Arts Follow-through Essay GPA Learning -Study

Skills

Writing Science Communication Community Service Weighted GPA Interest in Major

Metacognition Social Studies/

Humanities Conscientiousnes

s Extra-curricular Rank Self Efficacy

Creativity Foreign Language Leadership Work Experience Courses Completed Aspirations/

Practical Knowledge Language Proficiency Other Personality

Literacy in Second Lang Academic Rigor

Realistic Self-concept

Spatial Relations    Health/Lifestyle Teacher Ratings AP/Honors Courses

Intellectual Curiosity   Adaptability Gender School Size

 Technology – Research Skills     Ethnicity School Quality

      Ability to Benefit    

     Family Education/

Income    

      Ability to Pay    

    Residence    

13

• Identify a broader domain of college student performance:• Review university mission statements and

department objectives

• Interview with university staff responsible for student life

• Review of the education literature on student outcomes

• Our systematic search (A JOB ANALYSIS OF UNDERGRADUATE STUDENTS) resulted in 12 dimensions of student performance…

• Validate items with successful juniors – they are the experts.

Research collaboration with Michigan State University

14

12 Dimensions of Student PerformanceBroadening the Performance Domain in the Prediction of

Academic Success (Schmitt, Oswald, & Gillespie, 2004)

1. Knowledge, learning, mastery of general principles

2. Continuous learning, intellectual interest and curiosity

3. Artistic and cultural appreciation

4. Multicultural appreciation

5. Leadership

6. Interpersonal skills

7. Social responsibility, citizenship and involvement

8. Physical and psychological health

9. Career orientation

10.Adaptability and life skills

11.Perseverance

12.Ethics and integrity

15

Two “Noncognitive” Measures

• Situational judgment inventory• A situation is presented along with several

alternative courses of action.

• The respondent is asked to indicate what she/he would be most likely and least likely to do.

• Biodata• Short, multiple choice reports of past

experience/background and interests/preferences.

16

Sample SJI Item for LeadershipYou are assigned to a group to work on a particular project. When you sit down together as a group, no one says anything.

a)-1 Look at them until someone eventually says something

b)Start the conversation yourself by introducing yourself

c)+1 Get to know everyone first and see what they are thinking about the project to make sure the project’s goals are clear to everyone

d)Try to start working on the project by asking everyone’s opinion about the nature of the project

e)You would take the leadership role by assigning people to do things or ask questions to get things rolling

17

Sample Biodata Items for Leadership

1. The number of high school clubs and organized activities (such as band, sports, newspapers, etc.) in which I took a leadership role was:a) 4 or more

b) 3

c) 2

d) 1

e) I did not take a leadership role

2. How often do you talk your friends into doing what you want to do during the evening?a) most of the time

b) sometimes (about half the time)

c) occasionally (about as often as others in my group

d) seldom or infrequently

e) never

18

Study 1: Develop and refine the measures

• 644 MSU freshmen completed one of the two parallel forms of the biodata and SJI instruments at the beginning of the academic year.

• Results indicated significant incremental validity for some of the scales above and beyond the validity of SAT/ACT scores and existing measures of personality in predicting college GPA.

• The biodata and SJI demonstrated the greatest incremental validity when absenteeism, students’ self ratings, and peer-ratings of performance were examined ( .19, .22, and .14, respectively).

19

Study 2 Examine Validity & Subgroup Differences:

10 Participating Institutions & 2,700 Freshmen HBCU N

Winston-Salem (public) 229

Spelman College (private) 254

Big Ten (public) N

University of Iowa 335

Michigan State University 546

Ohio State University 304

University of Michigan 297

Indiana University 170

Other Institutions N

University of Chicago (private) 168

Cal State – Fullerton (public) 223

Virginia Tech (public) 237

20

Predicting FYGPA: Total Sample across 10 Institutions (N = 2443)

Non cognitive measures contribute little beyond tests and grades in predicting academic outcomes

21

Predicting Class Absenteeism: Total Sample across 10 Institutions (N = 899)

However, non cognitive measures will predict non cognitive outcomes – better than tests or grades (graduation, attendance, leadership, engagement)

22

Percent of Students Selected:Two Composites and Three Selection Strategies

Top 85% Top 50% Top 15%

Group AB AB+ AB AB+ AB AB+

Hispanic 4.4 4.6 4.1 4.9 3.9 5.5

(+.2) (+.8) (+1.6)

Asian 7.6 7.7 9.9 9.5 17.5 12.9

(+.1) (-.4) (-4.6)

African-American 17.9 19.8 9.6 13.6 1.3 7.2 (+1.9) (+4.0) (+5.9)

White 70.2 67.9 76.4 71.9 77.2 74.4

(-2.3) (-4.5) (-2.8)

AB = equally weighted composite of HSGPA and SAT/ACT.

AB+ = equally weighted composite of HSGPA, SAT/ACT, Biodata, and SJI.

Less selective Moderately selective Very selective

23

Correlations of Non-cognitive Measures with Cumulative GPA

and GraduationVariable Cumulative GPA Graduation

SAT/ACT scores 0.59 0.24

HSGPA 0.56 0.28

Knowledge 0.24 0.12

Continuous Learning 0.11 0.03

Artistic Appreciation 0.22 0.13

Multicultural Appreciation 0.12 0.11

Leadership 0.10 0.14

Responsibility 0.12 0.15

Health 0.13 0.08

Career Orientation -0.18 0.02

Adaptability 0.01 0.07

Perseverance 0.02 0.10

Ethics 0.15 0.11

SJI 0.20 0.14

Note. Bold values are significant at p< .01. N ranges from 1560 to 1798 across variables. Graduation is dichotomously scored (1, 0).

24

Study 3: Purpose & Research Questions• 15 institutions (n = 4,164 for SJI and 7,645 for biodata)

• Purpose: evaluating the utility of the biodata and situational judgment measures in as close to a real admissions situation as is possible

• Administer new measures to college applicants rather than college freshmen.

• On an annual basis, collect class absenteeism, self rated performance of the noncognitve dimensions, and commitment to the university from enrolled students; institutions will provide course grades and retention.• University of Washington • Meredith College

• Michigan State University • University of Southern California

• Lafayette College • Furman University

• Earlham College • University of North Carolina, Chapel Hill

• Ohio State University • Kenyon College

• Purdue University • Gonzaga University

• Spelman College • University of Puget Sound

• Johnson and Wales University

25

Incremental Validity of Biodata Measures

Outcomes N R2 (HSGPA,SAT) Overall R2 R2

BARS 57 0.023 0.443* 0.420*

OCB 57 0.017 0.392 0.374*

Deviance 57 0.025 0.373 0.348

Turnover Intent 58 0.077 0.248 0.172

Academic Satisfaction 58 0.008 0.353 0.345

Social Satisfaction 58 0.077 0.294 0.218

FYGPA 84 0.201* 0.335* 0.134

Absenteeism 58 0.061 0.234 0.173

• To preserve N in these regressions, the SJI was not included because of a relatively low response rate to this measure.

• It is worth noting that small sample sizes, such as those observed in these analyses, can seriously limit the ability to detect significant relationships due to decreased statistical power.

26

Next Steps• In need of a demonstration project – Implement with Research

across a few colleges!

• Encourage applicants to complete on-line as part of admissions and only use data as a “plus factor.”

• Provide incentives for applicants to complete the new measures and institutions to track student success over time.

• Likely outcomes will be more diversity, broader talent, greater retention, and standardized – defensible measures to evaluate applicants fairly and objectively.

• Increased efficiency and judgmental decisions based on data and comparability.

• For more information, go to http://www/iopsych.msu/cbstudy