non traditional measures in assessment 090712 (1)

1

BEYOND STANDARDIZED TESTS:

Assessing Creativity, Common Sense, Wisdom, and More in Admissions

2

Contact Information

Robert J. SternbergProvost and Senior Vice President George Kaiser Family Foundation Chair in Ethical

LeadershipRegents Professor of Psychology and EducationOklahoma State University;Honorary Professor, Heidelberg UniversityPresident, Federation of Associations in the Behavioral and

Brain Sciences;[email protected]

mailto:[email protected]

4

Goals of the Presentation

To show that, in admissions, in addition to memory and analytical skills, it is important rigorously to assess creative, practical, and wisdom-based/ethical skills of students

To show how to assess these additional skills in a rigorous way

To show data regarding what happens when an assessment process rigorously assesses these additional skills

5

Organization of Talk

Introduction Five Projects with Non-Traditional Measures

The Rainbow Project The Kaleidoscope Project The Panorama Project Advanced Placement Project The Choate Rosemary Hall Project

Implementing Non-Traditional Measures in Your Institution

Conclusions

6

The Problem

Traditional standardized-test measures in assessment, and even school grades, give us good information about some valued skills of students, but practically no information about other valued skills

7

In Particular…

If we wish to develop students who will be the active citizens and future leaders of tomorrow, we need to measure a broader range of the skills important to future success—not just the memory and analytical skills measured by standardized tests (such as IQ tests, the SSAT, SAT, and the ACT), but also creative, practical, and wisdom-based skills

8

What Skills do Standardized Tests Measure Well?

Memory skills Who? What? Where? When? Why? How?

9

What Else Do Standardized Tests Measure Well?

Analytical skills Analyze Critique Judge Compare and contrast Evaluate

10

What Skills Don’t Standardized Tests Measure Well?

Creative skills Invention Exploration Discovery Imagination Supposition

11


Practical Skills Application Utilization Implementation Putting ideas into practice Persuasion

12


Wisdom-Based Skills Ethical thinking Seeing multiple points of view Understanding how what is true

can change over time Understanding long-term as well as

short-term implications Thinking for the common good

13

Why These Particular Skills?

Motivation for the “Theory of Successful Intelligence” Alice Barbara Celia Diane

Zachary

14

Assessments on which Talk is Based

Rainbow (admissions) National project involving roughly

1000 diverse seniors in high school and college freshmen

Kaleidoscope (admissions) Project at Tufts University involving

over 30,000 applicants to the freshman class over a period of 5 years

15

Assessments on which Talk is Based Panorama (admissions)

New project at Oklahoma State University—no data yet

Advanced Placement (achievement) Project at diverse high schools in

the United States measuring (so far) achievement among several hundred students at high levels in psychology, statistics, and physics

16

Assessments on which Talk is Based Choate Rosemary Hall Assessment

An assessment to help select students who most can profit from, and in turn profit, the environment of the preparatory school

17

Augmented Theory of Successful Intelligence

People are successfully intelligent to the extent that they make the most of their lives

They do so by figuring out their strengths and weaknesses, and (a) capitalizing on the strengths while (b) compensating for or correcting weaknesses

They do so through a mix of analytical, creative, practical, and wisdom-based/ethical skills applied to a knowledge base

18

In particular…

Individuals are successfully intelligent to the extent they display Creative skills to generate novel ideas Analytical skills to ascertain whether the

ideas are good ones Practical skills in order to implement their

ideas and persuade others of their value Wisdom-based/ethical skills in order to

ensure the ideas help to achieve a common good based upon positive ethical principles

19

Source for Details

Sternberg, R. J. (2010). College Admissions for the 21st Century. Cambridge, MA: Harvard University Press.

20

The Rainbow Project

This project was conducted at Yale University in the early 2000s on roughly 1000 students with freshmen from 13 colleges and seniors from 2 high schools. The institutions were geographically very dispersed and were of greatly varying levels of selectivity, from community colleges to highly prestigious ones. There was a high level of ethnic diversity. Assessments were proctored in the students’ own institutions but scored at Yale.

21

Analytical Assessments

Learn meanings of words from context (The blen arose early in the

morning on the horizon.) Complete a number series

(3, 7, 13, 21, ….) Complete a figural matrix

22

Practical Assessment Sample ItemCollege Life Tacit-Knowledge Inventory

You are enrolled in a large introductory lecture course. Requirements consist of three term-time exams and a final. Please indicate how characteristic it is of your behavior to spend time doing the following, if your goal is getting an A in the course.

23

Practical Assessment Sample ItemCollege Life Tacit-Knowledge Inventory

Rate on a 1 (low) to 9 (high) scale:

___Attending class regularly.___Attending optional weekly review

sessions, if there are any, with the T.A.___Reading assigned text chapters

thoroughly.___Taking comprehensive class notes.___Speaking with the Professor after class

and during office hours.…

24

Practical Assessment Sample ItemGeneral Workplace/Common Sense Tacit-Knowledge Inventory

You’ve been assigned to work on a project for a day with a fellow employee whom you really dislike. He is rude, lazy, and rarely does a proper job. What would be the best thing for you to do?

25

Practical Assessment Sample ItemGeneral Workplace/Common Sense Tacit-Knowledge Inventory

Rate on a 1 (low) to 9 (high) scale:

___Tell the worker that you think he is worthless.

___ Warn the worker that, if he is not “on his toes” today, you will complain to the supervisor.

___ Avoid all conversation and eye contact with the other worker.

___ Be polite to the other worker and try to maintain as business-like a manner as possible so that hopefully he will follow your example for the day.

…

26

Practical Everyday Situational Judgment - Movies

Examinees see seven digitized movies depicting various real-life situations that college students confront or may confront: The Party: Entering a party where one does not know anyone

A Fair Portion: Discussing shares of rental payments for a flat

Professor’s Dilemma: Asking for a letter of recommendation from a professor who does not know you very well

27

Creative Written Stories

SHORT STORY TASK: TITLES

“A Fifth Chance”“2983”“Beyond the Edge”“The Octopus’s Sneakers”“It’s Moving Backwards”“Not Enough Time”

28

Creative Oral Stories

SHORT STORY TASK

Students see several pictorial collages. They have to tell a story about one of them.

30

Creative Cartoon Titles

Examinees see five cartoons and need to provide titles for three of the five.

31

________________________________________________________________________

________________________________________________________________________

Exploratory Factor Analysis: Rainbow Tasks

Promax rotation: 62.8% variance explained

Factor 1 Factor 2 Factor 3Oral Stories 0.57 -0.06 -0.06Written Stories 0.79 0.01 -0.02Cartoons 0.20 0.28 -0.08STAT-creative 0.00 0.73 0.09STAT-analytic -0.06 0.80 -0.04STAT-practical 0.03 0.81 -0.02Movies 0.12 0.05 0.52College Life -0.13 0.01 1.00Common Sense 0.12 -0.01 0.92

33

Predicting College GPA: SAT + Analytical

Step 1: SAT-Verbal, SAT-Math

Step 2: Analytic (STAT)

9.8 9.9

0

5

10

15

20

Step 1 Step 2

R s

qu

are

d (

%)

34

Predicting College GPA: SAT +Practical


Step 2: Practical (STAT + performance)

9.810.7

0

5

10

15

20

Step 1 Step 2

R s

qu

are

d (

%)

35

Predicting College GPA: SAT +Creative


Step 2: Creative (STAT + performance) 9.8

12.8

0

5

10

15

20

Step 1 Step 2

R s

qu

are

d (

%)

36

Predicting GPA: SAT + Analytic, Creative, Practical


Step 2: All Rainbow Project Items(STAT Analytic, Practical, Creative, Practical performance,Creative performance)

9.8

19.9

0

5

10

15

20

Step 1 Step 2

R s

qu

are

d (

%)

37

Predicting GPA: All measures (practical before creative)*

Step 1: SAT-M SAT-V HSGPA

Step 2: + Analytic

Step 3: + Practical

Step 4: + Creative

*Controlling for school quality independent variable

15.6 15.2 15.9

24.8

0

5

10

15

20

25

30

Step1

Step2

Step3

Step4

R s

qu

are

d

38

In sum

In the Rainbow sample, Adding Rainbow measures over SAT

roughly doubles prediction of college success

Adding Rainbow measures over SAT + High School GPA increases prediction by roughly half

39

Amount of Each Measure That Is Predicted by Racial / Ethnic Differences (ω²)

0.09

0.04

0.00

0.02

0.01

0.02

0.03

0.01

0.02

0.01

0.03

0.00

0.05

0.10

Ver

bal

Mat

h

ST

AT

ST

AT

Writ

ten

Sto

ries

Ora

lS

torie

s

Car

toon

s

ST

AT

Mov

ies

Col

lege

Life

Com

mon

Sen

se

SAT Analytic Creative Practical

Pro

po

rtio

n e

xp

lain

ed

by

ra

ce

dif

fere

nc

es

40

In Sum

In the Rainbow sample: Rainbow measures reduce ethnic-

group differences relative to the SAT alone

The new measures reduce differences because different ethnic groups show different average patterns

Differences are not eliminated, however

41

The Kaleidoscope Project

Project at Tufts University over a period of five years involving over 30,000 applicants to the freshman class. Additional admissions exercises were optional. Questions were placed as an optional Tufts-specific supplement to the Common Application. The study was done in collaboration with Dean of Admissions Lee Coffin and his staff.

42

Essay Prompts: Year 1

The late scholar James O. Freedman referred to libraries as “essential harbors on the voyage toward understanding ourselves.” What work of fiction or non-fiction would you include in personal library? Why? (Analytical)

An American adage states that “curiosity killed the cat.” If that is correct, why do we celebrate people like Galileo, Lincoln, and Gandhi, individuals who imagined longstanding problems in new ways or who defied conventional thinking to achieve great results? (Analytical)

43


History’s great events often turn on small moments. For example, what if Rosa Parks had given up her seat on that bus? What if Pope John Paul I had not died after a month in office in 1978? What if Gore had beaten Bush in Florida and won the 2000 U.S. Presidential election? Using your knowledge of American or world history, choose a defining moment and imagine an alternate historical scenario if that key event had played out differently. (Creative)

44


Create a short story using one of the following topics: The end of MTV Confessions of a Middle School Bully The Professor Disappeared The Mysterious Lab (Creative)

Using an 8.5x11 inch sheet of paper, illustrate an ad for a movie, design a house, make an object better, or illustrate an ad for an object of your choice. (Creative)

Do a creative YouTube Video and send us the web address (Creative)

45


Describe a moment in which you took a risk and achieved an unexpected goal. How did you persuade others to follow your lead? What lessons do you draw from this experience? You may reflect on examples from your academic, extracurricular or athletic experiences. (Practical)

A high school curriculum does not always afford much intellectual freedom. Describe one of your unsatisfied intellectual passions. How might you apply this interest to serve the common good and make a difference in society? (Wisdom)

Creative Essay: “What if…”

If the Trojans had heeded Laocoon’s advice and thrown Odysseus’ wooden horse into the sea, they would have defeated the Greeks at Troy. Aeneas would then never have had reason to flee the city, and he would never have ventured to Italy to found Rome. Without Rome, neither the Roman Republic nor a Roman Empire would have existed. Concrete, the arch, plumbing, and the sauna might never have been invented. The modern implications of Rome never having existed are indeed drastic. Lacking even concrete floors, people would resort to sleeping in the mud, and, without plumbing or saunas, they would be perpetually filthy and, generally, quite chilly. France could not have built the base of the Eiffel Tower without arches, so tourists would be unable to purchase miniature collectible Towers in Parisian convenience stores.

Good but Uncreative Essay:“What if…”

What if the ratification of the nineteenth amendment did not pass and women were never given the right to vote? What would life for women, like me, be like in the United States? For one thing, I probably would not be writing this essay. If women were not given their right to vote, I probably would stop going to school after this year and it would be unlikely that I would receive a college education. Without suffrage, my career options would be limited, if a career were a possibility at all. My accepted practices would be limited to staying home and taking care of the family. Rather than being equals, women would be subservient to men. I might not drive, I might not dress in the way in which I choose to, and I might not be able to live my life the way that I can in the twenty-first century.

48

Pilot Study Data

Number of applications rose Bottom third of old application pool

greatly diminished; many more top applicants

Average SATs rose slightly African-American applications up 25%,

acceptances up 30% Hispanic-American applications and

acceptances up 15%

49

Pilot Study Data

There were no significant ethnic-group differences on Kaleidoscope

Kaleidoscope correlated moderately with rated leadership/extracurricular activities (.44)

Being rated for Kaleidoscope was associated with higher freshman GPA, holding constant high school GPA and SATs

50

Pilot Study Data

Kaleidoscope predicted extracurricular, leadership, active-citizenship participation

Greater customer satisfaction Message to students, parents,

teachers, and counselors that Tufts is looking for more than just the high-SAT, high-GPA student

51

The Panorama Project

This is a new project at Oklahoma State University. All applicants are being given the opportunity optionally to answer questions that assess creative, analytical, practical, and wisdom-based skills. Each applicant who participates answers three questions, similar to those in Kaleidoscope. Scoring will be similar to that for Kaleidoscope. Ratings will be used for those evaluated by holistic admissions and for scholarship consideration.

52

Panorama Sample Analytical Question

An army colonel once stated, "Leadership is about comforting the disturbed and disturbing the comfortable." What did he mean? What do you think true leadership is? When have you been a leader, and how did you exercise leadership?

53

Panorama Sample Practical Question

If you were able to open a local charity or business of your choice, what type of organization would it be and whom would it benefit? Describe your start-up process.

54

Panorama Sample Creative Question

Write a story or poem that includes one of the following sets of words:

Purple, panic, panda, petunia, and popcorn

A stick, a light bulb, the Great Wall of China, and water

A bicycle, a clock, the Wild West, and duct tape

55

Panorama Sample Wisdom Question

After submitting a class project, you realize one of your partners committed plagiarism. Your teacher previously announced that if he or she learned that cheating had occurred, all members of the work group would receive an F grade. How would you handle the situation and what would be your ideal outcome?

56

The Advanced Placement Project This is a project to infuse measurement

of creative, analytical, and practical skills into tests of achievement given in high schools. Study has been run so far with Psychology, Statistics, Physics.

57

An Example from Psychology: Creative

Imagine that you had to produce a TV sitcom to illustrate Freud’s personality theory.

Which of the following characters would best represent the superego? (a) A Fire Fighter (b) An action-movie hero (c) A nurse (d) An artist *(e) A Supreme Court judge

58

An Example from Statistics: Practical

Mr. Smith, a politician, argues that the average family income in his state is $4,500 a month and that therefore complaints about massive poverty are ill-founded. What is wrong with Mr. Smiths’ claim?

(a) $4,500 a month is not very high. (b) Mr. Smith’s statement is obviously not true. *(c) Mr. Smith does not take into account the

standard deviation of incomes. (d) Mr. Smith is not an expert on poverty and

hence has no credentials to make any claim about it.

59

Choate Rosemary Hall Project

This battery included a variety of measures to enhance prediction of academic success in the environment of Choate Rosemary Hall

The samples that follow are from the School Life Questionnaire

School-Life Questionnaire1 2 3 4 5 6 7

Not a very good Average A very good

choice choice

In day schools you rarely see your teachers outside of class. Some of them might be engaged in sports or other extra-curricular activities, but mostly you only see them in school-related circumstances. At boarding school the situation is quite different, because many teachers live on campus and you get to see them outside the classroom a lot.



choice choice

Given this situation, rate the quality of the following behavior choices:

____ (a) Always greet teachers and smile, but avoid seeing them outside of class.

____ (b) Take advantage of this situation to talk to teachers about your school-related problems.

____ (c) Wait and see if teachers approach you, and if so, what kinds of things they talk to you about.



choice choice

____ (d) Talk to your teachers but avoid discussing your problems as this might give them a negative impression of you.

____ (e) Try to be sensitive and make a distinction between situations when teachers are available and unavailable to you.

____ (f) Always try to be noticed—the more teachers talk to you, the better your grades will be.

____ (g) Always ask whether it is a good time or not to discuss your problems with teachers.

1 2 3 4 5 6 7


choice choice

You are taking a math class that gives you a lot of trouble. On the first two tests you did poorly, and for tomorrow you have a homework problem that you are not quite sure how to solve. .

1 2 3 4 5 6 7


choice choice

Given this situation, rate the quality of the following behavior choices:

____ (a) Try to find a solution, or at least some explanation of how to arrive at a solution, in the book .

1 2 3 4 5 6 7


choice choice

____ (b) Give it a try; if you can’t solve the problem you’ll just have to tell the teacher you didn’t understand it.

____ (c) Ask some of your classmates if they’ve found the solution, telling them you want to compare it with yours’ (although you don’t actually have one).

1 2 3 4 5 6 7


choice choice

____ (d) Try hard, and if it doesn’t work, give up on it; you can always pretend you had forgotten you had homework.

____ (e) Suggest to some of your classmates that you study together.

1 2 3 4 5 6 7


choice choice

____ (f) Go to see the one student in the class who you know is really smart and ask him for help.

____ (g) Try hard, take a lot of notes, then come to the teacher before class and tell her/him that you tried and failed.

Predicting Choate GPA: Hierarchical Regression

31.3

50.9

0

10

20

30

40

50

60

Choate GPA 12/ 99 (adj Rsq)

Pre-Choate GPA, SSAT (all) Yale indicators (all)

Predicting Choate GPA: Hierarchical/Stepwise Regression

24.730.9

52.7

0

10

20

30

40

50

60


Pre-Choate GPA SSAT(Q) Yale indicators (3)

Predicting Choate GPA: Hierarchical Regression

42.5

52.858

0

10

20

30

40

50

60


Locus of Control Choate Tacit Knowledge Self-confidence

71

Main Results

Adding analytical, creative, and practical questions: Increases content validity Increases face validity Decreases ethnic-group differences

72

Putting Theory into Practice: Creating Rubrics

Test users are busy people. So it is preferable to use “holistic” ratings rather than trying to be excessively micro-analytic.

You need to decide what you value in responses

Some rubrics we have used are the following:

73

Analytical Rating

The extent to which the response is Analytical Organized Logical Balanced

74

Creative Rating

The extent to which the response is Novel Compelling Task-appropriate (either yes or no)

75

Practical Rating

The extent to which the response is compelling

The extent to which the response is practical with respect to Human resources Material resources Time and place

76

Wisdom Rating

The extent to which the response reflects A common good A balance of one’s own, others’,

and larger interests Thinking over the long-term as well

as the short-term Positive ethical values

77

A Common Error to Avoid

In scoring, you do not want merely to measure general academic skills; therefore, do not take into account grammar, punctuation, capitalization, and spelling except for the analytical rating. Otherwise, you create a general factor (halo effect) that has little to do with what you want to rate and that will increase correlation of the non-traditional assessments with conventional standardized tests

78

Conclusion

Creative, practical, and wisdom-based skills, like memory and analytical skills, can be rigorously assessed

Measurements such as those described here provide a means for rigorous assessment

Assessing such additional skills provides incremental prediction of academic and personal success and reduces ethnic-group differences

79

Conclusion

In order to develop the active citizens and leaders of tomorrow, who will make a positive, meaningful, and enduring difference to the world, one reliably and validly can assess in school creative, analytical, practical, and wisdom-based skills.