1

“How Optional Testing Works: Defining Promise in American Admissions”

William C. Hiss, Principal InvestigatorValerie Franks, Co-Author and Lead Researcher

Annual MeetingAmerican Association of Colleges and Universities

Washington, DCJanuary 23, 2015

1. Title page.

Thank you for coming to hear about “Defining Promise“ a study on optional standardized testing, which NACAC published electronically last February.

A 70-page scholarly study swimming with statistical charts might be expected to fall still-born from the electronic press—with a glass of sherry, the solution to your insomnia problems. But while about testing, this study

2

addresses the social ethics of who gets to go to college, and the same day that NACAC published this report, NPR, CBS Evening News and PBS NewsHour all covered the study.

A few weeks later, the College Board held their news conference to announce proposed changes to the SATs. The two, coming back to back, created something of a media typhoon, not yet fully died down. It has been a rare week since February without a trip like this one to present the study, a media contact, or more institutions going test-optional, five in January.

Lumina Foundation has calculated the immense pool of low-income students who need to get through higher education to have a chance. This study provides the research support for optional testing as at least one route by which that can happen.

Some of our most thoughtful educational researchers are coming to similar conclusions about the predictive value of testing. In their 2009 book, Crossing the Finish Line: Completing College at America’s Public Universities, Presidents Bowen and McPherson, with their colleague Matthew Chingos, examined what predicted graduation at a large set of public universities. I quote: “In all but one of these 50 public universities, high school GPA remains a highly significant predictor of six-year graduation rates after taking account of the effects of test scores… Test scores, on the other hand, routinely fail to pass standard tests of statistical significance when included with high school GPA in regressions predicting graduation rates—especially when we leave the realm of the most highly selective universities…. …the remaining predictive power of the SAT/ACT scores disappears entirely when we add controls for the high school attended, whereas the predictive power of the high school GPA increases.” (close quote, pp. 115-6)

We agree with another finding in Crossing the Finish Line: the controversy of “overmatching” –where students allegedly attend a college or university for which they are not qualified, versus “undermatching” –where students attend an institution that does not provide enough challenge or opportunity. Bowen and his colleagues wrote, “The findings reported in this study fail to

3

provide any evidence of overmatching, but demonstrate that undermatching is a massive problem.”

In a book just out, The Tyranny of the Meritocracy: Democratizing Higher Education in America, Professor Lani Guinier of Harvard Law School, citing our study, argues that testing principally constrains rather than opens paths to opportunity, creating what she calls a “testocracy.”

2. “Take Back the Conversation.” Ethical and Practical Issues.

A few years ago, the NACAC Commission on Standardized Testing chaired by Bill Fitzsimmons at Harvard made a recommendation, much cited since, that counselors and admissions officers “take back the conversation” about testing from organizations for whom it was either a highly profitable business or a cause. This study contributes to that conversation; it is the first national, peer-reviewed study to examine optional testing across institutional types. We make no claims to have gotten everything right; no study of this magnitude and scope should pretend to have slam-dunk results on every issue. We hope our study will serve as encouragement for other research, and will suggest a few studies in the last slide as “a legacy lap.”

4

Partly trained as an ethicist, I have worked on optional testing for 35 years. The faculty at Bates, where I led the Admissions office from 1978 to 2000, voted for an optional testing policy in 1984, and we have published careful statistical reports at 5-year intervals since. This current national research follows up on a 25-year look-back study of optional testing at Bates.

This study poses some broad ethical questions:

• Can optional testing lower access barriers and expand college populations? Fundamentally, does testing help find promising students, or does it artificially truncate the pools of students who would succeed if they could be convinced to apply and enroll? As you will see, at least based on this study, it is far more the latter.

• Who is exploring the breadth of human intellect and promise in imaginative ways? Institutions with optional testing policies are not all doing it the same way. Public universities, with optional testing policies quite different than at privates, are seeing success.

• Who is doing the heavy lifting, serving broad constituencies? In our study are two categories of institutions not getting much attention in the various top-25 lists, minority-serving, and arts institutions. Also included are a public university with 50,000 students and a long history of strength in science and technology, and two small colleges with traditions of creative curricular thinking, the College of the Atlantic in Maine and Pitzer in California. The study also includes a Native American college, and a minority-serving public flagship in a US Territory. So we tried to look across a broad landscape.

There are also practical questions: Very simply, how well have these policies worked? Others may raise the complex issues of test bias, but we ask a much simpler question: how well do non-submitters succeed? What are the four year cumulative college GPAs and graduation rates of submitters and non-submitters, as segmented by institutional types?

5

What are the practical enrollment outcomes for applicant pools, geographic reach, diversity, intellectual achievement, and accurate financial modeling? Unexamined assumptions about testing seem to be costing individual institutions millions of dollars. Optional testing mostly serves students, but it also has to work for institutions.

3. Research Sample Overview.

We focused on four-year institutions that are non-profit, report data to IPEDS, and have national visibility. We considered 400 institutions, and contacted 130 to choose the 33 in our study.

Here are the institutions: 20 private, 6 public, 5 minority-serving, and 2 arts institutions, with the exact numbers of students in each category listed. We requested four cohort years of data from each institution, two graduated, and two currently enrolled, to get a spread of data over time.

4. Publically Announced Institutions in the Study

6

After seeing the published study last March, 13 of the 33 institutions announced that they were included in our study, and here they are. All 33 institutions shared enormous amounts of data, 40 variables on each of the 123,000 student records, 5 million data items. We guaranteed everyone confidentiality—after all, none of us knew in 2010 how this would turn out. Each institution, as a gesture of thanks, received a proprietary institutional analysis, including a blind comparison study of the other institutions in their category, over 2000 pages of individual reports.

5. Summary of Institution Type

7

We focused tightly on the years of college enrollment: we did no analysis on the Admissions funnel before enrollment, nor on career or graduate school outcomes. But there are additional studies crying to be done, and here is one. From 2003 to 2010, the percentage of non-submitters enrolled at the 20 private institutions rose from 26% to 35%. But why? More non-submitters applying? More confidence among admissions deans that these students are doing fine, and therefore can be admitted at higher rates? Higher yield on non-submitters in the April wars? Higher rates of ED apps from non-submitters? Better retention of non-submitters, so that they show up in these enrolled student counts? We interviewed each dean of admissions in the study, and from their anecdotal information, the answer to all five of these questions may be “yes.” But we don’t know, and a national study of optional testing as it has shaped the admissions funnel lies awaiting.

A second issue is important for independent colleges and universities, competing with public institutions. Most people, if they know of optional testing at all, think it is used by private liberal arts institutions. But increasingly, large public institutions have test-optional admissions,

8

admitting students on the basis of HS rank or GPA, no matter what their testing. The obvious example is the “top 10%” policy in Texas, but also the entire CSU system in CA, and many state flagships such as Mississippi and Arizona. Some universities told us in early conversations, “We are not test optional, we collect tests from everyone.” Our answer was, “Yes, you collect tests, but if you are admitting students solely on rank or GPA, you are not using testing to make admissions decisions. Those are the students we want to study as ‘non-submitters,’ to use parallel terminology with the private institutions.”

Of course many of these public university students had the required rank or GPA for guaranteed admission, but also had very solid testing, above the average of the university where they enrolled. So while they are technically non-submitters, common sense tells you that they likely would have been admitted anyway, with or without the automatic admissions policy. So this study focuses on students admitted who met the HS rank or GPA criteria, but with test scores below the averages of their public university. They are the beneficiaries of their state’s assured or guaranteed admissions policy. You will see the phrase “without above-average testing students”, as we focus on students admitted with below-average testing. As you will see, the findings are dramatic; these public universities seem largely to be succeeding with their versions of optional testing.

Also, a comment also on the findings from the minority-serving and arts institutions. Their data largely lines up with the overall findings, and their students are included in all of the data dealing with the entire study. But in these institutions, we often found problems of missing data or small head counts. The minority-serving institutions do some of the heaviest of the heavy lifting, and the arts institutions meet a wide array of student expression. But we would quickly say that at these two types of institutions, we do not have slam-dunk results to present, based on much thinner data.

6. Principal Findings

9

What are the principal findings? There are no significant differences in either Cumulative GPA or graduation rates between submitters and non-submitters. Across the study, non-submitters (not including the public university non-submitters with above-average testing,) earned Cumulative GPAs that were only .05 lower than submitters, 2.83 versus 2.88. The difference in their graduation rates was .6 % . By any standard, these are trivial differences. On this we hang the national sluicegates about who can go to college and where they go? This does not make any sense. Valerie and I in the research often felt like the kid by the side of the road, taking a deep breath and saying, “The emperor doesn’t have any clothes on.”

College and university Cumulative GPAs closely track high school GPAs, despite wide variations in testing. Students with strong HSGPAs generally perform well in college, despite modest or low testing. In contrast, students with weak HSGPAs earn lower college Cum GPAs and graduate at lower rates, even with markedly stronger testing. A clear message: hard work and good grades in high school matter, and they matter a lot.

10

When compared to the submitter population, Non-submitters are more likely to be first-generation-to-college enrollees, all categories of minority students, Pell Grant recipients, and women. But across institutional types, white students also use optional testing policies at rates within low single digits of the averages, so the policies have broad appeal across ethnic groups. The breadth of students being served by optional testing is positive and promising.

College admissions decisions made without testing are apparently just as reliable as those made with testing. Testing may serve to artificially truncate the applicant pools of students who would succeed if they could be convinced to apply. The numbers of students in these categories are not small. In our study, about 30%, or roughly 37,000, of the students are non-submitters, not counting the above-average testers. Please remember that 30% figure, as it raises a key question at the end.

7. Additional Findings

11

In a surprise finding, non-submitters display a distinct two-tail or bimodal curve of family financial capacity. First-generation, minority and Pell-recipient students will often need financial aid support, but large pools of students not qualifying for or not requesting financial aid help balance budgets. Across institutional types, both low income and high income students will use an optional testing policy if offered.

LD students, from a modest sample of 1050 students at 8 institutions, are much more likely to apply as non-submitters, apply ED and perform at levels close to their classmates. The evidence from a long-term study at Bates found that given the modest accommodations to which these students are legally entitled, their GPAs and graduation rates come up to class averages, helping to increase the institution’s overall graduation rates. Another topic for future research.

Non-submitters are commonly missed in consideration for no-need merit awards, despite slightly better Cum GPAs and markedly higher graduation rates than the submitters who receive merit awards. Institutions should examine criteria for merit awards, especially standardized testing to qualify students for no-need merit funding. A chart in a moment makes this point painfully clear.

Based on our interviews with deans, non-submitters often expand applicant pools, apply Early Decision at higher rates, increase minority enrollments, expand geographic appeal, and allow for success by LD students.

8. College Cumulative GPA versus High School GPA.

12

Now the moment you’ve been waiting for: statistical charts.

This is the first: a scatterplot which simply graphs four-year high school GPA against four year college GPA. You see the pattern: With nothing else but HS GPA and College GPA being looked at, this is an extremely tight and evenly sloped set of data.

Most deans who have seen this chart have said, “Well, two four-year long-distance measures of curiosity, ambition, self-discipline, ability to be part of a team and listen to teachers—maybe not a surprise that these two criteria have such a tight overlap pattern.”

9. College Cumulative GPA Versus SAT

13

In contrast, in this chart college GPA is graphed against SATs, including translated ACT scores. The shape of this graph is more diffuse and scattered. It still shows some correlation, but clearly not a tight relationship.

Note also that we are using college cumulative GPA as our academic marker, rather than the more volatile first year GPA as the College Board does in its reports. It is a far more reliable marker of academic performance.

10. Summary of Key Statistics. All 33 Institutions.

14

Here is the other chart of summary information, and you will see four of these charts for different groups of students, so a moment to explain the layout. There are two parallel columns of findings, with non-submitters on the left and submitters on the right, with the N’s at the top. Each horizontal row contains data on a particular issue, such as HSGPA, SATs or graduation rates. The color coding reflects the level of statistical significance, with the coding explained in the color keys. Grey colors reflect trivial, small or insignificant differences, while the green colors reflect moderate, large or significant differences. As with the scatter charts, don’t try to follow each data point, but watch the patterns in various groups of students.

In this chart of all 33 institutions, the only difference is that the non-submitters come with moderately better HSGPAs, and then perform better at small levels in college. But now look at the second chart. Here, the above-average-testing students at the public universities have been removed. Now the statistically large difference is in SAT scores, with the non-submitters averaging 113 points below the submitters. But the core finding: their cumulative GPAs are only .05 below the submitters, 2.83 versus 2.88, and their graduation rates are .6% below the submitters.

15

11. Summary of Key Statistics, Private Institutions

These data are a summary snapshot for the 20 private colleges and universities, with about 37,000 aggregate records. Submitters and non-submitters enter college with very closely parallel GPA’s but large differences in SAT scores of 149 points. In college, non-submitters start with first-year GPA differences of .15 with submitters, but gradually close the GPA gap to .1 by graduation. Their graduation rates are different by rates of 1%-2% either in favor of submitters or non-submitters, depending on how they are calculated. Non-submitters at privates are somewhat less likely to be STEM majors--given the importance of STEM fields to our economy, a finding worth more exploration.

This pattern runs through much of the study. The only large difference in either entering credentials or in college performance is SAT scores.

12. Summary of Key Statistics, Public Institutions

16

Here is the same data for the public university students, with the above-average-testing students removed.

These lower-testing students enroll with HSGPAs that are much stronger, almost three-tenths of a GPA point, and average SATs that are 93 points below the submitters. They earn equal to stronger GPA’s, and significantly better graduation rates than the submitters.

Again we see the pattern: non-submitters are more likely to be minorities, first-gen, women and Pell recipients, at rather dramatic levels of 9 to 15 percentage points above the submitters.

At the publics, a small difference in STEM majors, only by 2%.

13. Application Comparisons, Merit Award Recipients

17

I promised an example of how policy on optional testing might allow millions of dollars to be more wisely spent, and here is that example. We collected data on no-need merit awards, and found that a bit over 27,000, or 22%, received a merit award.

When all students are included, as in the left hand graph, about equal numbers of submitters and non-submitters received a merit award, but the non-submitters have much higher college GPA’s and graduation rates, and are more likely to be STEM majors. In the right hand graph, we removed the above-average-testing non-submitters, and the graph reflects major changes. First, the number of non-submitters with a merit award drops dramatically, from 13,700 to a little over 5000. These below-average-testing non-submitters still have better HSGPAs than the submitters, but SATs that are 143 points below the submitters. Yet these below-average testers have college GPAs that are slightly better than the submitters, and graduation rates that are much better, by 6%.

Who are these lower-testing non-submitters? Once again—I hope by now you can chant these categories in unison--minorities, first-gen, women and

18

Pell recipients, and by quite significant differences. These lower-testing non-submitters are also more likely to be STEM majors, by 5%.

Acknowledging that the lion’s share of merit money is institutionally funded, let me ask a blunt question. Why are we funding 2.7 times as many submitters with lower HSGPAs and better testing, when those submitters graduate with slightly worse cum GPAs and much worse graduation rates? The answer, in all likelihood, is that using SAT and ACT scores to determine merit awards is an unexamined assumption. At many institutions, nobody has looked at the non-submitters versus submitters to see how they perform.

Perhaps these institutions believe that the higher average SAT scores will earn them a better ranking in USNews or elsewhere. If that is the strategy, a word of advice from a retired dean who founded and chaired for a decade the advisory committee of deans at USNews; I helped design the formulas. Currently, SAT and ACT scores account for 8% of the ranking formula, while graduation rates, when added up from three different parts of the formula, count for 30%. So if you think that the fancy SAT scores of kids with less strong HS records will help you, you might have your IR office calculate for your institution the almost 4-1 heavier weight in the USNews formula for graduation rate over SATs. In all likelihood, that 6% lower graduation rate for submitters is coming around to bite you in the backside. Unexamined assumptions about testing to determine who gets merit awards are in all likelihood the tip of a large fiscal iceberg. Perhaps another place to “take back the conversation.”

14. Not a Victory Lap but a Legacy Lap… The Next Steps?

19

The sponsoring foundation for our study gave us a second grant to cover travel expenses to present the study at conferences and on campuses. The foundation said that they hoped it would be “not a victory lap but a legacy lap—what are the next steps?”

Each of you might have suggestions for a legacy lap. But here are a few:

--A research team should examine the twin issues of “false negatives” of low testing with potential “false positives” of high testing created by coaching. This study provides evidence of the “false negatives” of students with modest testing and strong results on everything else in high school and college. We also know that many upper-income students are being coached to death for the tests, and there is not much argument any more that sustained coaching can significantly increase test scores. But we don’t know, at least on a national basis, whether the coaching that inflates test scores also improves college GPAs. My personal guess is that it does not, and we are perhaps seeing some evidence of that in the submitters in this study, with their less strong HSGPAs, stronger testing, and lower college

20

performance and graduation rates. If that is true, we may be seeing diminished predictive value of the tests on both ends, high and low.

--An ethicist’s question. If we had a medical test for a serious condition, such as liver or heart failure, that had a 30% rate of false negatives--that is, the test said you didn’t have the condition when in fact you did--would that be OK? Of course it wouldn’t. The firm supplying that test would be sued out of business or closed by the FDA. You will recall that roughly 30% of the students in our study are non-submitters. We offer a hypothesis, that the false negative results in testing are in fact quite close to a false negative in a medical test, given the importance of access to the strongest level of higher education for which a student is qualified.

--Further, if 30% is the non-submitter share of enrolling students, what is the true share of false negatives created by testing, including those who are refused, attending community colleges and for-profit colleges, or not attending at all? By definition, the 30% in our study are those who made it through, who by good guidance counseling or grit or family encouragement got into these institutions. So what is the real percentage, including those who do not make it to where they should be? Is it 35% or 40% or 50%? America cannot afford these levels of lost talent.

--Examine college success using 4-year Cum GPAs and graduation rates rather than first-year GPAs as the principal yardstick. For too long, we have held on to 1st year GPA to measure success, in part because of a din of information from the testing agencies that their tests could only predict 1st year performance, and nothing predicted after that, with all the changes in college. This study and others such as Crossing the Finish Line are providing the evidence that 4-year college cum GPAs and graduation rates ought to be our markers. We should also add alumni and grad school outcomes in future studies.

--Evaluate a broader band of research tools. Add Cohen’s d, Chi Square, bar charts and scatterplots to regression (R-square) analysis. There is a much richer array of tools than has commonly been used.

21

--Share published research on optional testing. Models are available from NACAC, Bates and Ithaca, and in Joseph Soares, ed: SAT Wars: The Case for Test-Optional Admissions, as well as Crossing the Finish Line. (I would add a profound bow of thanks to the Bates Presidents since 1980 who have each allowed us to get this research done: Clayton Spencer, Nancy Cable, Elaine Hansen, Don Harward and Hedley Reynolds.)

--NACAC will be hosting a pre-conference workshop on optional testing and research on testing next fall, to explore policy, non-cognitive skills, implementing an optional testing policy, and so on.

15. Institutions Recently Adopting Test-Optional Policies.

We mentioned that scarcely a few weeks went by without another college or university deciding to become test-optional. Here is a list. This seems an encouraging, interesting group, questioning assumptions that optional testing is for small privates. Wesleyan and Bryn Mawr are very high-end small privates. But look at the others. Hostra, a New York private, has over 12,000 students and law and medical graduate programs. And 6 of these 11

22

are public: with widely differing missions—Old Dominion, a public with 25,000 students, is in Norfolk, so with a military flavor to its students and culture. Just within the last week or so, VCU has become test optional: 32,000 students, 2200 faculty, and major medical and science graduate programs. Temple, with traditions of access for urban students, has an enrollment of 38,000, Montclair State has 18,000 and Plymouth State has 7300, with many rural NH students. Recently I said to an audience of high school students in Houston, it will not be long before you will choose to apply to 8-12 very strong colleges and universities, all of which are test optional. In recent months people have asked whether the optional testing movement is reaching a “tipping point.” Bob Schaeffer at Fairtest wisely said that we probably won’t know the tipping point until some time after we pass it. But if these institutions and the 33 in our study are any sign, we can feel the ground moving.

16. Video Clip of Megan Guynes and Cristal Martin.

In recent months Valerie and I have been working with a film company in San Francisco on a documentary about testing. Here is a very short clip from

23

their work to date. It is worth noting that these two recent Bates alumnae come from small towns a long way from Bates: Megan from Rancho Cucamonga, CA, 40 miles east of LA in the San Bernardino Valley, and Cristal from Wichita Falls, TX, 15 miles from the OK border where Texas turns to go up into the Panhandle. Are any of us doing Admissions recruiting travel to those areas? Of course not, but one of the persistent findings of the study is that optional testing helps institutions spread out geographically. Students who will be helped by an optional testing policy, if they learn about it, will apply from territories that are far beyond your normal geographic drawing zones.

Thank you for coming, and glad to take questions and comments.

17. Q&A