Observations on Developing a

Postsecondary Institution

Rating System (PIRS)

Dr. Braden J. Hosch

Asst. Vice President for Institutional Research, Planning & Effectiveness

Stony Brook University

January 22, 2014

2

Current role:

Asst. VP for Institutional Research, Planning &

Effectiveness at Stony Brook University (NY)

IPEDS National Trainer, Advisory Board Member

Former role:

Director of Policy & Research,

Connecticut Board of Regents for Higher Ed

CONTEXT (1)

3See Higher Education Coordinating Council. (2012). Public Policy Framework for Higher

Education. http://www.ct.edu/files/opr/A_CTFramework.pdf

• Connecticut changed its approach to higher education

accountability in 2011

• Broad stakeholder involvement

• 5 goals with 23 indicators, disaggregated

• Used data publicly available through IPEDS

• Institutional performance measured over time

compared to similar institutions

• Accountability more than consumer information

• Work in progress

CONTEXT (2)

4

Comparison Groups

Graduation Rates

Learning Outcomes

Affordability

Employment Outcomes

CHALLENGES

5

• Exclusively mathematical approach to identify institutions

for comparison groups was elusive

• A balance of data, statistics, and judgment was needed

• Data from IPEDS based on about 10 institutional characteristics

generated lists of 20-50 institutions in most cases

• Outliers remained a problem, likely due to data quality / unique

state practices

• Campuses asked to add or remove institutions from the

comparison group based on judgment to arrive at 10-15

comparison institutions

COMPARISON GROUPS

6

• Controls for student inputs are essential to the

interpretation of graduation rates; individual factors

matter much more than institutional factors

• Data from HERI illustrate this relationship, independent of

institutional effects

GRADUATION RATES

7

INSTITUTION-BASED 6-YEAR GRAD RATES BY

SAT COMPOSITE SCORE

Source: IPEDS Data Center Public Institutions Private Not-for-Profit Institutions

8Source: Higher Education Research Institute (UCLA)

INDIVIDUAL-BASED GRADUATION RATES

BY HIGH SCHOOL GPA

6.3 9.815.5

25.2

35.9

47.8

58.3

21.227.7

36.6

48.7

59.8

70.679.3

0

10

20

30

40

50

60

70

80

90

100

C or less C+ B- B B+ A- A+, A

Gra

du

atio

n R

ate

(per

cen

t)

HS GPA as Reported on the CIRP Freshman Survey

4-Years 6-Years

9Source: SBU Office of Institutional Research, Planning & Effectiveness

STONY BROOK UNIV. GRADUATION RATES

BY HIGH SCHOOL GPA

33.940.2

46.150.3

60.658.962.6

67.371.9

79.1

0

10

20

30

40

50

60

70

80

90

100

Below 87 87-89.9 90-92.9 93-95.9 96+

Gra

du

atio

n R

ate

(per

cen

t)

HS GPA

4-Years 6-Years

10Source: SBU Office of Institutional Research, Planning & Effectiveness

STONY BROOK UNIV. GRADUATION RATES

BY HIGH SCHOOL GPA – AFRICAN AMERICAN ONLY

39.7

51.7 52.3 49.156.9

66.3

75.2 74.5 71.980.4

0

10

20

30

40

50

60

70

80

90

100

Below 87 87-89.9 90-92.9 93-95.9 96+

Gra

du

atio

n R

ate

(per

cen

t)

HS GPA

4-Years 6-Years

11

Connecticut adopted a completions per FTE metric

intended to supplant graduation rate

• Statistical adjustments were necessary

– Lag enrollment by 2 years

– Weight certificates by 1/3

– Degree-seeking UG enrollment only

• Data adjustments were necessary

• Results not comparable across sectors

• Cannot be interpreted without reference groups

COMPLETIONS PER 100 FTE ENROLLMENT

12

Deemed very important for quality assurance but

measurement deferred pending further research

Research had demonstrated• Testing is sensitive to recruitment practices and testing conditions

• Student motivation affected institutional results

State pursuing external validation of learning using

“authentic assessment” through a nine state

multi-state collaborative

LEARNING OUTCOMES

13

• Net price involves reasonable components, but

wide variation exists in how institutions determine

costs associated with room & board and other

expenses for students living off-campus.

• Regulatory or legislative action needed to define

calculation method

AFFORDABILITY AND NET PRICE

14Source: IPEDS Data Center, components are for 2011-12

EXAMPLE NET PRICE ANOMALIES

$6,840 $5,823

$13,117

$4,900

$7,410

$2,146

$8,528

$3,450

$0

$5,000

$10,000

$15,000

$20,000

$25,000

Assoc(public under 4-yr)

Assoc(public rural)

Private, for-profitnon-degree granting

Private, not-for-profit4-yr Bacc

Expenses Living Off-Campus, Not with Family:4 Institutions within 1.6 miles

room and board other expenses

0 0.4 1.5 1.6Distance (miles)

15

Available employment data have severe limitations

• Alumni survey response rates for public institutions typically range

20-30%, with likely high levels of non-respondent bias

• State unemployment insurance (UI) data do not cross state lines or

identify full-time/part-time employment or partial/full quarter earnings

• IRS data are problematic for joint returns

• Age and prior employment confound central tendencies – more

research is needed

EMPLOYMENT OUTCOMES

16Source: CT Employment & Training Commission Legislative Report Card (2013)

EXAMPLE WAGE DATA ANOMALIES

$540

$1,076

$473 $507

$254 $270

$451 $472

$0

$200

$400

$600

$800

$1,000

$1,200

CT Community Colleges Charter Oak State College(Public online college)

CT State Universities(Public Regional)

Univ. of Connecticut(Public Flagship)

Avg. Weekly Wages & Change in Weekly Wages for Undergraduate Completers

Weekly Wages Upon Entering Employment Change in Wages from Start of Program

Data gathered from unemployment insurance records through CT Dept. of Labor

17

• Consider institutional input for comparison

groups

• Adopt individualized graduation rate calculator

• Define cost of attendance methodology for off-

campus arrangements

• Defer inclusion of and support research on:

• Learning outcomes

• Employment outcomes

RECOMMENDATIONS

18

Connecticut Dept. of Labor and Connecticut Dept. of Higher Education (2010). Building Connecticut’s

Workforce: Report on 2007-08 Graduates. Retrieved January 13, 2014 from

http://www.ctdhe.org/info/pdfs/2010/HigherEdReport-2008grads.pdf

Connecticut Higher Education Coordinating Council. (2012). Public Policy Framework for Higher

Education. Retrieved January 10, 2014 from http://www.ct.edu/files/opr/A_CTFramework.pdf.

DeAngelo, L., Franke, R., Hurtado, S., Pryor, J. H., & Tran, S. (2011). Completing college: Assessing

graduation rates at four-year institutions. Los Angeles: Higher Education Research Institute, UCLA.

Hosch, B. (2008). Institutional and student characteristics that predict retention and graduation rates.

Annual Conference of the North East Assoc. for Institutional Research, Providence, RI.

Hosch, B. (2012). Time on test, student motivation, and performance on the Collegiate Learning

Assessment: Implications for institutional accountability. Journal for Assessment and Institutional

Effectiveness 2(1): 55-76.

National Governors Association. (2010). Complete to Compete: From Information to Action: Revamping

Higher Education Accountability Systems

REFERENCES