factors influencing item nonresponse in the nsse survey:  why do students skip so many questions?

Factors Influencing Item Nonresponse in the NSSE Survey:  Why do Students Skip so many Questions?

Presented by Chris Maxwell

Purdue University

AIR 2009

Introduction

Purdue NSSE 2004: Max Item Nonresponse: 10%

Purdue NSSE 2007: Max Item Nonresponse: 18%

Purdue NSSE 2007:• Web Only• Targeted Oversample: Underrepresented• 24% response rate• 1,388 total respondents (703 FY and 685 SN)

All Students (FY and SN)

70%

75%

80%

85%

90%

95%

100%

1 11 21 31 41 51 61 71 81

NSSE 2007 Question Order

Pe

rce

nt

Re

sp

on

din

g

N=1,388

Investigation

Question order is an apparent factor, but is anything else at play?

Model the response probability with logistic regression:

• Separate models for students grouped by class, ethnicity, gender, and STEM status (18 total).

• Each model includes the explanatory variables question order, type, length, and first page status.

Model Details

Ln(P/(1-P)) =A+B1(order)+B2(page1)+B3(likert4_scale)

+B4(numerical_scale)+B5(length)

Use SAS proc Logistic (binary modeling, stepwise selection)

Input data set

Model Results

Stepwise selection rejected question types and length as useful predictors in every case

15 models use question order and page one status, with 3 models only using question order

Ln(P/(1-P)) = A + B1(order) + B2(page1) -or- P = exp(A + B1(order) + B2(page1)) 1 + exp(A + B1(order) + B2(page1))

70%

75%

80%

85%

90%

95%

100%

1 11 21 31 41 51 61 71 81

NSSE 2007 Question Order

Per

cen

t R

esp

on

din

g

SN

FY

FY and SN Results

N=685

N=703

FY and SN Results

70%

75%

80%

85%

90%

95%

100%

1 11 21 31 41 51 61 71 81

NSSE 2007 Question Order

Per

cen

t Res

po

nd

ing

SN

FYN=685

N=703

Model Parameters

FY SN

A 2.39 2.70

B1 -0.0141 -0.0130

B2 4.69 4.58

Percent Concordant

62.7 61.6

Thus, for FY students the model predicts a 9% decline in response after the first page, and in the middle of the survey a 1% drop for every 6 questions.

FY: STEM and Non STEM Majors

70%

75%

80%

85%

90%

95%

100%

1 11 21 31 41 51 61 71 81

NSSE 2007 Question Order

Pe

rce

nt

Re

sp

on

din

g

STEM

Non STEMN=301

N=399

70%

75%

80%

85%

90%

95%

100%

1 11 21 31 41 51 61 71 81

NSSE 2007 Question Order

Pe

rce

nt

Re

sp

on

din

g

STEM

Non STEM

SN: STEM and Non STEM Majors

N=459

N=219

FY: Male and Female

70%

75%

80%

85%

90%

95%

100%

1 11 21 31 41 51 61 71 81

NSSE 2007 Question Order

Pe

rce

nt

Re

sp

on

din

g

Female

Male

N=347

N=356

SN: Male and Female

70%

75%

80%

85%

90%

95%

100%

1 11 21 31 41 51 61 71 81

NSSE 2007 Question Order

Pe

rce

nt

Re

sp

on

din

g

Female

MaleN=339

N=346

FY: All Underrepresented

70%

75%

80%

85%

90%

95%

100%

1 11 21 31 41 51 61 71 81

NSSE 2007 Question Order

Pe

rce

nt

Re

sp

on

din

g

White/Other/Asian

Underrepresented

N=99

N=564

70%

75%

80%

85%

90%

95%

100%

1 11 21 31 41 51 61 71 81

NSSE 2007 Question Order

Pe

rce

nt

Re

sp

on

din

g

White/Other/Asian

African American

FY: African American

N=564

N=45

70%

75%

80%

85%

90%

95%

100%

1 11 21 31 41 51 61 71 81

NSSE 2007 Question Order

Pe

rce

nt

Re

sp

on

din

g

White/Other/Asian

Hispanic

FY: Hispanic

N=564

N=44

SN: All Underrepresented

70%

75%

80%

85%

90%

95%

100%

1 11 21 31 41 51 61 71 81

NSSE 2007 Question Order

Pe

rce

nt

Re

sp

on

din

g

White/Other/Asian

UnderrepresentedN=592

N=76

70%

75%

80%

85%

90%

95%

100%

1 11 21 31 41 51 61 71 81

NSSE 2007 Question Order

Pe

rce

nt

Re

sp

on

din

g

White/Other/Asian

African American

SN: African American

N=592

N=36

70%

75%

80%

85%

90%

95%

100%

1 11 21 31 41 51 61 71 81

NSSE 2007 Question Order

Pe

rce

nt

Re

sp

on

din

g

White/Other/Asian

Hispanic

SN: Hispanic

N=592

N=36

All

Mo

del

Par

amet

ers

Model A (Intercept) B1 (order) B2 (page1)

sn_stem 3.36 -0.0197  

sn_female 3.02 -0.0137 4.26

sn_white 2.74 -0.0131 4.68

sn_all 2.70 -0.0130 4.58

fy_female 2.67 -0.0153 5.35

sn_hispanic 2.65 -0.0123 3.07

sn_not_stem 2.58 -0.0127 4.30

fy_stem 2.50 -0.0126 4.25

fy_white 2.46 -0.0141 4.62

sn_male 2.46 -0.0128 4.83

fy_hispanic 2.42 -0.0184  

fy_all 2.39 -0.0141 4.69

fy_not_stem 2.31 -0.0152 5.13

sn_black 2.28 -0.0183  

sn_underrep 2.27 -0.0118 4.19

fy_male 2.16 -0.0134 4.44

fy_black 1.98 -0.0158 3.98

fy_underrep 1.92 -0.0133 4.81

Summary

Study Limitations

Conclusions

Discussion

Available at: www.purdue.edu/OIR/irvba/AIR2009/nsse.ppt