increasing survey cooperation: motivating chronic late responders to an annual survey national...

Increasing Survey Cooperation: Motivating Chronic Late Responders to

an Annual Survey

National Science Foundation

Division of Science Resources Statistics

Ronda Britt and Fran Featherston

ICES IIIJune 19, 2007

National Science FoundationDivision of Science Resources Statistics

www.nsf.gov/statistics

2

NSF Academic R&D Survey


• Voluntary survey of 700 research & development - performing universities and colleges

• Conducted annually since 1972

• Web only since 2001

• Requests expenditures for R&D performed during previous year

• Survey data based on respondent’s fiscal year

3

Historical Response Pattern


•Early November email launches survey & sets January 31 deadline

•50-60% respond by deadline, despite multiple reminders

•Several months after deadline to reach response rate of ~95%

•Respondents were allowed to extend deadline, creating cycle of late response

4

Goals of Experiments FY 2004 - FY 2005


•Increase response rate at survey deadline

-Speed up first interaction with web survey

-Increase number of reminders

-Vary mode of reminders

•Decrease number of weeks to reach 95% response rate

5

Experiment 1- FY 2004Increase Response Rate at Survey Deadline, Part 1


• Hypothesis: Respondents who visit website soon after launch (acknowledge survey) are more likely to respond by the survey deadline

•3 study groups received varying modes of reminders:

Group 1 received one e-mail reminder in December (control group)

Group 2 received two e-mail reminders in December

Group 3 received 2 phone calls in December

•All groups received final e-mail in January requesting acknowledgement

6

Experiment 1 Design


Key Dates

Study Groups

Group 1(control)

Group 2 Group 3

November 23 Survey Launch

December 14Acknowledgement

e-mailAcknowledgement

phone calls

December 30Acknowledgement



phone calls

January 11 Acknowledgement e-mail

January 31 Survey Due Date

7

Experiment 1 Results


•You can lead a respondent to the website, but you can’t encourage an earlier survey completion

Multiple reminders to acknowledge succeeded in decreasing elapsed days until acknowledgement

Multiple reminders to acknowledge had no effect on decreasing elapsed days to survey completion

8



ControlGroup 1

E-mail reminders

Group 2

Phone Reminders

Group 3All

Groups

Mean days from survey launch to acknowledgement

37 days

(n=177)

29 days

(n=179)

30 days

(n=179)

32 days

(n=535)

Response rate at survey deadline

46.3% 49.7% 49.2% 48.4%

9

Experiment 2 – FY 2004 Increase Response Rate at Survey Deadline, Part 2


• Hypotheses:

Respondents who receive more reminders are more likely to respond by the survey deadline

Respondents who receive reminders that vary in mode are more likely to respond by the survey deadline

•During month prior to survey deadline:

Group 1: One e-mail reminder (control group)

Group 2: One e-mail, one mail reminder

Group 3: Two e-mail reminders

10

Experiment 2 Design


Key DatesStudy Groups

Group 1 Group 2 Group 3

January 18-20(expecteddelivery)

ReminderMailing

January 19Reminder E-mail 1

ReminderE-mail 1

January 26 Reminder E-mail 2

January 31 Survey Due Date

11



•Variety and intensity will spice up the response rate.

Group who received both mail and e-mail responded at a higher rate by deadline than group receiving only e-mail

Experiment 4 (FY 2005) validated the effect of two contacts vs. one, but no significant difference between two e-mails vs. e-mail and mailing

12

Experiment 2 Results – Single mode, 1 contact vs. mixed mode, 2 contacts


Group 11 e-mail

Group 21 e-mail

and 1 mailing

Responded by deadline 35.4% 45.9%

51 62

Total 100% 100%

144 135

13

Experiment 3 – FY 2004Increase Timeliness after Survey Deadline


•Hypothesis: Respondents will respond sooner after the deadline if follow-up messages are individualized and less frequent than weekly

•During 24 weeks after survey deadline:

Group 1: Low tailoring – standard weekly messages (control group)

Group 2: Medium tailoring – less frequent messages with different content each time

Group 3: High tailoring –adjusted to individual respondent history (more frequent and tailored contacts for previous non-responders)

14

Experiment 3 Design


Follow-up Period

Group 1 Low Tailoring

Group 2Medium Tailoring

Group 3High Tailoring

Likely non-responders

All others

0-11 weeks

10 e-mail or phone contacts

Same messages

3 e-mails

Varied messages

Dependent on past behavior

Tailored messages

2 e-mails

Same messages

12-24 weeks

12 contacts; alternating e-mail or phone weekly

4 contacts spaced 3 to 4 weeks; e-mail, phone, letter, and final phone call

6 contacts; alternating e-mail and phone every two or three weeks

Same messages Varied messages and senders

Same messages

15



•It is more important to stay in contact than it is to use a particular contacting strategy

Little variation of final response rate across 3 groups

Weekly reminders were not found to perform better than tailored reminders sent every 3 to 4 weeks

16



Group 1Low

Tailoring

Group 2Medium Tailoring

Group 3High

TailoringAll

Groups

Number of respondents

105 95 102 302

Response rate on January 31

43.9% 49.2% 45.5% 46.2%

Final response rate 90.5% 94.7% 90.2% 91.7%

Response added between January 31 and August 10

50.8% 48.1% 49.2% 49.3%

17

Changing Procedures


•Used experimental results as starting point for making changes

•Consulted with Dr. Don Dillman on message crafting and changing survey procedures

•Determined four areas for change:

Acknowledgement process

Mode/frequency of contacts

Extension policy

Survey close-out policy (letters to institution presidents)

18

Acknowledgement Process


•Simplified acknowledgement process in FY 2006 — less burdensome for respondents

Prior to FY 2006, asked respondents to log-on to survey website to acknowledge continuation as survey respondent

Now, respondent replies to e-mail

Prior to FY 2006, 23% acknowledged in first week of survey, 72.5% month before due date

Now, increased to 63% and 98.7%, respectively

Results: Earlier identification of changes in respondents

19

Mode/frequency of Respondent contacts


•Changed mode and frequency, varied content in FY 2006

Pre-due date reminders increased in frequency: Early December – additional e-mail reminder January - 2 reminders (1 mail and 1 e-mail)

Post-due date follow-ups decreased in frequency: First 6 weeks - 3 e-mails (every 2 weeks) 7th week - phone call 8th week - letter to president’s office 9th week – phone call 11th week - final e-mail announcing survey closing date

Results: Increased response rate at deadline from 56% in FY 2005 to 68% in FY 2006, shortened post-deadline follow-up period

20

Extension Policy


•Eliminated unlimited extensions

Prior to FY 2005: Extension requests up to 36 weeks after survey due date

FY 2005 and FY 2006:

Requests for extensions restricted to shorter time frame (only granted for dates before President’s letter)

After president’s letter mail-out, respondents were told survey closes when target response rate reached and 2 week final warning would be given

Results: Shortened post-deadline follow-up period

21

Survey Close-out Policy


•President’s letter was added in FY 2004 as final call for response

Goal set to mail President’s letter to nonrespondents 1 month earlier each year

FY 2004 - May 25 (87% response rate at mailing)

FY 2005 - April 21 (88% response rate at mailing)

FY 2006 - March 23 (92% response rate at mailing)

Results: Earlier survey closeout on May 21, 2007 v. August 10, 2005

22

Response Rate History, FY 1997- FY 2006


0

5

10

15

20

25

30

35

40

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

Fiscal Year

Nu

mb

er

of

po

st-

du

e

da

te w

ee

ks

Weeks to reach final response rate

Weeks to reach 80%

Weeks to reach 88%

23

Final Response Rates, FY 1997- FY 2006


97.9 98.6 98.5 97.3 95.2 96.0 94.1 94.0 93.397.2

0

20

40

60

80

100

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

24

Questions?

For more information contact:Ronda Britt, Academic R&D Survey Manager

[email protected]


increasing survey cooperation: motivating chronic late responders to an annual survey national...

Documents

email group

email reminders group

survey deadline respondents

email experiment

phone reminders group

survey deadline speed

survey launch

time group