electronic questionnaire collection at statistics canada milana karaganis marc st-denis

Electronic Questionnaire Collection at

Statistics Canada

Milana KaraganisMarc St-Denis

Outline

Electronic Questionnaire project Electronic Questionnaire for business surveys

23-04-21Statistics Canada • Statistique Canada2

Electronic Questionnaire Project Build generic EQ service to be used by all surveys

• EQ platform

• Standard approach to EQ Create a critical mass of EQ service users

• 160+ surveys

• Business surveys and some social surveys Reduce collection and operations costs Timeline: April 1, 2010 -> March 31, 2016


Electronic Questionnaire Project

E-questionnaire solution

• Support all EQ applications for all surveys

• Use Statistics Canada infrastructure

• Comply with security and confidentiality standards

• Automate EQ survey application development -> build an engine to render EQ applications based on provided inputs, E-questionnaire Generation System (EQGS)




Respondent

EQ Collection Portal (with EQ application for a

survey)

EQ backend:- Respondent repository

- Database- E-mail service

- Data management service- Survey management service

- Questionnaire generation system

etc.

PUBLIC ACCESS ZONE

Statistics Canada internal network


Standard approach to EQ – corporate solution• Develop EQ applications according to EQ standards

• Apply standard look and feel

• Build functionalities required for all surveys

• Use standard collection strategies

EQ as initial collection mode in sequential mode collection

• Use standard development process

Shorten development cycle



Progress To-Date

EQ solution implemented• Change Management process

• First set of EQ standards released

• Survey development process created

• Initial collection strategies established

Deployed 39 EQ applications and realized efficiencies

Overall success:• Lessons learned

• Big cultural shift

EQ Among Business Surveys

Outline• Evolution of EQ collection strategies

• Results observed to date

• Survey collection strategy guidelines

• Results of tests


Collection Strategies Among Business Surveys

Moving from conventional collection modes (PAPI & CATI) to EQ (over 100 surveys in 5 yrs)

Evolution of multi-modal collection with EQ as primary mode

Measuring results and improving • Respecting data quality, timeliness, budgets

Reviewing collection strategies


Reviewing Collection StrategyQuarterly Services Indicators (QSI) Survey


Services Indicator Survey (quarterly ) (QSI) 2011

Q4 2011 Q1 2012

Mail out / E-invitation Oct 3, 2011 Mail out / E-invitation Jan 4, 2012e-reminder / fax Oct 17 e-reminder Jan 18Phone reminder (all) Oct 24 e-reminder Jan 25Phone reminder (all) Nov 7 e-reminder Feb 8e-reminder / fax Nov 14 Phone reminder (all) Feb 15e-reminder / fax Nov 28 e-reminder Feb 29Fax to EQ Dec 5 Phone and fax (all) Mar 7End of collection Dec 9, 2011 End of collection Mar 9, 2012

Q4 2011 EQ Mail Total

Final Sample 1,525 98.3% 27 1.7% 1,552

Response Rate 58.5% 50.0% 58.0%

# of cases in NRFU 905 59.3% 22 81.5% 927


Q1 2012 EQ Mail Total

Final Sample 1,351 96.4% 50 3.6% 1,401

Response Rate 65.6% 46.2% 65.6%

# of cases in NRFU 683 50.6% 41 82.0% 724

Reviewing Collection StrategyQuarterly Services Indicators (QSI) Survey

EQ Collection Strategy Guidelines

Developing tools to ensure optimal collection strategies (monthly; quarterly; annual)• Follow-up schedule geared to multi-mode collection

Standardise approach towards efficiencies Considering respondent burden, data quality,

response rates, costs


Date Days after Mail out Type of FUPre-contact n/aMail-out / E-invitation send date: 0Collection Start date:

Bounce-backs follow up 21 week max to complete then switch to mail for those unable to get correct email

1st Follow up attempt date: Phone 30 only for those without e-mail address (mail units)EQ 30 1st reminder emailFax X no fax unless requested

Mail X no mail unless requested

2nd Follow up attempt date: Phone 60 only for those without e-mail address (mail units)EQ Every 2nd week 2nd reminder emailFax X no fax unless requested


3rd Follow up attempt date: Phone 90 only for those without e-mail address (mail units)EQ Every 2nd week 2nd reminder emailFax X no fax unless requested


4th Follow up attempt date: Phone 120 only for those without e-mail address (mail units)EQ Every 2nd week 3rd reminder email

Fax 120fax reminder + copy of questionnaire to those without email address

Mail 120 mail only to those without email address or fax number

5th Follow up attempt date: Phone 140 all outstandingEQ Every 2nd week 5th reminder email

Fax 140fax reminder + copy of questionnaire to those without email address

Mail 140 mail only to those without email address or fax number

6th Follow up attempt date: Phone 160 all outstanding unitsEQ 105 6th and final reminderFax X no fax

Mail X no mail

EQ Experimental Design for the UES

Mailout Mar 18E-invitation Mar 28

T1 (Standard)

T2(E-mails only)

T3(phone attempt)

T4(phone attempt + e-mails)

Reminders E-mail only E-mail only 1st attempt – phone;3 next – E-mail

1st attempt – phone;3 next – E-mail

Reminder schedule

1st April 262nd May 263rd June 234th July 25

1st April 262nd May 93rd May 264th June 7

1st April 26-292nd May 263rd June 234th July 25

1st April 26-292nd May 93rd May 264th June 7

Phone follow-up Yes No (firm appt only)

Yes No (firm appt only)

Experiment end date

August 1 June 20*July 11

August 1 June 20*July 11

Sample size allocated to the treatment

1125 1102 1148 1114


Results of the experimental design

T1 T2 T3 T4 %u %u %u %uApril 26 13% 12.5% 15% 12%May 9 24% 22% 24% 22%May 26 36% 41% 37% 45%June 7 44% 43% 46% 49%June 20 51% 46% 53% 52%August 1 66% 54% 68% 58%Sept 12 77% 76% 78% 75%Oct 14 82% 81% 82% 81%

Progressive unweighted return rate (%) for each treatment (2011)

T1 T2 T3 T4

3.8 2.8 4.3 3.9

Number of call attempts by survey treatment

17 Statistics Canada • Statistique Canada 23-04-21

Highlights

Treatments consisting of 4 e-reminders only resulted in 46+% as of June 20th

At end of collection (Oct 14) the 4 treatments had very similar response rates

T2 showed the lowest attempts and lowest TPU Limited worth conducting telephone NRFU for

first few months; still reach similar final response rates with fewer phone attempts


Access Code Letter

Piloted for UES 2012 collection Conducted pre-contact for birth cases in sample Unable to contact cases = mailed an access

code letter providing hyperlink to an EQ More difficult cases since unable to reach during

pre-contact After 30 days, telephone NRFU


Access Code Letters: Results

StatusAccess code letter

# units (total)

Resp. (EQ) Resp. (Mail)

Non-Resp. Out of Scope

In Progress

Total 765 150 (20%) 39 (5%) 52 (7%) 74 (9%) 450 (60%)

Collection status on July 16, 2012 for units that received an access code letter

20 Statistics Canada • Statistique Canada 23-04-21

The Future Successful in achieving its objectives Mission critical monthly surveys UES (60 surveys) + LFS Add new functionalities to EQ platform Respondent EQ = Interviewer EQ Assess respondent’s level of tolerance to edits Streamline processes and operations Monitor quality, response rates, costs Research



[email protected]

[email protected]

mailto:[email protected]

electronic questionnaire collection at statistics canada milana karaganis marc st-denis

Documents