un/ece work session on statistical data editing
DESCRIPTION
An embedded experiment to test non-response follow-up strategies when using electronic questionnaires. UN/ECE Work Session on Statistical Data Editing Topic (iv): Using metadata and paradata to analyse the efficiency of editing processes. Claude Poirier * - PowerPoint PPT PresentationTRANSCRIPT
An embedded experiment to testnon-response follow-up strategies
when using electronic questionnaires
UN/ECE Work Session on Statistical Data EditingTopic (iv): Using metadata and paradata to analyse the efficiency of editing processes
Claude Poirier *
Oslo, Norway, 24 – 26 September 2012
* Work carried out by Jeannine Claveau and Claude Turmelle
2
Outline Overview of the Unified Enterprise Survey Collection Methodology of the Experimental Design Results Future work
Statistics Canada • Statistique Canada
Statistics Canada • Statistique Canada3
UES Web-based Collection Electronic Questionnaires (EQ) for 7 surveys
• 6 service industries – stratified samples• Head Office Survey – census
Telephone pre-contact• Confirm industrial classification and ask e-mail address
EQ invitation to units with e-mail addresses Telephone follow-up on failed edits Follow-up of non-respondents
• Experimental design
Statistics Canada • Statistique Canada4
Objectives of the Experiment Evaluate four non-response follow-up strategies
when using EQ for annual business surveys Some questions to be answered:
• Which strategy produces the best response rates?• Which strategy is the most efficient?• How far can we get with e-mail reminders only?• What is the importance of having at least the first
follow-up contact by phone?
Experimental Design
Statistics Canada • Statistique Canada
Overall sample
Telephone pre-contact or heads-up e-mail
Paper questionnaire
mail-out
NRFU by fax/phone reminders
EQ mail-out
T1
1 e-mail, then
+
Phone follow-up
+
3 e-mails,one per month
T2
4 e-mails,one every 2
weeks
T3
Phone follow-up
+
3 e-mails,one per month
T4
1 phone attempt
+
3 e-mails,one every 2
weeks
9,324 units
6,457 units (70%)
03/2
8
04/0
1
04/0
7
04/1
3
04/1
9
04/2
5
04/2
9
05/0
5
05/1
1
05/1
7
05/2
3
05/2
7
06/0
2
06/0
8
06/1
4
06/2
0
06/2
4
06/3
0
07/0
6
07/1
2
07/1
8
07/2
2
07/2
8
08/0
3
08/0
9
08/1
5
08/1
9
08/2
5
08/3
1
09/0
6
09/1
2
09/1
6
09/2
2
09/2
8
10/0
4
10/1
0
10/1
4
0
10
20
30
40
50
60
70
80
90
100
Cumulative unweighted return rate (%) for each treatment(March 28 to October 14, 2011)
T1
T2
T3
T4
Cu
mu
lati
ve
un
we
igh
ted
re
turn
ra
te (
%)
March 28EQ mail-out
03/2
8
04/0
1
04/0
7
04/1
3
04/1
9
04/2
5
04/2
9
05/0
5
05/1
1
05/1
7
05/2
3
05/2
7
06/0
2
06/0
8
06/1
4
06/2
0
06/2
4
06/3
0
07/0
6
07/1
2
07/1
8
07/2
2
07/2
8
08/0
3
08/0
9
08/1
5
08/1
9
08/2
5
08/3
1
09/0
6
09/1
2
09/1
6
09/2
2
09/2
8
10/0
4
10/1
0
10/1
4
0
10
20
30
40
50
60
70
80
90
100
Cumulative unweighted return rate (%) for each treatment(March 28 to October 14, 2011)
T1
T2
T3
T4
Cu
mu
lati
ve
un
we
igh
ted
re
turn
ra
te (
%)
E1
March 28EQ mail-out
E1 -> April 26T1/T2: E-mailT3/T4: phone attempt
03/2
8
04/0
1
04/0
7
04/1
3
04/1
9
04/2
5
04/2
9
05/0
5
05/1
1
05/1
7
05/2
3
05/2
7
06/0
2
06/0
8
06/1
4
06/2
0
06/2
4
06/3
0
07/0
6
07/1
2
07/1
8
07/2
2
07/2
8
08/0
3
08/0
9
08/1
5
08/1
9
08/2
5
08/3
1
09/0
6
09/1
2
09/1
6
09/2
2
09/2
8
10/0
4
10/1
0
10/1
4
0
10
20
30
40
50
60
70
80
90
100
Cumulative unweighted return rate (%) for each treatment(March 28 to October 14, 2011)
T1
T2
T3
T4
Cu
mu
lati
ve
un
we
igh
ted
re
turn
ra
te (
%)
E1 E2
March 28EQ mail-out
E1 -> April 26T1/T2: E-mailT3/T4: phone attempt
E2 -> May 9T2/T4: E-mail
03/2
8
04/0
1
04/0
7
04/1
3
04/1
9
04/2
5
04/2
9
05/0
5
05/1
1
05/1
7
05/2
3
05/2
7
06/0
2
06/0
8
06/1
4
06/2
0
06/2
4
06/3
0
07/0
6
07/1
2
07/1
8
07/2
2
07/2
8
08/0
3
08/0
9
08/1
5
08/1
9
08/2
5
08/3
1
09/0
6
09/1
2
09/1
6
09/2
2
09/2
8
10/0
4
10/1
0
10/1
4
0
10
20
30
40
50
60
70
80
90
100
Cumulative unweighted return rate (%) for each treatment(March 28 to October 14, 2011)
T1
T2
T3
T4
Cu
mu
lati
ve
un
we
igh
ted
re
turn
ra
te (
%)
E1 E2 E3
March 28EQ mail-out
E1 -> April 26T1/T2: E-mailT3/T4: phone attempt
E2 -> May 9T2/T4: E-mail
E3 -> May 26T1/T2/T3/T4: E-mail
03/2
8
04/0
1
04/0
7
04/1
3
04/1
9
04/2
5
04/2
9
05/0
5
05/1
1
05/1
7
05/2
3
05/2
7
06/0
2
06/0
8
06/1
4
06/2
0
06/2
4
06/3
0
07/0
6
07/1
2
07/1
8
07/2
2
07/2
8
08/0
3
08/0
9
08/1
5
08/1
9
08/2
5
08/3
1
09/0
6
09/1
2
09/1
6
09/2
2
09/2
8
10/0
4
10/1
0
10/1
4
0
10
20
30
40
50
60
70
80
90
100
Cumulative unweighted return rate (%) for each treatment(March 28 to October 14, 2011)
T1
T2
T3
T4
Cu
mu
lati
ve
un
we
igh
ted
re
turn
ra
te (
%)
E1 E2 E3 E4
March 28EQ mail-out
E1 -> April 26T1/T2: E-mailT3/T4: phone attempt
E2 -> May 9T2/T4: E-mail
E3 -> May 26T1/T2/T3/T4: E-mail
E4 -> June 7T2/T4: E-mail
03/2
8
04/0
1
04/0
7
04/1
3
04/1
9
04/2
5
04/2
9
05/0
5
05/1
1
05/1
7
05/2
3
05/2
7
06/0
2
06/0
8
06/1
4
06/2
0
06/2
4
06/3
0
07/0
6
07/1
2
07/1
8
07/2
2
07/2
8
08/0
3
08/0
9
08/1
5
08/1
9
08/2
5
08/3
1
09/0
6
09/1
2
09/1
6
09/2
2
09/2
8
10/0
4
10/1
0
10/1
4
0
10
20
30
40
50
60
70
80
90
100
Cumulative unweighted return rate (%) for each treatment(March 28 to October 14, 2011)
T1
T2
T3
T4
Cu
mu
lati
ve
un
we
igh
ted
re
turn
ra
te (
%)
E1 E2 E3 E4
March 28EQ mail-out
E1 -> April 26T1/T2: E-mailT3/T4: phone attempt
E2 -> May 9T2/T4: E-mail
E3 -> May 26T1/T2/T3/T4: E-mail
E4 -> June 7T2/T4: E-mail
E5 -> July 8T2/T4: NRFUstarted
E5
Statistics Canada • Statistique Canada12
Percentage of units receiving NRFU
13
Savings
Statistics Canada • Statistique Canada
0 4 8 12 16 200
20
40
60
80
100
Unweighted Return Rate vs. Time per Unit for NRFU
as of October 14, 2011
T1
T2
T3
T4
TPU for NRFU (minutes)
Un
we
igh
ted
Re
turn
Ra
te (
%)
Statistics Canada • Statistique Canada14
Summary of results E-mail reminders generated “spikes” in responses Sending only e-mail reminders achieved a 45%
response rate at very little cost Collection staff can catch up to target response rate at
the end of the collection period, when needed. E-mail reminders help to reduce the cost in terms of
call attempts (≈ 30%) and TPU (≈ 25%) Treatment 2 was the most efficient
Statistics Canada • Statistique Canada15
Future Work Treatment 2 will become the standard for EQ All UES surveys will continue to collect e-mail
addresses In 2014, we will implement EQ as the principal
collection mode for business surveys An active collection management based on
estimates and their quality will be implemented in order to further reduce telephone follow-up
Statistics Canada • Statistique Canada16
Thank you Merci
For more information, please contact:
Claude Poirier
Jeannine Claveau
Claude Turmelle
Pour plus d’information, veuillez contacter :