Online Opinions – A Pilot to Extend ONS Social Data Collection Capabilities

Ed DunnONS Social Survey Division

Overview

• Background and context setting• The pilot study; our key objectives and design• Findings • Next steps

Pilot Key Objectives – ‘Dipping our Toes’

• Need to examine web collection• Investigate unit and item response rates• Investigate respondent characteristics• Evaluate the hardware/software capabilities

Examine the effect of offering an incentive• Design and administration of web surveys• Recommendations for further work

• Deliver to a short timescale (by December 2008)

Sample Design and Respondent Selection – Web Pilot

• Simple random sample of 2,000 households from English Postal Address File

• Advance letter • Half offered a £10 Amazon gift certificate to

take part online• Letter asked adult with most recent birthday

to complete the survey• Unique username and password provided to

access survey from ONS website• Follow-up letter

Sample Design and Respondent Selection- Opinions Survey (OPN)

• OPN is multi-stage cluster design • Respondent selection via KISH grid• November OPN sampled 2,010 households• Book of postal stamps sent with advance

letter

Questionnaire Design (1)

• Based upon ONS Opinions Survey (OPN)• Used question blocks equivalent to a ‘normal’

month of OPN• Questions reviewed for suitability for web• Some modifications necessary• Questionnaire tested internally

Questionnaire Design (2)

• Design and layout reviewed for web survey*

• Page Design

• Next/Back Button

• Progress Bar

* See Dillman, D.A (2007) Mail and internet surveys - the tailored design method (2nd edition) J Wiley & Sons, New Jersey

Questionnaire Design (2)

Unit ResponseInternet pilot

• 364 fully complete responses received (18%)• 32 partial responses (2%)• Little difference between incentive (18%) and non-

incentive (19%) group

• But, Full sample = 2000

10% unknown eligibility and only 61% have internet access at home (LCF)Available sample = 1098 New Response = 33% complete responses

November OPN

• 59% overall response (with known eligibility)

Metrics

• 80% of respondents completed web survey in under 25 minutes

• 75% completed 8am-6pm• Most popular hours to begin survey were

4pm-5pm (10%) and 7pm-8pm (11%)• 25% completed on weekend• Wednesdays and Fridays most popular

• OPN face to face takes c.45mins (excl. admin time)

Results (1)

• Weighted results; OPN routinely weighted but pilot required some methodological work. OPN was then re-weighted to same population totals.

• 2-tailed t test applied to differences

• Some clear differences in profile of internet pilot respondents compared to OPN and key question responses

Results (2)

• The pilot respondents were more likely to be:• Aged 25-44 and 55-64 • Married and living with partner• White • Better educated (with degree level

qualification)• Managers or supervisors • In good health

Results (3)

• In terms of key question responses pilot respondents were less likely to:

• Smoke• Have a disability• Think charities played an important role in

society• Think HM Revenue & Customs treated them

fairly

• Pilot respondents were more likely to:• Eat healthily (5 portions of fruit or vegetables a

day)

Next Steps

• Limitations of this pilot widely accepted• Pilot has demonstrated viability of conducting a major

ONS social survey online and receiving a significant, if minority, response

• Pilot has highlighted logistical and technological issues

• Pilot has highlighted risk of significant bias• Identified areas for further work via more

sophisticated pilot designs• Plan to run a further pilot on OPN in 2010 and pilot

on LFS early in 2011

References

• Dillman, D.A. 2007. Mail and Internet Surveys – The Tailored Design Method (2nd ed). J Wiley and Sons, New Jersey

• Flatley, J. 2001. The Internet as a Mode of Data Collection in Government Social Surveys: Issues and Investigation Social Survey Methodology Bulletin, ONS 49(7)

• Fricker, R and Schonlau, M. 2002. Advantages and Disadvantages of Internet Research Surveys: Evidence from the Literature. Field Methods 14(4)

• Solomon, D. 2001. Conducting Web-Based Surveys. Practical Assessment, Research and Evaluation 7(19)

Internet and the UK LFS

Dean FletcherUK ONS

Internet and the UK LFS

•Current Modes:

•Wave 1 Face to Face Wave 2-5 TU or ftf

•Response on UK LFS all time low 65% wave 1, 57% overall.

•Achieved sample size decreasing or much more costly to obtain

•Efficiency savings face to face costs 3 times TU•Pilot shows some groups with current high non-response will take part in the internet.•TU costs 3 times Internet.

Internet and the UK LFS

• Pilot showed technology exists.• 4 main Problems:• Collecting close to the Reference week • Problem of integrating the internet with the

other modes to prevent bias• Mode effects• Collecting from Households not persons

Face to Face W1 TU W2-5

Internet and the UK LFS

• Analysed other approaches but derived duel sample approach after considering results of the pilot

Initial Sample

Registered for internet

Respond to internet

FtF W1 TU W2-5

Internet and the UK LFS

• Currently reviewing questionnaire for internet suitability based on pilot results and others research to minimise mode effects.

• Household collection seems feasible• Use Confirmit to test ideas• Use BLAISE IS? To run pilot in 2011 in a few

NUTS2 areas• If successful run full design by 2012• Modify collection by 2013

Internet and the UK LFS

• Issues

• Internet pushes up refusals

• Unequal response

• Changes to internet over time

Internet and the UK LFS

• Also monitoring wave 2-5 work by other NSi’s to implement if shown to be successful

• But current systems do not allow internet, TU and ftf integration.

• Cost savings and non-response mainly achieved for UK at Wave 1.

Internet and the UK LFS

• So far we have had:

• OPN Pilot

• Input at International Workshop

• Prof. Mick Couper’s work and training

• Literature review

• But interested to have design scrutinised for issues.

Internet and the UK LFS

• Concluding remark

• UK Can’t afford not to implement an internet collection mechanism if possible in the next 2 years.