© 2009 Knowledge Networks, Inc.

Mario Callegaro

Charles DiSogra

Knowledge Networks

Computing response metrics foronline panels

DC AAPOR Workshop on Web Survey Methods, September 9 th 2009

What metrics for what panel

Pre-recruited probability-based online panels Response rates can be calculated because the frame is known

(AAPOR, 2006)

Volunteer opt-in panels “Response rates” cannot be computed (AAPOR, 2007) However, other metrics can be calculated, e.g. completion rate

2

3

Current status• Volunteer, non-probability (opt-in) panels, widely used in

market research, outnumber probability-based Web panels

• More and more probability-based online panels being built 2007-2009 American National Election Studies (ANES) Panel Face-to-Face Recruited Internet Survey Platform (FFRISP, 2008) Dutch Long-term Internet Study for the Social Science (LISS) panel (2007)

• Still no officially agreed standard on how to compute response rates for online panels

Review of current standards

Many efforts and proposals by different national and international organizations:

European Society for Opinion and Marketing Research – ESOMAR European Federation of Associations of Market Research Orgs. –EFAMRO Interactive Marketing Research Organization – IMRO Advertising Research Association – ARF quality initiative Bob Lederer proposal endorsed by the American Marketing Association

(AMA) Latest effort by ISO (standard #26362) touches on subject

4

Some journals are giving guidelines on how response rates should be computed specifically for online surveys (not necessarily online panels)

Journals enforcing AAPOR standards: (e.g. POQ, IJPOR…) Journal of Medical Internet Research

Journal of Medical Internet Research (Eysenbach, 2004):

Journal recommendations

5

In online surveys, there is no single response rate.

Rather, there are multiple potential methods for calculating a response rate, depending on what are chosen as the numerator and denominator.

As there is no standard methodology, we suggest avoiding the term “response rate” and have defined how, at least in this journal, response metrics such as, what we call, the view rate, participation rate and completion rate should be calculated.

ESOMAR and IMRO examples

• ESOMAR (2005) metrics: “Response based on the total amount of invites (% of full numbers)

per sample drawn (country, questionnaire) % questionnaire opened % questionnaire completed (including screen-out) % in target group (based on quotas) % validated (the balance is cleaned out, if applicable)” (p. 20).

• IMRO (2006) metrics: Response rate is “based on the people who have accepted the

invitation to the survey and started to complete the survey. Even if they are disqualified during screening, the attempt qualifies as a response” (p. 13).

Completion rate “is calculated as the proportion of those who have started, qualified, and then completed the survey” (p. 13).

6

AMA platform for data quality progress:

7

removal)for Request or Errors ks,(Bouncebac - tss/intercepinvitation Total

responses attempted of #Tot Rate Response

criteria screening passing # Total

CompletesRate Completion

criteria screening passing # Total

resuming without pausingor Quitting #Raten Terminatio-Mid

responses attempted # Total

criteria screeening passing # TotalRateion Qualificat

Platform for Data Quality Progress, Bob Lederer(under AMA umbrella, Nov 2008)

ISO 26362:2009

• Participation rate: ‘number of panel members who have provided a usable response divided by the total number of initial invitations requesting members to participate (p. 3)

• Usable response is one where the respondent has provided answers to all the questions required by the survey design

• The term “response rate’ cannot be used to describe respondent cooperation for access panels

8

Necessary information to compute response metrics

• In order to compute response metrics for online panels we need to understand how panel members are recruited and what stages are used to build a panel

• Volunteer-opt-in design

• Probability-based design

9

Generalized volunteer opt-in panel design

• Stage 1: Encounter, discover, or seek out to join

• Stage 2: Provide profile information

• Stage 3: Get and do surveys

10

Volunteer opt-in panels: Stages

11

Respondent decidesto opt-in

Opt-in panel portalEnter some basic

information

Email confirmation(double opt-in)

Profile survey

Double opt-in

Single opt-in

Active panel

Postoaca, 2007

Stages for probability-based online panels

• Stage 1: Recruitment from frame

• Stage 2: Welcome and get profiled

• Stage 3: Active membership, ready for surveys, actual study

12

Common steps in building a probability-based panel

1. Recruitment Rate (RECR): the recruitment of potential panel members

Recruitment rate calculation will depend on the recruitment mode: face -to-face, telephone, mail

2. Profile Rate (PROR): empanelling recruited persons This stage counts panel members that answered their profile survey,

generally a questionnaire collecting background information and welcoming respondents to the panel

The computation of the profile rate (a.k.a., connection rate) will depend on the data collection mode

Profiled members are considered to be “active members” in the pool from which study samples can be drawn

13

Probability-based design features Implications for computing response rates

1. Single recruitment cohort (one-time effort) vs. multiple recruitment cohorts (on-going recruitment)

2. Within-household selection to recruit one person vs. whole household recruitment of all eligible persons

3. The data collection mode used for non-internet households (no access to online surveys at time of recruitment)

14

Methods of dealing with non-Internet households

15

Probability-basedsample

Internethousehold

Non- Internethousehold

Member(s) use theirdevice and Internet

connection tocomplete surveys

Member(s) aregiven a device andInternet connectionto complete surveys

Member(s)complete surveys in

another mode

All members are given adevice to complete

surveys no matter theirInternet status

Mail Phone IVR

Assessment ofInternet status

Probability- based web panels: Recruitment

16

Probability Sample (PS)

Knowneligibility

Unknowneligibility(UH UO)

EligibleNot

eligible

Initial consent(IC)

Stage 1 Recruitment

Refusals andbreak offs

(R)

Non -contacts

(NC)

Others(O)

Probability- based web panels: Profile

17

"Profile"survey

Returnedquestionnaire

(I or P)Connection stage

Eligible - noninterview

(NC)

Refusals andbreak offs

(R)

Non -contacts

(NC)

Others(O)

Probability- based web panels: Actual studySame design for volunteer-opt-in panels

18

Active Panel (AP)

Sample frame forspecific study (SF)

Complete(I)

Partial(P)

Break-off(R)

Refusal(R)

Non contact(NC)

Other noninterview

(O)Eligible?

Not eligibleUnknowneligibility(UH UO)

NoN

o

Yes

Yes

Prescreening

on database

Active panel dynamics

19

Active Panel

Temporaryinactive

Notavailable

forsampling

Continuousrecruitment

Involuntaryattrition

Voluntaryattrition

Mortality

Re-recruitment

Re-recruitment

Stage 1 of probability-based web panels

IC = Initial consent

R = Refusal

UH and UO = Unknown if household or unknown “other”

NC = Non-Contact

O = Other non-interview

e = Estimated proportion of unknown eligibility cases

R Refusal (REFR) = Rate IC + (R + NC + O) + e(UH + UO)

IC Recruitment (RECR) = Rate IC + (R + NC + O) + e(UH + UO)

20

Example: P_RECR = .4 x 100% = 40%

I = Profile survey complete

P = Profile survey partial but acceptable

Stage 2 (more likely for probability-based panel)

* Opt-in panels may not know the denominator components.

(I + P) Profile Rate (PROR) =

(I + P) + (R + NC + O)*

RRefusal to Profile (REFP) =

(I + P) + (R + NC + O)*

21

Example: PROR = .6 x 100% = 60%

Stage 3 Specific Study Rates

BF = Break-offs -- when the number of answers is below the definition of partial interview, it can be considered a break-off. R = Other than for the break-off rate, R includes break-offs as refusals

(I + P) Completion Rate (COMR) =

(I + P) + (R + NC + O)

BF Break-off Rate (BFR) =

(I + P) + BF

Study R Refusal (SREF) = Rate (I + P) + (R + NC + O)

22

Example: COMR = .7 x 100% = 70%

Cumulative Response RateOnly for pre-recruited probability-based online panels

P_RECR = Person recruitment rate

PROR = Profile rate

COMR = Completion rate for the single study

RETR = Retention rate

A multiplicative function

Cumulative RR (CURR) = P_RECR x PROR x COMR

23

Cumulative RR2 (CURR) = P_RECR x PROR x RETR x COMR

Example CURR= .4 x .6 x .7 = .168 x 100% = 16.8%

Example CURR2= .4 x .6 x .8 x .7 = .134 x 100% = 13.4%

Recruitment

CohortRecruitedmembers

RecruitmentRate Profiled

members

ProfileRate

Activemembers

RetentionRate

Sample

Respondents

Completionrate

The computation of a CUMRR isstraightforward when the panelis built with a single recruitment cohort

Computing CUMRR with 1 cohort

24

Study Respondents

RECR PROR RETR

COMR

Recruitment

Cohort1

Recruitedmembers

RecruitmentRate Profiled

members

ProfileRate

Activemembers

RetentionRate

Sample

Recruitment

Cohort2

Recruitedmembers

RecruitmentRate Profiled

members

ProfileRate

Activemembers

RetentionRate

Sample

Recruitment

Cohort3

Recruitedmembers

RecruitmentRate Profiled

members

ProfileRate

Activemembers

RetentionRate

Sample

Unequal cohort contributions to a

study sample selected from

among all active members

25

Computing CUMRR with 3 cohorts

Formulas dealing with multiple cohorts (1.)

RECR, PROR, RETR are calculated as the weighted average of the size contribution of each cohort

Example to calculate RECRtotal

26

cnccc

cncncccccctotal WWWW

RECRWRECRWRECRWRECRWRECR

...

...

321

332211

Where Wcn = the number of cases contributed to the sample from cohort n

Example of RECR with 3 cohorts

Cohort 1 Cohort 2 Cohort 3

Size in the final sample

200 100 50

Recruitment rate (RECR)

.35 .27 .15

27

50100200

15.5027.10035.200totalRECR

2985.350

5.104

350

5.72770

totalRECR

Formulas dealing with multiple cohorts (2.)

28

cnccc

cncncccccctotal WWWW

RECRWRECRWRECRWRECRWRECR

...

...

321

332211

cnccc

cncncccccctotal WWWW

PRORWPRORWPRORWPRORWPROR

...

...

321

332211

cnccc

cncncccccctotal WWWW

RETRWRETRWRETRWRETRWRETR

...

...

321

332211

Full example with 3 cohorts

Cohort 1 Cohort 2 Cohort 3 ___Rtotal

Size 200 100 50

RECR .35 .27 .15 .299

PROR .57 .65 .70 .611

RETR .50 .67 .85 .599

29

%0.13%100130.713.611.299.CUMRR1total %8.7%100078.713.599.611.299.CUMRR2 total

Assume a survey completion rate (COMR) of .713

Non- Internethousehold

Member(s) aregiven a device andInternet connectionto complete surveys

Member(s)complete surveys in

another mode

Mail Phone IVR

Computing completion rate (COMR) when multiple data collection modes are used

Completion rates need to be computed separately for each mode

Web survey Mail, phone or IVR

These rates should also be combined as a weighted average

30

Technical condition in order to compute response metrics

• In order to compute response metrics each panel organization must keep an historical database with rates for each member

• More specifically for probability-based online panels it is necessary that: Each panel member ever recruited must have a record of his/her:

– Recruitment rate cohort value– Profile rate cohort value– Retention rate cohort value

31

Which formula for which panel?

Metric Probability-based

Volunteer opt-in

Recruitment Yes N/A

Refusal to be recruited Yes N/A

Profile Yes Maybe

Refusal to profile Yes Maybe

Screening Yes Yes

Eligibility Yes Yes

Completion Yes Yes

Break-off Yes Yes

Refusal Yes Yes

Cumulative Response Yes N/A32

Which formula for which panel? II

Metric Pre-recruited Volunteer

Attrition cross sectional Yes Yes

Attrition longitudinal Yes Yes

Reinterview Yes Yes

33

Dutch study (Vonk, van Ossenbruggen, & Willems, Esomar 2006)

Panel Management or Manipulation?

34

Some factors affecting each rateRecruitment rate

Recruitment methods Incentives

Profile rate Incentives Panel management efforts

Retention rate Time elapsed since recruitment Incentives Panel management efforts

Survey completion rate Field time Incentives Reminders

35

References

• Callegaro, M. and DiSogra, C. (2008). Computing response metrics for online panels. Public Opinion Quarterly, 72, pp. 1008-1032.

• DiSogra, C. and Callegaro, M. (forthcoming). Computing response rates for probability based web panels. In American Statistical Association (Ed.). Proceedings of the joint statistical meetings: section on survey research methods [Cd-Rom]. Alexandria, VA: American Statistical Association.

36

Future work

• Recruitment level computed at a household or at a person level (when recruiting multiple members per household)

• Attrition rates for cross sectional design

• Attrition rates for longitudinal designs

• Response rates for longitudinal designs

37