Data Burger:Response Rate Improvement

andBasic Quantitative Data Analysis

Nikki DettmarNational Network of Libraries of Medicine

Outreach Evaluation Resource CenterJuly 22, 2014

This webinar is adapted from classes created by Cynthia Olney, PhD, NN/LM Outreach Evaluation Resource Center

Today’s Topics

• Response rate overview:

– What is a ‘good’ response rate?

– Maximizing response rate

– Assessing response rate

• Basic quantitative data analysis– Descriptive statistics

– Levels of data

– Graphs and charts

(Adapted from a class created by Cynthia Olney, PhD,

Outreach Evaluation Resource Center)

Part One:

Response Rate Overview

Why use a questionnaire?

• Learn about characteristics of a group

• Evaluate quality of programs and services

• Document results of programs and services

OERC Resources Questionnaire Design

• Details about questionnaire design:http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st1

• Webinar recording:https://webmeeting.nih.gov/p90742547/

• “Writing Good Questions” handout:http://nnlm.gov/pnr/training/Writing_good_questions.pdf

Questionnaires primarily gather quantitative data

• Exception:– Open-ended questions gather qualitative data

– Information about basic qualitative data analysis is here:

http://nnlm.gov/evaluation/booklets508/bookletThree508.html#qual

Three goals when using questionnaires

Collect just the right amount of data

Write questions that get the information you want

Get a high rate of participation from respondents (response rate)

GOAL 3: GET A HIGH RATE OF PARTICIPATION FROM RESPONDENTS (RESPONSE RATE)

Why?

“A low cooperation or response rate does more damage in rendering a survey's results questionable than a small sample, because there may be no valid way scientifically of inferring the characteristics of the population represented by non-respondents.”

American Association of Public Opinion Research, 2002,Standards and Best Practices

Equation

# of completed and partially completed questionnaires

# of eligible participants in your sample

Response rate defined

What is a “good” response rate?

A rule of thumb from a standard textbook:

• 50% is adequate• 60% is good• 75% is very good

The Practice of Social Research. Earl R. Babbie. Belmont, Calif : Wadsworth Cengage, 2007.

Standard• 5-day • 1997: 36% • 2003: 25%

Rigorous• 21 weeks• 1997: 61%• 2003: 50%

Pew Research Center Experiment*

*Telephone Survey

Response rates have been declining

http://poq.oxfordjournals.org/cgi/content/full/70/5/759

Dillman has found the following principles increase response rates

Dillman, et al., Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method, Third Edition. Hoboken: Wiley, 2009.

Perceived

CostPerceived reward

Trust Response rate

Here’s how to… Decrease cost

Print questionnaires:

• Make questionnaires as short as possible (1-2 pages)

Online questionnaires

•Make first questions easy to complete•Describe computer actions at point of usage•Use scrolling rather than screen-to-screen format•Show progress bars•Don’t require responses to items

Here’s how to… Increase trust

All questionnaires Use a credible sponsor Have sponsor send out preliminary mailing of endorsement Use a “special” follow-up Use first-class postageOnline questionnaires Use motivational welcome screen (instead of cover letter)

Improving response rates

• Short surveys (1-2 pages)

• Special third contact (e.g., certified mail, telephone call)

• Content meaningful to respondents

• Government sponsorship (as opposed to corporate or marketing firm)

• Survey population (employee, school, military)

• Pre-paid incentives (cash works best)

Improving response rates

http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st2

Effective Mail Procedure

Timing (approx) Activity

Day 1 Pre-notification

Day 5 USPS questionnaire w/stamped return envelope + $2

Day 18 USPS replacement to nonrespondents (thank-you postcard to respondents)

Day 32 FedEx final replacement to nonrespondents(thank-you postcard to respondents)

Adapted from:Don A. Dillman,Washington State UniversityJanuary, 2010

Surface mail more effective than email for healthcare providers

Klabunde, CN, et al. “Facilitators and Barriers to Survey Participation by Physicians: A Call to Action for Researchers.” Evaluation & the Health Professions 2013 36(3):279-295.

http://nnlm.gov/evaluation/booklets508/bookletThreeFigures508.html#F5

Timing (approx) Activity

Day 1 Pre-notification

Day 3 Email with link to questionnaire

Day 9 Email reminder to non-respondents with URL link

Day 13 Email and USPS mail questionnaire to non-respondents

Day 17 Email reminder to non-respondents

Adapted from:Don A. Dillman,Washington State UniversityJanuary, 2010

Example of email/online administration

http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st2

Improving response rates

Pre-notification letter/email:

• Signature that respondents know and trust. Also, for electronic questionnaires, use that person’s name and email address in the “FROM” field if possible.

• Briefly describe the project and why it is important that they respond. (Emphasize how it is important to them as well as how their responses are important to you and your organization.)

• Explain when and how they can expect the questionnaire and who will send it to them.

• Thank them in advance for their help.

Tips for pre-notification letters

http://nnlm.gov/evaluation/booklets508/bookletThreeFigures508.html#F6

Cover letter/email:

• Briefly describe your project and why you want respondents to reply to your questionnaire.

• Include a motivational appeal and explain how to complete the questionnaire.

• Describe who will see their individual responses and how you will maintain confidentiality and any risks to the respondent if he or she chooses to respond.

• Describe incentives.

Tips for cover letters

http://nnlm.gov/evaluation/booklets508/bookletThreeFigures508.html#F6

Checklist for reminder letters/emails:

• State that you are sending a reminder for them to complete the questionnaire.

• Thank those who have already responded.

• Request those who have not responded to do so by a given date.

• For electronic questionnaires, include a link to the questionnaire in every follow-up.

Tips for reminder letters

http://nnlm.gov/evaluation/booklets508/bookletThreeFigures508.html#F6

Incentives work better when sent before or with the questionnaire

Obligation is cheaper than motivation

Cost ofMotivation

Cost ofObligation

• Prepaid cash had strongest effect

• Contributions to charity had little effect

• Lotteries had no effect

Warriner K, Goyder J, Gjertsen PH, McSpurren, K. Charities, no; lotteries, no; cash, yes: Main effects and interactions in a Canadian incentives survey. Public Opinion Quarterly, 1996, 60(4), 452-563 3

Incentives: cash is best

Bigger incentives are not necessarily better

Adapted from:Don A. Dillman, Washington State University, January, 2010

• Use time-tested methods to boost response rates

• Check for bias regardless of response rates

• Use more than one source of information to assess the accuracy of your findings

Summary: Maximizing response rates

http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st2

Use some key tools to assess response rates

Strategy 1 Compare sample profile to population

Strategy 2 Compare early and late responses

Strategy 3 Contact a small percentage of non-respondents with key questions for comparison

• Purposeful sampling: quota sampling

• Triangulate (corroborate) - collect information from multiple sources and look for consistency

More tools to assess response rates

(Adapted from a class created by Cynthia Olney, PhD,

Outreach Evaluation Resource Center)

Part Two:

Basic Quantitative Data Analysis

You probably have been – or will be – called to present a "case" for your programs

It helps to approach data analysis in an organized way.

• Gather data (use credible procedures)

• Summarize (reduce and display data)

• Apply findings (pass judgment & make decisions)

http://nnlm.gov/evaluation/booklets508/bookletThree508.html#st3

There is a process that will help you analyze and use data

• Learn the logic of evaluation

• Use data to describe what happened

• Judge value

• Make decisions

Step One: Learn the

logic of evaluation

Evaluation is a logical process

1. Establish criteria

3. Describe what happened

4. Compare with standards

2. Construct standards

5. Make judgment

6. Make recommendations

Step Two:

Describe what happened

Data analysis starts with gathering your data

Trees Forest

Evaluations usually require analysis of quantitative and qualitative data

Quantitative methods for numbers

Qualitative methods for text

Today’s focus

http://nnlm.gov/evaluation/booklets508/bookletThree508.html#ste3

Excel is a good program to use for gathering your data.

Handy Excel tip: "Format Cells" feature helps you format data

Use Excel Functions to

perform calculations for you. For this, I

used the COUNT function:=COUNTIF(A2:A1675,“x")

Handy Excel tip 2: "Format Cells" feature helps you format the data

Using your mouse,

“right-click” to display this

menu to format a

number as a percentage.

Descriptive statistics summarize distributions of numbers

• Frequencies• Percentages• Charts• Measures of Central Tendency (mean, median, mode)

Frequency distributions show patterns of response

Yes No Don’t Know

Blank

Frequency 52 35 8 5

Percent 52/100= 52%

35/100=35%

8/100=8%

5/100 = 5%

Valid Percent

52/95=55%

35/95 =37%

8/95 = 8%

*

Would you attend an instructional session on Survey Monkey? (N=100)

Frequencies help to "clean" data and check for vulnerabilities

• Outlier responses• Socially desirable

responses• Low or high response rate

to response options • Low response rate for

certain questions

Charts show pictures of frequency distributions

Bar charts display comparisons across categories

Would you attend an instructional session on Survey Monkey?

Bar charts can compare subgroups

Do you need easy-to-read patient informationIn the following languages?

English Spanish Other0%

20%

40%

60%

80%

100%

Clinic 1Clinic 2Clinic 3

Line charts are particularly useful for showing change over time

Line charts also can show changes over time for multiple factors

0

50

100

150

200

Winter Spring Summer Fall

People Trained in Year 1

Consumers

Health professionals

Pie charts show sub-group breakdown of a larger group

(demographics)

In which type of library do you work? (check one)

23%

20%

16%

17%

22%Hospital

HSC

Public

Academic

Other

Visual comparisons in pie charts are less obvious than in bar charts

In which type of library do you work? (check one)

Hospital

HSC

Public

Academic

Other Hospital HSC Public Academic Other0%

5%

10%

15%

20%

25%

30%

Hospital

HSC

Public

Academic

Other

Measures of central tendency are the "most representative" number in a data

set

• Mean (average)• Median (middle)• Mode (most frequent)

The mean (average) is the sum of all scores divided by total number of

responses

Years Employed at Library X

Person 1 8

Person 2 12

Person 3 11

Person 4 9

Mean (average) years of service to Library X

10

Computation: 8 + 12 + 11 + 9 = 40 40 years total/4 people = 10

The median (middle) is the number that divides a distribution equally

Distribution 1

Distribution 2

Distribution 3

4 5 4

5 6 4

5 7 5

5 7 6

6 8 8

7 24 10

7

5 7 5.5 =(5+6)/2Median

The mode (most frequent) can be found on any frequency table

In what state do you work? (N=200)

DE NJ NY PA Other

Frequency 24 28 24 54 70

Percent 12% 14% 12% 27% 35%

Handy Excel tip: Calculate measures of central tendency in a column or a row

• Mean (average)=AVERAGE(A2:A7)

• Median (middle) =MEDIAN(A2:A7)

• Mode (most frequent)=MODE(A2:A7)

The type (level) of data determines how you summarize it

Nominal:(Label)

Ordinal:(Ranking)

Interval/Ratio:(Equal intervals)

Nominal Data (Labels)Nominal Data:

Labels

"Type of institution" is an example of nominal data (labels)

Hospital Clinic Library Other

# Participants 18 42 19 5

% 21% 50% 23% 6%

Mode

Bar graphs usually provide the best visual representation of nominal data

Hospital Clinic Library Other 0%

10%

20%

30%

40%

50%

60%

21%

50%

23%

6%

Participants’ Work Settings

These statistics and charts are most typically used for

nominal data analysis

Nominal dataFrequenciesPercentagesModeBar charts

Ordinal Data: Rankings(1=Top Rank)

Rank the following features for their influence over your decision to attending [Your Profession’s Important Conference], with “1” a the most influential and “5” as the least influential.

__ Registration cost__ Location__ Hotel accommodations__ CE offerings__ Networking opportunities

Rankings are “relative” to the list of items to be ranked

Rank the following features for their influence over your decision to attend [your profession’s national conference]

Resp’t1

rank

Resp’t2

rank

Resp’t3

rank

Resp’t4

rank

Resp’t5

rank

MedianRank

Registration fee 2 5 1 5 4 4

Location 1 3 2 1 2 2

Hotel 5 1 4 4 5 4

CE offerings 3 2 5 3 3 3

Networking 4 4 3 2 1 3

Median rankings summarize ranked data

Rank the following features for their influence over your decision to attend [your profession’s national conference]

Resp’t1

Resp’t2

Resp’t3

Resp’t4

Resp’t5

% with“Top 3”

Registration fee 2 5 1 5 4 40%

Location 1 3 2 1 2 100%

Hotel 5 1 4 4 5 20%

CE offerings 3 2 5 3 3 80%

Networking 4 4 3 2 1 60%

*Percentages for demonstration purposes only. Do not report percentages on groups of less than 50.

You can also report percentage of “Top 3”

“Top Rankings” percentages can be charted

Registration fee

Location Hotel CE Offerings Networking0%

20%

40%

60%

80%

100%

120%

40%

100%

20%

80%

60%

Percentage of Respondents Giving “Top 3” Rank for Each Item

Report the following statistics for ordinal data (rankings)

Median ranks per item Percent of "Top 3" Tables of items in

descending order of importance

Interval/Ratio Data: Equal Intervals

Interval/ratio data (equal distances) allow comparisons and computation

Health Fair Location Visitors

Community Center 300

High School 150

YWCA 150

Mean = 600/3 = 200

You can chart interval-level data

Community Center High School YWCA 0

50

100

150

200

250

300

350

300

150 150

Visitors to Health Fair LocationsF

req

ue

nc

ies

Number of visits to a web site is a typical example of interval data

(equal distances)

Visits to NN/LM WebsiteMonth # VisitsAugust 2008 140,027July 2008 167,445June 2008 180,605May 2008 241,052April 2008 203,084March 2008 203,390

Mean = 189,267

Line charts are appropriate for monthly statistics.

Mar-08 Apr-08 May-08 Jun-08 Jul-08 Aug-080

50,000

100,000

150,000

200,000

250,000

300,000

# Visits to the NN/LM Web Site by Month

# Visits

Some things to think about with interval data

Rate the following features for their influence over your decision to attend [your profession’s national conference]

No Influence

MuchInfluence

Registration fee 1 2 3 4 5

Location 1 2 3 4 5

Hotel 1 2 3 4 5

CE offerings 1 2 3 4 5

Networking 1 2 3 4 5

How do ratings compare to rankings?

Which is the most representative: mean or median?

How would you rate this instructor? (N=20)

High=4 3 2 Low=1

Fr. Count 11 7 1 1

Mean = 3.4

Median = 4

Web Site Visits

Month # Visits

August 2008 275

July 2008 167,445

June 2008 180,605

May 2008 241,052

April 2008 203,084

Mean 158,492Median 180,605

When in doubt, use median or report mean and median

"Outliers" affect means but not medians

Report the following statistics to describe interval data (equal distances)

Frequencies and percentages

Mean or medianBar charts of ranges

ORLine charts for means

at different points in time

Qualitative data fills in the details

ThemesDescriptive details Examples and quotes

Qualitative analysis is beyond the scope of this webinar

It's like déjà vu all over again-- Yogi Berra

Review:Evaluation is a logical process

1. Establish criteria

3. Describe what happened

4. Compare with standards

2. Construct standards

5. Make judgment

6. Make recommendations

Handy reporting tip: Stephanie Evergreen

http://stephanieevergreen.com

Handy Excel tip: Free tutorials

http://annkemery.com/excel

For More Information:

Planning and Evaluating Health Information Outreach – Series of 3 booklets:

http://nnlm.gov/evaluation/guides.html#A2

1. Getting Started with Community-Based Outreach

2. Including Evaluation in Outreach Project Planning

3. Collecting and Analyzing Evaluation Data

Contact Information

Nikki Dettmar

Evaluation Librarian

snydern@uw.edu

206- 543-3409

Cindy Olney

Acting Assistant Director

olneyc@uw.edu

678-682-3864

NN/LM OUTREACH EVALUATION

RESOURCE CENTER

OERC website:http://nnlm.gov/evaluation/

Booklets website: http://nnlm.gov/evaluation/guides.html

OERC tools:http://guides.nnlm.gov/oerc/tools