research 101 - memberclicks

Dave Ward Puget Sound Partnership

RESEARCH 101 The Basics

Agenda

• Purposes of research

• Research methods

• Cost

• Process

• Lessons learned

• Reading and interpreting research

Purposes of Research

PERCEPTION RESEARCH to understand public perception of issues

FORMATIVE RESEARCH to inform program development

EVALUATIVE RESEARCH to measure program effectiveness

BENCHMARK RESEARCH to measure long term change


• What do people think?

• What do people know?

• What do people think they know?

• What are people doing?

• What do people think?

– If a biosolids soil mix was available in bags, how likely would you be to buy and use it?

– How important is it to provide financial incentives to property owners for shoreline protection?


23%

61%

• What do people know?

– For each of the following, please tell me whether you think King County provides that service (water quality, stormwater, groundwater)?

– Is your home served by a septic system?


• What do people think they know?

– On a scale of 1 to 5, how at risk do you think salmon populations in our region are?

– Of the following list of problems facing the environment, which would you say are the most serious?


1 in 7 people

47% answered correctly on average 30% answered incorrectly on average 22% don’t know on average

• What are people doing?

– Do you fertilize your lawn?

– How many times has your septic tank been pumped in the past ten years

– How frequently do you pick up your dog’s waste in your yard?



• What is the scope and scale of the problem?

• What is the appropriate BMP?

• How can we motivate people to implement the BMP?

• What are the barriers and motivators to implement the BMP?

• What is the best message?

• What is the best communication method?


1. Review existing programs

2. Review academic research

3. Inventory relevant businesses

4. Inventory relevant products

5. Quantify the scope and scale of the problem

6. Conduct focus groups

7. Conduct a telephone survey

8. Analyze relevant demographic data

9. Identify problem areas using GIS and water quality data


• Did the program do what we expected?

• Did people implement BMPs as a result of the program?

• Did people change their behavior/practices as a result of the program?


• X% of people started picking up pet waste

• Y% of homeowners replaced their lawns with trees

• Y% of people stopped washing their cars on pavement

• Z% of homeowners installed a rain garden


• What are the long term trends?

• Did people implement BMPs?

• Did people change their behavior/practices?

• Are we on an upward or downward trend?

• Did water get cleaner?

SOUND BEHAVIOR INDEX


• X% decrease in pesticide use since 1990

• Y% increase in aluminum recycling over the past ten years

• Z% decrease in coliform bacteria level since 2001

RESEARCH METHODS

Research Techniques (Conventional Methods)

• Telephone surveys

• Focus groups

• Interactive polling sessions

• Informant interviews

• Intercept surveys

• Mail surveys / questionnaires

• Internet surveys

Research Techniques (Formative Methods)

All Conventional Methods plus: • Staff / audience communication • Consumer research data • Demographic research data • Key informant interviews / ethnographic assessments • Inventory/analysis of existing programs • Review existing research • Message testing (neighbors, friends, focus groups,

surveys) • Evaluate BMPs – life cycle cost, targeting,

effectiveness

Research Techniques (Program Measures)

All Conventional Methods plus: • Staff / audience communication • Audience observation • Windshield surveys • Program/event attendance • Program/event follow-up contact • Reply cards • Promotional item distribution (quantity/location) • Responses to invitations/solicitations • Calls for technical assistance • Return on investment

Telephone Surveys

Quantitative information

Statistically valid

Results can be extrapolated

Large numbers of respondents

Tightly scripted

Questionnaire

Fixed questions

Costly ($00,000)

Asks “How many?”

Objective (in method only)

Focus Groups

Qualitative information

Not statistically valid

Results cannot be extrapolated

Low numbers of respondents

Loosely scripted

Discussion guide

Can change on-the-fly

Not as costly ($0,000)

Asks “Why?”

Subjective

Telephone Surveys and Focus Groups

Cou

rt D

ecis

ion

Source: The Gallup Poll

Surveys and Focus Groups are a Snapshot in Time

1. Determine goals You

2. Select a consultant You

3. Identify audience You and consultant

4. Develop screener You and consultant

5. Develop discussion guide You and consultant

6. Select participants Consultant

7. Arrange for and set-up room Consultant

8. Order and pay for food Consultant

9. Facilitate group sessions Consultant

10. Observe sessions You

11. Record sessions – video/audio Consultant

12. Pay participants Consultant

13. Produce report Consultant

14. Interpret results You and consultant

Focus Group Process

• Screener • Discussion guide • Transcript and/or digital video • Sound recording • Report/analysis • Participant list

Focus Group Work Product

• About $7,000 - $9,000 per focus group

Cost

• Recruiting from small audiences

• Recruiting from unusual audiences

• Recruiting from a list (sometimes)

• Looking up phone numbers

What adds cost?

1. Determine goals You

2. Select a consultant You

3. Identify audience You and consultant

4. Develop questionnaire You and consultant

5. Pretest questionnaire You and consultant

6. Field questionnaire (make calls) Consultant

7. Produce topline results Consultant

8. Produce cross-tabulated data Consultant

9. Conduct statistical analysis Consultant

10. Produce report Consultant

11. Interpret results You and consultant

Telephone Survey Process

• Questionnaire • Topline results • Cross-tabulated results • Report/analysis

Telephone Survey Work Product

• 15-20 minute survey of 400: About $20,000 - 40,000

Cost

• Polling small audiences

• Polling unusual audiences

• Polling from a list (sometimes)

• Looking up phone numbers

What adds cost?

1. Meet the focus group facilitator before you select the contractor

2. First decide who you want to hear from; then let the consultant develop the screener

3. First decide what you want to know; then let the consultant develop the questionnaire/discussion guide

4. Do everything you can to ensure the contractor understands what you want to know

5. Stay close to the contractor

Lessons Learned

Lessons Learned 6. Wordsmith

7. Pay close attention to sequence, flow, and how information is revealed

8. Meet with the facilitator immediately before every focus group

9. Schedule focus groups in pairs using the same discussion guide

10. Don’t allow the consultant to start calling without your approval of the questionnaire / screener

11. Don’t allow the focus group to begin without your approval of the discussion guide

READING & INTERPRETING

RESEARCH

Three things to question about ALL research

• Method Understand the methodology section

• Data Is the data sound?

• Interpretation Do the data support the conclusions?

Three things to remember about ALL research

• Nothing is isolated Everything is connected

• Every study reflects a moment in time

• One data point is not a trend

Do you approve or disapprove of the job John Boehner is doing?

Approve 33% Disapprove 37% Not sure 30%

Public Policy Polling, July 15-17 2011, N=928

If God exists, do you approve or disapprove of its performance?

Approve 52% Disapprove 9% Not sure 40%

Public Policy Polling, July 15-17 2011, N=928

Pay Attention to the Margin of Error!

Approximate Sample Size Margin of Error 200 6.9% 300 5.7% 400 4.9% 800 4.0% 1,300 2.7% Split samples increase the margin of error!

Would you support rebuilding salmon runs if it meant an increase in your taxes?

Would you support a tax increase to rebuild salmon runs for future generations?

Do you support rebuilding salmon runs?

Many projects that benefit salmon also reduce erosion, water pollution, and flooding of downstream properties. Knowing that, would you support rebuilding salmon runs if it meant an increase in your local taxes?

Question Phrasing

3. Do you support or oppose laws that prohibit killing unborn babies?

2. Do you support or oppose laws that prohibit abortions?

1. Do you support or oppose laws that protect abortion rights?

Question Bias

Triangulation

Question 10

Question 24 Question 3

The real answer

HAVE FUN

It’s an exploration!

research 101 - memberclicks

Documents