When breakthrough innovation comes from where you least expected it.

Linda GrosjeanOrlando Wood

Slaying a Sacred Cow

q15

185 kg105 kg

200 kg 145 kg

What is the combined weight of Linda and Orlando, when dressed up as a cow?

A: B:

C: D:

185 kg105 kg

200 kg 145 kg

What is the combined weight of Linda and Orlando, when dressed up as a cow?

A: B:

C: D:

q15d

185 kg105 kg

200 kg 145 kg

What is the combined weight of Linda and Orlando, when dressed up as a cow?

A: B:

C: D:

a15RIGHT

£1 MILLION!

1197lb

1198lb

The Wisdom of Crowds

How can a crowd be as accurate as or even more accurate than an expert?

We are good at anticipating others’ behaviour, but the truth is we are self-deceit machines

No – not this one!

Using the Wisdom of Crowds to Slay the Sacred Cow

BrainJuicer’s Predictive Markets Technique

• A crowd of 500 respondents (invited on a nationally representative basis)

• Crowd shown up to 15 ideas and asked to judge which is most likely to work in market

• Asked as a share buying/selling game

BrainJuicer’s Predictive Markets Technique

0

10

20

30

40

50

60

-40 -30 -20 -10 0 10 20 30 400

10

20

30

40

50

60

-30 -20 -10 0 10 20 30

Correlation 0.92

Savoury Snacks Parallel Test

Household Cleaners Parallel Test

Correlation 0.79

Predictive Market Result (Net Preference)

Traditional Monadic Result

(Purchase Intention)

Concepts green-lighted in both traditional and Predictive Markets techniques were the same

Predictive Market Picks out the Same Winners as Traditional Testing

Predictive Market Result (Net Preference)

Picking out the Same Winners as Traditional Testing

Monadic Ideas Tested No. of Green Lighted Concepts

Savoury snacks 15 5 out of 5Household cleaners 10 5 out of 5Average 10 out of 10 Sequential-Monadic No. of Green Lighted Concepts

Telecoms UK 14 4 out of 5Telecoms France 14 2 out of 3Telecoms Spain 14 2 out of 3Chilled Food 11 3 out of 3Window cleaners 17 3 out of 4Chewing gum 7 4 out of 4Lighting ideas 20 9 out of 10Average 8.5 out of 10

Greater discrimination vs. traditional tests

-9 -7 -10-3

-14

-26 -30

31

19 177 7

136

22

12 7

4-24

-13

-7

-35

-25

-15

-5

5

15

25

35

% o

f res

pond

ents

dou

blin

g (p

ositi

ve) v

s se

lling

(neg

ativ

e) s

hare

s

0

10

20

30

40

50

60

70

80

90

100

Sequ

entia

l Mon

adic

% D

ef/P

rob

Scores from traditional monadic test can give a flat and unhelpful result

Predictive Market is more discriminating, separating the wheat from the chaff

The Crowd In Action – Linda & John!

Validation of Technique in Pharmaceutical Research

• A series of experiments have been undertaken between Roche and BrainJuicer. For two of these parallel monadic data is available from traditional monadic research among health professionals (a targeted sample)

– Ad concept testing for MabThera RA in a European country

– Actemra Ad concept testing in the US

• The Actemra Ad concept testing was also conducted among Roche employees (an Internal Roche ‘crowd’)

Experiment 1

Traditional480 rheumatologists in 6 countries

Face to face; followed by phone

CHF 400,000

12 weeks

Roche proprietary questions

Wisdom of the Crowd500 ordinary people in one European market

Online interview

£12,000 (25’000 CHF)

2 weeks

Standard questions

Interview flow

Traditional• Assessment of awareness/

perception of current biologics

• Glimpse evaluation

• Detailed evaluation

• Fit of concept with MabThera personality

• Comparative evaluation of each concept

• Ranking of concepts vs. positioning statement

• Link with then existing campaign

Wisdom of the Crowd • Would you probably buy or probably sell

shares in each of these ideas?

• If you owned shares in these ideas, which would you.. Double .. Definitely Sell shares in?

• Which of these faces (next slide) best expresses how doctors would feel about this ad?

• Plus other diagnostic questions

The crowd delivered a very similar result to the targeted sample of rheumatologists

Most encouragingLeast encouraging Encourage to find out moreLeast likely

Most likely ‘Passport

’

‘Boots’

‘Postcard’

‘Switch’

‘New route’

‘Making History’

Net Preference Score=Percentage doubling shares minus percentage

selling shares

‘Switch’

‘Making History’

‘New route’

‘Boots’

‘Passport’

‘Postcard’

= Same Result

Traditional Wisdom of the Crowd

Experiment 2

Traditional202 US rheumatologists

On line interview

$120’000 (122’000 CHF)

10 weeks

Roche proprietary questions

Wisdom of the Crowd500 ordinary US people +

200 Roche Employees

On line interview

£12,000 (25’000 CHF)

Took 2 weeks

Standard questions

Cement Creative Tree Creative

Experiment 2

Same results – with clear winner in Cement advertising concept

Predictive MarketNet Preference Measure

Traditional ResearchPreference Measure

82%

18%

87%

13%

Internal Crowd also returned the same overall result – Cement route is clear winner

Predictive Market External Crowd

Predictive MarketInternal Crowd

82%

18%

93%

7%

January 09 FuturePast

30 - 50 K

Average200 - 500 K

CHFCost of ad concept

marketresearch historically

90% cost reduction feasibleif parallel pilots successful

-> Crowd method as possible new standard

Cost Savings of Predictive Market Approach

3 Pharmaceutical Research Innovations in One

• Reduction of the enormous costs and time associated with ad and concept screening in pharmaceutical research

• Avoids overuse of expensive and rare doctor samples

• Ability to research sensitive subjects that should not go beyond the company walls

What next for Roche and Predictive Markets?

• Continue validation of different types of “emotional research” with non-traditional respondents, including

– Name development– Name testing– Ad concept testing– Brand icon/image testing– Consumer message development and testing