www.andco.uk.com oliver mantell myths and magic tricks

www.andco.uk.com

Oliver MantellMyths and Magic Tricks

What I’m Going to Talk About

The two basic myths about data:Myth 1: Statistics are irrelevantMyth 2: Statistics are hostile

Types of magic tricks:What you can find out with no data at allGetting complexity from simple dataGetting simplicity from complex data

www.andco.uk.com

Warm-Up and Stretches

What are the odds?

If your coin is fair, that:

It turns up heads on the next toss?

It turns up heads more often than tails?

It turns up tails, if it turned up heads last time?

What are the odds?

If your audience are 50:50 male to female, that:

The next random person you survey is male?

You survey more women than men?

The next randomly selected visitor is male, if the last was female?

What are the odds?

If by knowing that the sample is 50:50 male to female, you can work out the odds of all 10 people you survey being male...

...you can also work out the probability that the sample is 50:50 male to female, if all 10 people you survey are male.

You don’t need to be able to work out the exact answer,just to know that it IS possible to work out.

www.andco.uk.com

Myths

MYTH 3: Randomness is difficult

Randomness makes things easy:its when they’re not random it gets difficult.

MYTH 4: My Survey is Sampling My VisitorsSampling works on selecting items from a population that are equally likely, so that they can represent the whole.

So you’re probably sampling visits, not visitors.

VS.

MYTH 5: 51% is bigger than 50%

Just because your result is bigger than before, doesn’t mean that that’s what is happening in reality. You have to look at the margin of error.

If it’s different, but not significant, it isn’t different.

Most newspaper stories about changes in opinion polls aren’t true.

MYTH 6: Significant changes matter

There’s a difference between a change being real and it being big.

You should only care about substantial AND significant changes.

MYTH 7: ‘Significant’ changes are significantWe only said we wanted to be 95% sure.

If you look at enough examples, one in twenty won’t be significant, although it will look it.

LIAR?

% % % % % % % % % %% % % % % % % % % %

MYTH 8: A Bigger Sample is Always Better

The fewer people you ask randomly, the less chance of a significant result.

But asking more, non-randomly, doesn’t help.

MYTH 9: Changes to outliers show changes to the odds

If you praise people who do well, they do worse.If you shout at people who do badly, they do better.

That doesn’t mean that praise is bad or shouting is good.

MYTH 10: 67% of Our Focus Group Liked It

There’s a reason qual and quant are kept separate.

What would it take for that result to be totally different?

Yes! No!

Erm...?

www.andco.uk.com

Magic tricks

Say It Ain’t So, Joe!

‘Null hypotheses’ are a very powerful trick.

‘A must be true, because B happens’.

Assume A isn’t true: does B still happen?

Welcome to Monte Carlo!

It’s ok to completely make up the numbers (sometimes).

Slice and Dice

What if: 50% of all visitors go to the cafe.and 50% of all visitors are vegetarian.

A Veggie

Not Veggie B Veggie Not

Veggie

Cafe 50% 0% Cafe 25% 25%

Not Cafe 0% 50% Not Cafe 25% 25%

OR

Judge by results: segmentation by response

Response-based – based on likelihood of an answer.

Uses same significance tests mentioned earlier.

Automatic, shows you what has the biggest effect (and is significant).

Judge by results: segmentation by response

Example:

Judge by results: segmentation by response

Example:

Judge by results: segmentation by response

Birds of a feather: segmentation by cluster

Based on best grouping of clusters.

Birds of a feather: segmentation by cluster

Shows ‘natural groups’ within your audience

Gets beyond using single categories to describe visitors

Identifies similarities and differences between individuals based on patterns in whole audience.

Birds of a feather: segmentation by cluster

Example:

Used attitudinal and behavioural only

Showed real differences that made sense

There were real demographic differences between the groups.

Birds of a feather: segmentation by cluster A B C D E F

Main reason Social / Passing

Social / Passing

Park / Work-shop

Park / Work-shop

Art / Kids / Day Out

Art / Kids / Day Out

Local / / Very Quite / /

Repeat / Q Low High Low / /

Satisfied? Quite OK Quite Not Very

Very Very

Dwell Time Long Q Short Medium Short Medium Long

Gender 2/3 F 2/3 F 1/2 M All F 2/3 F 2/3 F

Ethnicity Asian Mixed Asian White White White

Age Young Young / Middle Older Older

Group Type Adult Adult Solo Family Family Family

Etc...

% respondents 12% 14% 6% 2% 37% 30%

Summary

Statistics matter, but they have to be used with care.

Used well, however, that can provide real insight that help you to make decisions and do things better.

Just To Remind You...

Myth 1: Statistics are irrelevantMyth 2: Statistics are hostileMyth 3: Randomness is difficultMyth 4: My survey is sampling my visitorsMyth 5: 51% is bigger than 50%Myth 6: Significant changes matterMyth 7: ‘Significant’ changes are significantMyth 8: A bigger sample is always betterMyth 9: Changes to outliers show changes to the oddsMyth 10: 67% of our focus group liked it.

Just To Remind You...

Trick 1: Say it ain’t so! – Null hypothesesMyth 2: Welcome to Monte Carlo! – Using simulationsMyth 3: Slice and dice – Cross-tabulationMyth 4: Judge by results – segmentation by responseMyth 5: Birds of a feather –segmentation by clusters

www.andco.uk.com

Thank you.

Any questions?

46 The Calls

Leeds officeContact us

LS2 7EY

Telephone: 0113 234 6857Email: info@andco.uk.com

www.andco.uk.com