www.andco.uk.com oliver mantell myths and magic tricks
TRANSCRIPT
www.andco.uk.com
Oliver MantellMyths and Magic Tricks
What I’m Going to Talk About
The two basic myths about data:Myth 1: Statistics are irrelevantMyth 2: Statistics are hostile
Types of magic tricks:What you can find out with no data at allGetting complexity from simple dataGetting simplicity from complex data
www.andco.uk.com
Warm-Up and Stretches
What are the odds?
If your coin is fair, that:
It turns up heads on the next toss?
It turns up heads more often than tails?
It turns up tails, if it turned up heads last time?
What are the odds?
If your audience are 50:50 male to female, that:
The next random person you survey is male?
You survey more women than men?
The next randomly selected visitor is male, if the last was female?
What are the odds?
If by knowing that the sample is 50:50 male to female, you can work out the odds of all 10 people you survey being male...
...you can also work out the probability that the sample is 50:50 male to female, if all 10 people you survey are male.
You don’t need to be able to work out the exact answer,just to know that it IS possible to work out.
www.andco.uk.com
Myths
MYTH 3: Randomness is difficult
Randomness makes things easy:its when they’re not random it gets difficult.
MYTH 4: My Survey is Sampling My VisitorsSampling works on selecting items from a population that are equally likely, so that they can represent the whole.
So you’re probably sampling visits, not visitors.
VS.
MYTH 5: 51% is bigger than 50%
Just because your result is bigger than before, doesn’t mean that that’s what is happening in reality. You have to look at the margin of error.
If it’s different, but not significant, it isn’t different.
Most newspaper stories about changes in opinion polls aren’t true.
MYTH 6: Significant changes matter
There’s a difference between a change being real and it being big.
You should only care about substantial AND significant changes.
MYTH 7: ‘Significant’ changes are significantWe only said we wanted to be 95% sure.
If you look at enough examples, one in twenty won’t be significant, although it will look it.
LIAR?
% % % % % % % % % %% % % % % % % % % %
MYTH 8: A Bigger Sample is Always Better
The fewer people you ask randomly, the less chance of a significant result.
But asking more, non-randomly, doesn’t help.
MYTH 9: Changes to outliers show changes to the odds
If you praise people who do well, they do worse.If you shout at people who do badly, they do better.
That doesn’t mean that praise is bad or shouting is good.
MYTH 10: 67% of Our Focus Group Liked It
There’s a reason qual and quant are kept separate.
What would it take for that result to be totally different?
Yes! No!
Erm...?
www.andco.uk.com
Magic tricks
Say It Ain’t So, Joe!
‘Null hypotheses’ are a very powerful trick.
‘A must be true, because B happens’.
Assume A isn’t true: does B still happen?
Welcome to Monte Carlo!
It’s ok to completely make up the numbers (sometimes).
Slice and Dice
What if: 50% of all visitors go to the cafe.and 50% of all visitors are vegetarian.
A Veggie
Not Veggie B Veggie Not
Veggie
Cafe 50% 0% Cafe 25% 25%
Not Cafe 0% 50% Not Cafe 25% 25%
OR
Judge by results: segmentation by response
Response-based – based on likelihood of an answer.
Uses same significance tests mentioned earlier.
Automatic, shows you what has the biggest effect (and is significant).
Judge by results: segmentation by response
Example:
Judge by results: segmentation by response
Example:
Judge by results: segmentation by response
Birds of a feather: segmentation by cluster
Based on best grouping of clusters.
Birds of a feather: segmentation by cluster
Shows ‘natural groups’ within your audience
Gets beyond using single categories to describe visitors
Identifies similarities and differences between individuals based on patterns in whole audience.
Birds of a feather: segmentation by cluster
Example:
Used attitudinal and behavioural only
Showed real differences that made sense
There were real demographic differences between the groups.
Birds of a feather: segmentation by cluster A B C D E F
Main reason Social / Passing
Social / Passing
Park / Work-shop
Park / Work-shop
Art / Kids / Day Out
Art / Kids / Day Out
Local / / Very Quite / /
Repeat / Q Low High Low / /
Satisfied? Quite OK Quite Not Very
Very Very
Dwell Time Long Q Short Medium Short Medium Long
Gender 2/3 F 2/3 F 1/2 M All F 2/3 F 2/3 F
Ethnicity Asian Mixed Asian White White White
Age Young Young / Middle Older Older
Group Type Adult Adult Solo Family Family Family
Etc...
% respondents 12% 14% 6% 2% 37% 30%
Summary
Statistics matter, but they have to be used with care.
Used well, however, that can provide real insight that help you to make decisions and do things better.
Just To Remind You...
Myth 1: Statistics are irrelevantMyth 2: Statistics are hostileMyth 3: Randomness is difficultMyth 4: My survey is sampling my visitorsMyth 5: 51% is bigger than 50%Myth 6: Significant changes matterMyth 7: ‘Significant’ changes are significantMyth 8: A bigger sample is always betterMyth 9: Changes to outliers show changes to the oddsMyth 10: 67% of our focus group liked it.
Just To Remind You...
Trick 1: Say it ain’t so! – Null hypothesesMyth 2: Welcome to Monte Carlo! – Using simulationsMyth 3: Slice and dice – Cross-tabulationMyth 4: Judge by results – segmentation by responseMyth 5: Birds of a feather –segmentation by clusters
www.andco.uk.com
Thank you.
Any questions?
46 The Calls
Leeds officeContact us
LS2 7EY
Telephone: 0113 234 6857Email: [email protected]
www.andco.uk.com