intro to rapid experimentation - intel july 2015

Handicapping & Dark Horses

What Design Thinking Can Learn from Horse Racing

2 Techniques

#1 Handicapping

handicapping:the art of evaluating a horse’s performance in order to determine the outcome of a race

handicap the ideas

running a rapid experiment

rapid experimentation:using prototypes to evoke behaviors that quickly tell us whether our ideas meet users’ needs

rapid

experiments

usability test

survey

feedback methods

what will someoneactually do

what they might do

rapid experiments separate what customers say from what they do in the real world

what is an experiment?• a scientific procedure—a controlled empirical test of a

hypothesis

• hypotheses include (never null!):-A causes B-A is better, bigger, faster than A’-A changes B more when we do/provide X

• requirements:– independent variable that can be manipulated– dependent variable that can be measured– random assignment to condition (conservatively)

+

What can design experiments test?• are people interested?

• will they do/use this given all other choices/demands available?

• does it meet the need you designed for?• does it have the desired (and not

undesired) effects?• can people figure out how to use/do it?• which design is better (at any of the

above)?

How do you do this at home?1. Pick an idea 2. Make a list of all the questions you have

about the efficacy of the idea3. Select the most critical question to

success4. Generate a hypothesis5. Design an experiment to test your

hypothesis6. Create the prototype to support the

experiment

Before you move into prototyping, make a plan:• What questions do you have about the

effectiveness of the ideas?• What is my hypothesis about what will happen?• What kinds of observations would validate my

hypothesis?• What prototype(s) do I need to create for these

observations?• How should I run this experiment?

Use the prototype to• Create an experience that evokes the desired

behaviors• Evaluate your working hypotheses

Don’t create the prototype before designing your experiment

Make a list of questions

Concepts we generate often have implicit assumptions about how people will respond and what people will doWhat are the crucial questions that could make or break the success of your idea?

Select the most critical question

Will people be more likely to select a rental if it has a nice photo?Will homeowners be willing to let photographers photograph their space?How expensive is it create this service?Will the photos really look better?

Generate a hypothesisA hypothesis is your prediction about how people will behave.

A lot more people will rent an apartment that has been professionally photographed than one that has not.

Design an experiment• the goal is to evoke “real”

behaviors in “real” situations • the scenario must be

– believable– immersive– natural

• and, allow you to test/measure what you need to

• a word on “confederates”

Plan for measurement

people are notoriously bad at predicting (and also remembering/reporting) their own behaviors

Plan for analysis• before conducting your experiment, figure out

how you’re going to analyze your data• what conclusions will you be able to draw from the

data?

normal photo

pro photo

4 5Did theyrent theapartment?

Design your experiment– A/B Test – Show ½ the people the a rental with

normal photos, ½ the same rental with professional photos.

– Constrain to one geographic area– Select 2 popular and 2 unpopular

apartments.–Measurement: What percentage of people

who view the apartment rent it in condition A vs. B.

Create the prototype

run the experiment&

analyze your results

Tip: You can follow up with qualitative interviewing

#1 Handicapping

Dark Horse#2 Dark Horses

dark horse:a contestant that seems unlikelyto succeed, but emerges to prominence against all odds

dark horse idea1. dark: explores a space that is risky,

difficult, radical, or orthogonal

2. brainstormed after more traditional ideas

3. refined enough that can be prototyped and tested with a rapid experiment (can’t be infeasible)

rational1. Designers need to preserve ambiguity to

leave themselves open to new ideas

2. Taking the dark horse idea to the point you can evaluate it keeps the design space from shrinking too fast

3. Even if the dark horse doesn’t work, you will gain insights that will make your final design better

Example 1

Design the next generation cinema/broadcastingimaging system

Original idea: Immersive movie watching experience where viewer’s head and eye position affect the perspective and focus of a sceneDark Horse: What if the camera was everywhere?Rigs of multiple Kinects and software capture 3D info of a scene for post capture scene manipulation.

Example 2

Design a system to improve street navigation forpeople on vacation

Original idea: Hotter/ColderPhone vibrates differently as you are getting closer or farther from the direction you should be going.Dark Horse: Stalker NavigationNo maps, crowdsourced directions, follow the stranger.

Dark Horse#2 Dark Horses

Handicapping & Dark Horses

What Design Thinking Can Learn from Horse Racing