good hypothesis testing is surprising!
TRANSCRIPT
![Page 1: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/1.jpg)
GOOD HYPOTHESIS
TESTINGIS SURPRISING
Timothy Fitz (.com)
![Page 2: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/2.jpg)
Timothy Fitz
Coined Continuous DeploymentTechnical Lead at IMVU
CTO at CanvasSoftware Consultant
TimothyFitz.comSystemsLive.com
@TimothyFitz
![Page 3: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/3.jpg)
WHICH SHADE OF BLUE?
Buy Buy Buy
Buy Buy Buy
Buy Buy Buy
Buy Buy Buy
![Page 4: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/4.jpg)
MULTI-ARMED BANDIT
![Page 5: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/5.jpg)
GOOD HYPOTHESIS
• Documented
• Testable
• Actionable
• Interesting
• Surprising
![Page 6: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/6.jpg)
DOCUMENTEDNCSU Libraries Mobile Redesign for 2012
Objective The redesign of the NCSU Libraries Mobile website will improve the extensibility of the service and introduce several user experience enhancements to the existing set of mobile library services.
Requirements ● Redesign the homepage interface in a way that enables deployment of new apps with minimal effort ● Migrate mobile dev framework from forked MIT Mobile Web framework to jQuery Mobile framework ● Implement editorial recommendations on app names, taglines, and graphic identity; update the color scheme to be consistent with
the NCSU Libraries website ● Perform a comprehensive review of the usage stats; use this information to remove underutilized services ● Maintain or improve usage data collection; improve usage data reporting tools ● Retire legacy MoboiLIB service
Out of Scope ● Development of brand new apps ● Deep redesign of existing apps ● Custom iPad mobile website ● Native app deployment (iOS/Android)
Team Project Manager: Tito Sierra UI/UX Lead: Adam Rogers Tech Lead: Cory Lown
Schedule Launch: 2012.12.08
![Page 7: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/7.jpg)
DOCUMENTED
Hypothesis Students want to use their phones to track their outstanding library books.
![Page 8: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/8.jpg)
The details really matter.
Tiny code changes can change results dramatically.
![Page 9: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/9.jpg)
TESTABLE
• Are you collecting the data?
• Can you collect and analyze it quickly?
• Is it statistically significant?
![Page 10: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/10.jpg)
HIGH TEMPO TESTING
3 experiments per week!
![Page 11: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/11.jpg)
Hypothesis:
Adding Group Chat (a 12-month effort) will result in a 25% increase in 3-month retention.
![Page 12: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/12.jpg)
Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers about specific topics.Hypothesis: Users want a directory of group chats.
Hypothesis: Users who group chat will have higher retention.
![Page 13: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/13.jpg)
Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers about specific topics.Hypothesis: Users want a directory of group chats.
Hypothesis: Users who group chat will have higher retention.
✓
![Page 14: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/14.jpg)
Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers about specific topics.Hypothesis: Users want a directory of group chats.
Hypothesis: Users who group chat will have higher retention.
✓
![Page 15: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/15.jpg)
Hypothesis: Users want to chat with strangers.
Hypothesis: Users want to chat with strangers about specific topics.Hypothesis: Users want a directory of group chats.
Hypothesis: Users who group chat will have higher retention.
![Page 16: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/16.jpg)
ACTIONABLE
The results of testing your hypothesis cause action.
![Page 17: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/17.jpg)
ACTIONABLE
Hypothesis: Adding a new payment method will increase overall revenue by 5%
• If confirmed, we will keep the payment method
![Page 18: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/18.jpg)
INTERESTING
The action you take is important and
dependent on the outcome.
![Page 19: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/19.jpg)
Hypothesis: Adding a new payment method will increase overall revenue by 5%
• If confirmed, we will keep the payment method
• If denied, we will keep the payment method
INTERESTING+DOCUMENTED
![Page 20: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/20.jpg)
GOOD PRODUCT MANAGERS
Can rationalize ANY action from ANY outcome.
:( :( :(
![Page 21: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/21.jpg)
Hypothesis: Adding a new payment method will increase overall revenue by 2%
• If confirmed, we will keep the payment method
• If denied, we will DELETE the payment method
INTERESTING+DOCUMENTED
![Page 22: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/22.jpg)
SURPRISING
Bayesian definition of surprise:
“The total number of wows experienced when simultaneously considering all models is obtained
through the integration over the model class.”
http://ilab.usc.edu/surprise/
![Page 23: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/23.jpg)
SURPRISING
surprise
=
change in beliefs
![Page 24: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/24.jpg)
SURPRISING
The results of your hypothesis tests
should frequently and significantly
change your beliefs about your product!
![Page 25: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/25.jpg)
Don’t let a single hypothesis test invalidate
your whole vision.
![Page 26: Good hypothesis testing is surprising!](https://reader031.vdocument.in/reader031/viewer/2022022123/589e81c41a28ab443e8b46ed/html5/thumbnails/26.jpg)
GOOD HYPOTHESIS
• Documented
• Testable
• Actionable
• Interesting
• Surprising