product madness - a/b testing
DESCRIPTION
Andy Toben from Product Madness' presentation on A/B Testing, from GIAF.TRANSCRIPT
![Page 2: Product Madness - A/B Testing](https://reader036.vdocument.in/reader036/viewer/2022062319/557cbd9bd8b42a1b0c8b5027/html5/thumbnails/2.jpg)
What is an A/B test ?“A/B testing is a methodology in advertising of using randomized experiments with two variants, A and B, which are the control and treatment in the controlled experiment.”
- wikipedia.org
![Page 3: Product Madness - A/B Testing](https://reader036.vdocument.in/reader036/viewer/2022062319/557cbd9bd8b42a1b0c8b5027/html5/thumbnails/3.jpg)
What Can We A/B Test?Basically anything:• Page Colour • Layout• Call to action • Images• If you can change it, you can test it!
![Page 4: Product Madness - A/B Testing](https://reader036.vdocument.in/reader036/viewer/2022062319/557cbd9bd8b42a1b0c8b5027/html5/thumbnails/4.jpg)
Step by Step OptimizationTesting one variable at the time works, but can create issues:
- Time consuming:Needs a lot of traffic or data
- Local verses Global Optimum:Can lead to accepting the best variant for that test, but it is not optimised against all possible variants
![Page 5: Product Madness - A/B Testing](https://reader036.vdocument.in/reader036/viewer/2022062319/557cbd9bd8b42a1b0c8b5027/html5/thumbnails/5.jpg)
Multi Variant TestingThis allows you to test all combinations at once however:
- Requires massive data sets – much more than Step by Step
- Requires mathematical tools
- No intuitive / building insights
![Page 6: Product Madness - A/B Testing](https://reader036.vdocument.in/reader036/viewer/2022062319/557cbd9bd8b42a1b0c8b5027/html5/thumbnails/6.jpg)
How do we know who’s winning ?
![Page 7: Product Madness - A/B Testing](https://reader036.vdocument.in/reader036/viewer/2022062319/557cbd9bd8b42a1b0c8b5027/html5/thumbnails/7.jpg)
How do we know who’s winning ?
![Page 8: Product Madness - A/B Testing](https://reader036.vdocument.in/reader036/viewer/2022062319/557cbd9bd8b42a1b0c8b5027/html5/thumbnails/8.jpg)
Advertising is a game, so how about games ?
![Page 9: Product Madness - A/B Testing](https://reader036.vdocument.in/reader036/viewer/2022062319/557cbd9bd8b42a1b0c8b5027/html5/thumbnails/9.jpg)
Strategies
What can we use A/B testing to discover?
• Search - Option A / Option B• Optimize - Option A / Option A+ / Option A-• Change management - treatment verses control• Measure - No treatment verses treatment
![Page 10: Product Madness - A/B Testing](https://reader036.vdocument.in/reader036/viewer/2022062319/557cbd9bd8b42a1b0c8b5027/html5/thumbnails/10.jpg)
A/B Testing User ExperienceWhen A/B testing user experience you can test out the following and create a big impact… for better or worse!
- Order of game levels / features- Bonus system- Available content- P2P assignment - Pricing and Economy
![Page 11: Product Madness - A/B Testing](https://reader036.vdocument.in/reader036/viewer/2022062319/557cbd9bd8b42a1b0c8b5027/html5/thumbnails/11.jpg)
Challenges We FaceSome A/B tests are very expensive - You may need to commit development or art resource for something that may not work.
Variant assignment needs to persistent - In some cases you can’t just change a player’s track to the
“winning” branch What to measure:
• Responsiveness (CTR)• Engagement• Retention • Monetization
![Page 12: Product Madness - A/B Testing](https://reader036.vdocument.in/reader036/viewer/2022062319/557cbd9bd8b42a1b0c8b5027/html5/thumbnails/12.jpg)
Branching user experience
![Page 13: Product Madness - A/B Testing](https://reader036.vdocument.in/reader036/viewer/2022062319/557cbd9bd8b42a1b0c8b5027/html5/thumbnails/13.jpg)
Understanding Results
Feature cost 1st day retention 7th day retention User value
10,000 coins 37% 10% 10 USD
25,000 coins 36% 6% 12 USD
Which metric is the best one to look at, and which result should we take action on?
![Page 14: Product Madness - A/B Testing](https://reader036.vdocument.in/reader036/viewer/2022062319/557cbd9bd8b42a1b0c8b5027/html5/thumbnails/14.jpg)
But after all, It’s just a tool
Allocate Variant
No Change
No
Should we test the User?
User
NoIs there an
active Experiment?
Yes
Yes