it worked for ustream
Post on 08-May-2015
203 Views
Preview:
DESCRIPTION
TRANSCRIPT
!
It Worked for UstreamProduct Development @ Ustream
Balázs Kereskényi product manager
API, discovery
Gergely Schmidt product manager,premium products
Making decisions by assumptions:“Let’s do this, because it worked for Spotify!” - you will never know why it worked for them.
Making decisions by experiments:“Let’s try this out, measure effectiveness,and see if it works!
What to test?
Icon & text: +12% conversion.
Myth #13: Icons enhance usability
What to test?
• CTA microcopies: “Get started”, “Try out for free”, “Sign up”, “Learn more” etc.
• Layout: arrangement of form fields, mixing elements on the whole page.
• “Meta parameters” of a product: trial period length, price.
• ...
How to measure effectiveness?
Conversion (basically any kind of your metrics):
• Increase in purchases,
• Engagement,
• Number of clicks,
• Number of signups,
• Form completions,
• ...
Always question assumptions
but be prepared that only 1 out of 8 A/B testswill drive you to significant changes.*
*based on the tests done by appsumo.com
FAMOUS!A/B!
TESTS
Marissa Mayer:41 shades
of blue
How Obama raised $60M by running a simple experiment
Optimizely, Dan Siroker
Who to blame for the slow loading up? Facebook or iOS?
How to set up a test?• Targeting (when the test should start, visitor characteristics &
segmentation, technical circumstances etc.),
• Experiment type (A/B, multivariate, multi page),
• Traffic allocation,
• Goals,
• Integration with other kind of analytics.
Technical background of a test• Frontend
• Quick and dirty methods
• Always having a usable fallback is anything goes wrong
• Backend for bigger and more complicated tests
• Measurement
• GA and Optimizely events are our best friends
Understanding your resultsChance to beat baseline.
…or you can simply use isvalid.org anytime.
USTREAM!CASE!
STUDIES
Changing directions: SaaS vs. content.
Started with concluding tiny preliminary experiments: less content, pro CTAs. Ended up with split testing the whole page.
editorializedpro purchases++
Ustream homepage test results
• Tested on approx. 700k unique visitors.
• No important general metrics harmed (pages / visit time on site & time on homepage).
• 1% increase in purchases starting from the home page.
• Showing 1/3 of content resulted only in halving the clicks.
A DIFFERENT APPROACH FOR VALIDATION:!GHETTO TESTING
http://bit.ly/ghettozynga
Ghetto test: Pay with PayPal
• Built by frontend engineers, only triggered from Optimizely
• Decided not to build it because of low conversion rate (~2%)
Pro Purchase Flow: 1 or 2 steps?
• Built by frontend engineers, only triggered from Optimizely
• Currently live, analysis is on the way
This is not a test
!
https://ustream.tv/platform/unlimited!
Thanks!
@bkereskenyi@sgergely
top related