practitioner web analytics presentation may 2010
DESCRIPTION
Presentation to PWA 2010 Madrid 2010-26-05TRANSCRIPT
![Page 1: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/1.jpg)
I’m Jim Williams,
Head of Customer
Analytics
Split Testing to Optimize Digital Sales
www.weeworld.comwww.ju2.comwww.twitter.com/ju2ltd
![Page 2: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/2.jpg)
Weeworld
• Weeworld is a social networking site/
virtual world
• WeeWorld is the home of the
WeeMee
• The world's favourite avatar
• 33M weemees and growing
•Users Express themselves visually by
dressing up the weemee with latest
fashions and accessories
Hi… I’m a WeeMee
![Page 3: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/3.jpg)
Building out community
I’m so popular, I have 10,000 friends!
![Page 4: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/4.jpg)
Rooms: extension of your WeeMee
A new room is created about like every 10 seconds.
******
![Page 5: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/5.jpg)
World: Synchronous interaction
• Doesn’t have to be 3D, that’s not important as far as building a Virtual World is concerned
Hey you got a boyfriend?
Maybe yes, maybe no.
![Page 6: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/6.jpg)
Getting Gold Points to Spend on Site
•Subscriptions•PayPal•Prepaid Cards•PSMS Mobile Payments•Surveys•Offers•Pay by Cash•Launch Credit Cards
•Original Get Points Page too Complicated•Get Points Page Redesign
![Page 7: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/7.jpg)
New Tabbed Design
Credit Card Tab Surveys Tab
Which Page Design gave the highest Revenue per User?
![Page 8: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/8.jpg)
Which Design is Best?
Wanted to answer 2 questions:
• Is the tabbed design better than the original design
• Which tab should be default
Average Revenue per User
Tab or not to Tab that is the question?
![Page 9: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/9.jpg)
Objective: To Identify whether new tabbed design more effective than original.
Challenges:
•All transactions completed on third party sites•Could not guarantee to capture conversion•Unable to use Google Website Optimizer•Multiple micro transactions•User experience must be consistent
Whys life so complicated?
![Page 10: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/10.jpg)
Original Get Points Page
Allocate page versions according to userID
MOD5- 0 Original 20%- 1 or 2 Credit Card 40%- 3 or 4 Surveys 40%
Track average revenue per user ARPU
Statistics 95% confidence IntervalsNo significant differences between groups prior to tests
![Page 11: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/11.jpg)
New Tabbed Design
Credit Card Tab Surveys Tab
Which Page Design gave the highest Revenue per User?
![Page 12: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/12.jpg)
Improvement in Revenue per Login
Version Response(improvement in
revenue per login)
p<0.05
Original - -
Credit Card 7.7% yes
Surveys 7.1% yes
Tabbed design 7% more effective than original one page design no difference between tabs
![Page 13: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/13.jpg)
Difficulties – Thing about your control
• Don’t be impatient• Change only one thing at a
time• Original Page had no credit
card option• Was result due to new design
or just credit card?• Another follow up experiment
showed that the design was better than Original -Credit Card
Change only one thing?
![Page 14: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/14.jpg)
Difficulties – Don’t breakout the champagne too early!
• Cyclical Effects - don’t breakout the champagne too early.
• Improvement in ARPU much higher early in experiment
• Maintaining Interest 28 days to see result.
![Page 15: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/15.jpg)
Difficulties – Avoid Bias
• Site constantly changing• Marketing initiatives• Store Promotions• Avoiding bias• Live in real world
That’s biased it?
![Page 16: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/16.jpg)
Difficulties -
• PayPal Revenue dropped• Cannibalization by Credit Card• Average Transaction Value fell
28%• Changed price points• Reversing option order removed
fall in average transaction value
![Page 17: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/17.jpg)
Lessons Learned
• Using userID for split testing valid – continue to use• Tabbed Design increased revenue per user compared to
original design even with credit card. • Give careful consideration to controls• Change only one thing at a time.• Try to avoid bias with other changes on the site• Average transaction values very sensitive to price points• Maintain stakeholder interest throughout experiment
![Page 18: Practitioner Web Analytics Presentation May 2010](https://reader036.vdocument.in/reader036/viewer/2022081602/555c38bed8b42a0b038b488c/html5/thumbnails/18.jpg)
Questions?