(mobile app) Instrumentation & Experimentation FTWDavid TesarSenior Technical Evangelist, Microsoft@dtzar
Representing:OneDrive app TeamOutlook app iOS TeamNext Lock Screen Team
OneDrive mobile – circa 2013
Feedback without the information to act
Instrumentation & Experimentation
1. Direct App Feedback2. Automated Builds + Instrumentation3. Scaling Feedback Channels4. Telemetry5. A/B Testing
FTW?
Direct App Feedback (OneDrive)Parature Console
Direct App Feedback (Next)
1.Friday 1 am – Deployment to Play Store2.Friday 1:04 am – 1 email for feedback3.<sleep>4.Friday 9 am – 200 emails5.Saturday – 2,000 emails
Direct App Feedback (Next)
Direct App Feedback (Outlook)
Powered by Helpshift
Direct App Feedback – FTW
Results• Higher ratings!• Better prioritization of features• Solve some bugs where there are no
crashes• Actionable New bugs and features
discovered Lessons• Have a way to organize feedback at start• Centralized feedback across platforms
useful
Automated builds to phones + instrumentation
Power of Instrumentation (Outlook)
Power of Instrumentation (Outlook)
Automated builds to phones + instrumentation - FTW
Results• Found many more bugs in dev• Saved hours every week getting builds to
phones• Cross team feature testing and feedbackLessons• Be super responsive on daily builds• Have as many people possible using these
builds• Balance frequency of updates
Scaling Feedback (OneDrive)
Feature Flags Production
Test Flight
Hockey App
Dev Machine
Scaling Feedback (Next)
Radek – screenshot of Google+ Alpha->Beta channels and/or (ideally) Azure Mobile Engagement instrumentation with being able to slice/dice crash data info
Scaling Feedback (Next)
Capptain Azure Mobile Engagement
Scaling Feedback - FTW
Results• Faster deployments/agility• Increase in quality, reduced bugs in
productionLessons• Get all crash reports• Capture log information in the crash report• Don’t release builds on Friday• Develop a beta community
Telemetry (Outlook)
Internal MSFT tool
Telemetry (OneDrive)
App Insights / HockeyApp + PowerBI
Telemetry (Next)
Radek – AzureME screenshot of usage
Azure Mobile Engagement
Telemetry - FTW
Results• Know if features are useful and successful• Reduction in technical debt• Prioritize features/platforms to no longer support• Quicker determination of service or app problemLessons• Capture all unexpected error scenarios• Instrument any major changes to see impact• Have a single view of app and service
A/B Testing (OneDrive)
A/B Testing (OneDrive)
A/B Testing* (Next)
Radek – screenshot of Google+ Alpha->Beta channels and/or (ideally) Azure Mobile Engagement instrumentation with being able to slice/dice crash data info
*not scientifically
A/B Testing - FTW
Results• > 50% increase in clicks to learn
more about purchase of more storage link• Shipped the i button to 100% of our users• Next: Fast feedback for ideas, low costLessons• Data trumps opinion!
OneDrive – Outlook – Next Apps
For The Win
Resources
Learning• DevOps Dimension Show on @CH9
Special episode on mobile DevOpshttp://aka.ms/DoDmobile
Products Mentioned• MSFT: http://hockeyapp.net• MSFT: http://parature.com• MSFT: http://aka.ms/azme (Azure Mobile Engagement)• MSFT: http://aka.ms/VSTSinfo (Visual Studio Team Services)• 3rd party: http://helpshift.com• 3rd Party: http://optimizely.com• 3rd Party: http://uservoice.com
Special thanks to:@RayyanJaber@ogkent@krakookerad