a/b testing in mobile
TRANSCRIPT
![Page 1: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/1.jpg)
A/B TESTING IN MOBILE
MAURICIO T ZAQUIA@mtzaquia
![Page 2: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/2.jpg)
MAURICIO TZAQUIA
APPLE DISTINGUISHED EDUCATOR, CLASS OF 2015
iOS DEVELOPER @
![Page 3: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/3.jpg)
MOBILEMARKET
![Page 4: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/4.jpg)
NEW USER RETENTION
0x
1x
2x
Web Apps
MOBILE TRANSACTIONS
Apps 54 %
Web 46 %
![Page 5: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/5.jpg)
MOBILE CONVERSION
Web
Apps
0x 1x 2x 3x
![Page 6: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/6.jpg)
”A/B TESTING IS A TERM FOR A RANDOMIZED EXPERIMENT WITH TWO VARIANTS, A AND B, WHICH ARE THE
CONTROL AND VARIATION IN THE CONTROLLED EXPERIMENT.”
Wikipedia, 2017
![Page 7: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/7.jpg)
![Page 8: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/8.jpg)
HYPOTHESIS EXPECTATION
IMPLEMENTATION
EXECUTION
ANALYSIS
DECISION
![Page 9: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/9.jpg)
/* code */
![Page 10: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/10.jpg)
button.setTitle("Buy", for: .normal)button.setTitleColor(UIColor.green, for: .normal)
// controller code...
// controller code...
![Page 11: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/11.jpg)
// controller code...
if (Experiments.ios_button_urgency.variant == 1) { button.setTitle("Buy before it's too late!", for: .normal) button.setTitleColor(UIColor.red, for: .normal) } else { button.setTitle("Buy", for: .normal) button.setTitleColor(UIColor.green, for: .normal) }
// controller code...
35% 42%
![Page 12: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/12.jpg)
button.setTitle("Buy before it's too late!", for: .normal) button.setTitleColor(UIColor.red, for: .normal)
// controller code...
// controller code...
![Page 13: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/13.jpg)
Experiments.ios_button_urgency.variant
![Page 14: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/14.jpg)
FEATURE FLAGS
![Page 15: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/15.jpg)
Experiments.ios_button_urgency.variant
![Page 16: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/16.jpg)
WHAT HAPPENS IF A WEBPAGE CRASHES?
![Page 17: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/17.jpg)
WHAT HAPPENS IF
AN APP CRASHES?
![Page 18: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/18.jpg)
”47% OF THE APPS CRASH AT LEAST 1% OF THE TIME, AND 32% OF THE APPS HAVE A
CRASH RATE OVER 2%”
Crittercism, 2016
![Page 19: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/19.jpg)
EVERYBODY BUGS
![Page 20: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/20.jpg)
FINDING CRASHES IS ”EASY”…
![Page 21: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/21.jpg)
…BUT WITH THE RIGHT TOOLS
NETWORKLOAD
SCREENISSUES
MEMORYWARNING
![Page 22: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/22.jpg)
NETWORKLOAD
Experiments.ios_deep_search
DAT
A E
XCH
AN
GE
OVE
R TI
ME
BASE VARIANT
![Page 23: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/23.jpg)
NETWORKLOAD
Experiments.ios_deep_search
APP
ABA
ND
ON
MEN
T O
VER
TIM
E
BASE VARIANT
![Page 24: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/24.jpg)
MEMORYWARNING
Experiments.ios_revamped_card
MEM
ORY
WA
RNIN
GS
OVE
R TI
ME
BASE VARIANT
![Page 25: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/25.jpg)
SCREEN ISSUES
Experiments.ios_table_refactor
1.26% ± 1.64%
BASE
VARIANT
SCREEN FREEZE OVER 1,5 SECONDS
![Page 26: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/26.jpg)
SCREEN ISSUES
Experiments.ios_layout_flow
-95.78% ± 7.19%
BASE
VARIANT
SCREEN VIEWS
![Page 27: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/27.jpg)
SCREEN FREEZE
Experiments.ios_table_refactor
BASE VARIANT
![Page 28: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/28.jpg)
FRAMEWORKS
HTTPS://GITHUB.COM/KEEPSAFE/SWITCHBOARD HTTPS://GITHUB.COM/MATTT/SKYLAB
![Page 29: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/29.jpg)
COMPANIES
![Page 30: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/30.jpg)
Q&A
![Page 31: A/B Testing in Mobile](https://reader031.vdocument.in/reader031/viewer/2022030309/58f2aec81a28abe40e8b4581/html5/thumbnails/31.jpg)
THANK YOUMAURICIO T ZAQUIA@mtzaquia