A/B Testing in Android

Nir HartmannDrippler

Droidcon Tel-Aviv 2014

Why do we need A/B Testing?

• Tests takes the guesswork out

• Enables data-backed decisions

• Enhances engagement and retention

Road map

• What is an A/B test ?

• Segmentation

• Multiple Experiments

What is A/B Test ?

• Case study – onboarding screen

Define the test

• Hypothesis – The layout with the Google+ button at the left will increase the number of total registered users

• Goal – A registered user (the user can skip registration)

• View event- Login fragment onCreate(),

setRetainInstance(true)- Login activity onCreate(), null

savedInstanceState• Variables – Facebook button position (left or right)• Participants – New users

Amazon A/B Testing SDK

Like Android:• Very customizable, you can do just about anything as

long as you know what it is you want to do

• It’s free

• Drippler created an open source library that simplify the process

Setup the A/B test

• Setup identifier – https://developer.amazon.com/al/index.html

Setup the A/B test

• Create a project

Setup the A/B test

• Create the test

Dive into the code

• https://github.com/Drippler/ABTester

public class MyApplication extends Application {

@Overridepublic void onCreate() {

super.onCreate();ABTester.init(getApplicationContext(),

"my_public_key", "my_private_key");}

}

private void initLoginActivityTest() { try { ABTester.syncPreFetch( TimeUnit.SECONDS.toMillis(15),

new ABTest("Login page test", false, "Facebook is first”) ); } catch (TimeoutException e) { // Couldn't reach amazon servers }}

Fetch the test

Login Fragment

@Overridepublic void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);setRetainInstance(true);ABTester.recordEvent(

"Login fragment shown", false);

}

Login Fragment@Overridepublic View onCreateView(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState) { boolean shouldShowFacebookFirst = ABTester.getBoolean("Login page test", "Facebook is first", false); if (shouldShowFacebookFirst) return inflater.inflate(R.layout. facebook_first, null); else return inflater.inflate(R.layout. google_first, null);}

Report goal event

public void onUserLoggedIn() { ABTester.recordEvent("Sign in", false);}

@Overrideprotected void onPause() { super.onPause(); /* Submit the events that were previously stored locally. * asynchronously * call it in the onPause() method of an activity */ ABTester.submitEvents();}

Analyze the resultsVariation Views Conversions Conversion

rateChange

Google+ first

1064 320 30.08%

Facebook first

1043 250 23.97% -20.30%

Dice experiment

Goal: maximize the amount of 3’s we get in a 100 dice roll

Dice experiment

Hypothesis: wearing a hat will increase the chance to roll a 3

Analyze the resultsVariation Views Conversions Conversion

rateChange

Hat off 100 31 31.00%Hat on 100 38 38.00% +22.58%

Conversion is never a single number.

Confidence level

• Measure the reliability of an estimate– The confidence levels help us understand if the

results are different merely by chance or by reason

• 95% confidence level is considered good

Analyze the resultsVariation Views Conversions Conversion

rateChange

Google First

1064 320 30.08% ± 2.32%

Facebook First

1043 250 23.97% ± 2.18%

-20.30%

• Confidence level of 99%

Analyze the results

https://developer.amazon.com/public/apis/manage/ab-testing/doc/math-behind-ab-testing

Choose the best variation

• Launch– Choose the winning variation– Control the percentage of customers that receive

a new feature

Road map

• What is an A/B test ?

• Segmentation

• Multiple Experiments

Segmentation

Define the test

• Hypothesis – Coloring the “Rate” button, will increase the button’s click rate

• Goal – Click event on the “Rate” button• View event – RateUsDialogFragment show();• Variables – “Rate” button color• Participants – All users

Create the test

Rate us DialogFragment

public class RateUsDialog extends DialogFragment {

public static void show(FragmentManager fm, int color) { RateUsDialog rateUs = new RateUsDialog(); Bundle extras = new Bundle(); extras.putInt(“color”, color); rateUs.setArguments(extras); rateUs.show(fm, “my tag”); ABTester.recordEvent("Rate us dialog shown", false); }

Rate us DialogFragment

@Overridepublic Dialog onCreateDialog(Bundle savedInstanceState) { int color = getArguments().getInt("color"); return createColoredDialog(color);}

Rate us DialogFragment

private Dialog createColoredDialog(int color) { ... .setPositiveButton("Rate", new OnClickListener() { @Override

public void onClick(DialogInterface dialog, int which) {

ABTester.recordEvent("Rate button click", false);}

}); return myDialog;}

Rate us test1) Asynchronously prefetching SplashActivity

Default timeout is 60 seconds, and can be overriden by using preFetch(long timeout, ABTest... Test)

ABTester.preFetch( new ABTest("Rate us test", false, "Rate button color") );

Rate us test2) Show the dialog String fetchedColor = ABTester.getString( "Rate us test", "Rate button color", "#F5F5F5"); int color = Color.parseColor(fetchedColor); RateUsDialog.show(getFragmentManager(), color);

3) Submitting the results onPause() ABTester.submitEvents();

Analyze the resultsVariation Views Conversions Conversion rate Change Confidence

Control (white) 865 234 27.05%

± 1.51%

Variation A (green) 904 250 27.65%

± 1.49% -0.2%

Variation B (red) 830 230

27.71% ± 1.55% +2.4% 51%

What can I do with these results?

Segmentation

Variation Views Conversions Conversion rate Change ConfidenceControl (white) 432 92 21.30% ± 1.97%

Variation A (green) 464 165 35.56% ± 2.22% +66.9% 98.7%

Variation B (red) 420 120 28.57% ± 2.20% +34.1%

Variation Views Conversions Conversion rate Change Confidence

Control (white) 433 142 32.79% ± 2.26% +22.2% 97.1%

Variation A (green) 440 85 19.32% ± 1.88% -27.9%

Variation B (red) 410 110 26.83% ± 2.19% -18.8%

Under 40

Over 40

Define the test

• Hypothesis – Coloring the “Rate” button, will increase the button click rate

• Goal – Click event on the “Rate” button• View event – RateUsDialogFragment show();• Variables – “Rate” button color• Participants – All users

Define the test

• Hypothesis – Coloring the “Rate” button, will increase the button click rate

• Goal – Click event on the “Rate” button• View event – RateUsDialogFragment show();• Variables – “Rate” button color• Participants – Age specific tests

Setup the A/B test

• Create a segment

Create the segment

Assign the segment

• In your code before the fetch– ABTester.addDimension(”age", myAge);

• Dimension will not change during the experiment

• ABTester library will automatically add a “percentile” dimension– ABTester.addDimension(“percentile”, new Random().nextInt(100));

Road map

• What is an A/B test ?

• Segmentation

• Multiple Experiments

Multiple Experiments

Multiple Experiments

Multiple Experiments

• Serial tests– Run the tests one after the other, without the

need to redistribute your app– More accurate but takes more time

ABTester.preFetch( new ABTest("Rate us test", false, "Rate button color", "Rate actionbar icon"));

Multiple Experiments

• Parallel tests– Run the tests together, increasing the ‘noise’ for

dependent tests – Faster

ABTester.preFetch( new ABTest("Rate us button", false, "Rate button color”) new ABTest("Rate us actionbar", false, "Rate button icon”) );

Not a replacement for common sense

Thanks, any questions ?

Don’t overthink it, use Drippler’s A/B test library

https://github.com/Drippler/ABTester

Nir Hartmann, nhartmann@drippler.comDrippler

Droidcon Tel-Aviv 2014