ab testing
Post on 15-Apr-2017
290 Views
Preview:
TRANSCRIPT
What is A/B testing?
● Getting info directly from the user’s behavior
● Change only one element, see how it goes● An excuse to test stuff in prod
How does that work?Access: the user got the content we are testingConversion: the user did what we wanted him to do (buy, submit address,...)
For each group, we compute:conversion rate = conversions/accesses
The group/content with the highest conversion rate wins.
The tools of the trade
● Varnish Cache Plus● Varnish Configuration Language (VCL)● Varnish Custom Statistics (VCS)● Some know-how
Varnish & VCL
● Awesomely fast HTTP Cache● Highly flexible● VCL allows you to inspect http objects and
make decisions/modifications based on this
Varnish Custom Statistics
● Special purpose in-memory time series database
● Aggregates data about transactions you care about, grouping them by key
● Uses VCL to assign keys to transactions● Plugs into Varnish log
Time windows{ "timestamp": "2015-11-18T14:33:30", "n_req": 434876, "n_miss": 3, "n_bodybytes": 248314196, "reqbytes": 32615703, "respbytes": 409345801, "berespbytes": 2268, "bereqbytes": 566, "ttfb_miss": 0.000231, "ttfb_hit": 0.000028, ... }
sub vcl_recv {if (req.url ~ “^/admin/”) {
std.log(“vcs-key: AdminAccess”);return (pass)
} }
Tagging transactions
Ok, back to A/B testingWe have to:
1. retrieve the group of a user, or select one for new users
2. make that group stick3. deliver different content based on group4. log accesses and conversions5. compute the results6. make a decision7. become rich thanks to that
Introduction togroup theory
sub vcl_recv {if (cookie.get("abgroup") ~ “~[AB]$”) {
set req.http.abgroup =cookie.get("abgroup");
} else { if (std.random(0, 99) < 90) {
set req.http.abgroup = "A";} else {
set req.http.abgroup = "B";}
} }
sub vcl_deliver{set resp.http.Set-Cookie =
"abgroup=" + req.http.abgroup +"; Expires=" +
cookie.format_rfc1123(now, 60m) +"path=/; httpOnly";
}
Come to the dark side,we have cookies
Content selection
Three (main) ways:● url rewriting in Varnish● Backend selection in Varnish● Vary headers
vcl_recv {if (req.url ~ "^/content.jpg") {
if (req.http.abgroup == "A") {set req.url = "/A/content.jpg";
} else {set req.url = "/B/content.jpg";
}}
}
Location rewrite
vcl_recv {if (req.url ~ "^/content.jpg") {
if (req.http.abgroup == "A") {set req.backend_hint = backendA;
} else {set req.backend_hint = backendB;
}}
}
Different backend
Vary headers
So, no extra VCL, but we need:● to set the abgroup header (done)● configure the backend to serve different
content depending on that header● configure the backend to reply with
“vary: abgroup”
if (req.url == "/") { std.log("vcs-key: access" +
req.http.abgroup); } else if (req.url == "/thanks.html") { std.log("vcs-key: conversion" +
req.http.abgroup); } }
Log accesses/conversions
top related