ab testing & optimization for creatives in user acquisition | erdem inan
TRANSCRIPT
AB Testing & Optimization for Creatives in Mobile User Acquisition
Erdem İnanDirector, BI & Marketing
Peak Games
Overview
• Peak Games & Marketing Team & Myself
• Why Creative Optimization is important?
• Creative Testing Process: From Planning to Execution
• Real Life Cases on Creative Optimization
• Q&A
Peak Games
• Founded in October 2010, based in Istanbul / Turkey, a team of 100 people• 300M+ Installs globally, 2 Games in Top 100 Grossing in US Stores, 2
Games in Top 5 Grossing in TR Stores.• Active in 150+ Countries with 10+ Titles in Synchronous Multiplayer &
Casual Categories• Flagship Titles: Toy Blast (US), Spades Plus (US), 101 Okey Plus (TR), Okey
Plus (TR)
Marketing Team & Myself
• BI & Marketing Team – A Team of 15 People: Analytics, User Acquisition & Art, Big Data Team – Providing services to 10+ products in User Acquisition, CRM and Analytics– 10+ Game Launches; Regular User Acquisition Campaigns in many countries at
large scale.
• Myself: – 3.5 years at Peak Games. – Leading the User Acquisition + CRM + Analytics teams– Engineering Background with an Economics MSc.
Why Creative Optimization is important?
Ad Impression
Ad Click
Ad Install
CTR
CVR
cpm
cpc
cpi
• Publishers want to maximize the revenue from their ad inventory, advertisers should be giving them highest cpms.
• The system is based on eCPM maximization
Advertisers / Metrics
Advertiser A Advertiser B Advertiser C
Target CPI 3 USD 2 USD 10 USDImpressions 1000 1000 1000
CTR 10% 20% 5%Clicks 100 200 50CVR 15% 25% 10%
Installs 15 50 5Expected Cost 45 USD 100 USD 50 USD
Calculated CPM 45 USD 100 USD 50 USD
• Advertiser B wins the auction, Advertiser C gets the second place, Advertiser A gets the third place.
Why Creative Optimization is important?
• Better creatives can lead to higher eCPMs => a. Can lead to higher scale at same ratesb. Can lead to cheaper installs.c. Can put you in an earlier impression in most ad networks: (Earlier
rewarded video impressions have higher CTRs and CVRs which will lead even better eCPMs and create a self-fulfilling loop)
d. Can even lead to higher quality installs.
Creative Testing Process: Understanding Users
• Research: – Facebook Audience Insights– Google Search Trends– Twitter Hashtags– Profiling Competitors’ Audiences– Searching Web
• Competitors’ Creatives:– Tools to check competitors’ best
performing creatives
• Creating a Typical Player: • Geographics: US, UK, CA• Demographics: F, 45+• Other Interests:
• Country Music, Bingo, Slots, Puzzle Games
• Music Shows, TV Shows, Celebrities• Things to consider
• Politically conservative• Raising Grandkids• Middle Income• More affinity: IL, TX, OH • Religious…
Creative Testing Process: Executing Tests
Ad Pattern 1 v1.0
Ad Pattern 2 v1.0
Ad Pattern 3 v1.0
Ad Pattern 4 v1.0
Ad Pattern 5 v1.0
Ad Pattern 2 v1.0
Ad Pattern 2 v1.1
Ad Pattern 2 v1.2
Ad Pattern 5 v1.2
Ad Pattern 5 v1.0
Ad Pattern 5 v1.1
Ad Pattern 5 v1.2.3
Ad Pattern 5 v1.2
Ad Pattern 5 v1.2.2
• Some Ground Rules:– Total image test budget should not exceed x
% of the total budget– ~30 installs per creative is usually enough for
a rough evaluation.– Use the previous test know-how in creating
next creatives. (buttons, cta, banners, backgrounds etc.)
– All creatives should run at the same time, for the same duration, either with same budget, or with same impressions
Creative Testing Process: Reporting & Analysis
• Finding an exceptional creative instead of many good creatives• Mind the “risky” inverse correlation between CTR and CVR• Remember the second and third best performing: Ad Fatigue • Checking Significance• Chartboost, Unity Ads, Facebook for easy & proper AB Testing.
Case: Image Optimization towards CPI
• Gin Rummy Plus => Image Optimization towards lower CPI• Fail: Lower CPI does not necessarily mean lower CPE.
• Above three has better CTRxCVR and lower CPI to our winning creatives• Misleading creatives can lead to lower % Actives thus higher CPEs• 30% lower performance in CPEs compared to our regular creatives.
Creative Testing: Iterating Images
12
4
3
5
• Discussing each experiments within the team.
• Decomposing Images: What made this image work?
1. CTA in the image2. Rubic Cube Item3. Game Play Background4. Shiny Play Button5. Contrast between background and
foreground items
• Create iterations of each element.• Test again!
Case: Calibrating between CTR and CVR
Base Image: Very Good CVR 1st Iteration: Increasing CTR 2nd Iteration: Further Increasing CTR
Pattern 1: Good CTR, Good CVR Pattern 2: High CTR, Low CVR Pattern 3: Above Average CTR & CVR
Case: Reaction to Different Creatives (1/2)
United StatesCTR: 1.55%
CR: 33%
TurkeyCTR: 1.58%
CR: 28%
NetherlandsCTR: 2.03%
CR: 34%
United StatesCTR: 0.88%
CR: 36%
TurkeyCTR: 3.44%
CR: 33%
NetherlandsCTR: 1.33%
CR: 37%
• Cultural Similarity & Localization in Ads are crucial in creative performance• Finding the right creatives for a segment of users may cut costs easily by 40-50%.
Case: Reaction to Different Creatives (2/2)
Good Performers Bad Performers
Turkey
Italy
US
• There are absolute winners & losers in creatives; but cultural effects can kick in.• Expectations & Prejudices might work: But A/B Testing for each region is critical
Case: Preparing Creatives for Different Times of the Year
• Easter, Valentine’s Day, 4th July, Month of Eid, Halloween Christmas are extraordinary days for users. Make sure you respond that.
• Such specific days in calendar might affect costs, available traffic and virality of ads
Creative Testing: Things to Know
• 500+ creatives for a mid-large scale game• Different ad units, channels & sizes might have different winning creatives.• Interstitials and banners usually have different winning patterns.• Dropping CTR with the same volume of installs is a warning sign to change creatives.• In creative testing the prices will be higher than an optimal campaign. • Emerging markets might have bandwidth problems => Make sure you have lower
sized creatives as well. • Video Testing:
– Video networks are extra sensitive to creative performance– Videos are difficult to test.– Test concepts => Make sure execution is as good as possible in each concept. – Thumbnail image has a high effect on performance.
Wrapping Up
• Creative Optimization brings greater scale at a cheaper price
• Success in creative optimization needs a systematic approach and a constant effort with a strong team.
• There are more failures than successes in creative optimization.
• Ad Creatives are cultural, contextual and usually a composition of certain elements.
