acquisio 2015 advancements in machine learning
TRANSCRIPT
BRIGHT IDEASSTART HERE.
2015 SUMMIT MONTREAL | JULY 29+30
Advancements in Machine Learning
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 2ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
Advancements in Machine Learning
30 Jul 2015Bryan M. Minor, Ph.D.
Chief Scientist, Acquisio
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 3ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
Outline
What is Machine Learning?
BBM problem statement
Examples of how we use Machine Learning with BBM
Gamification
Acquisio and Machine Learning
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 4ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
What is Machine Learning?
Machine learning is a subfield of computer science that evolved from the study of pattern recognition and computational learning theory in artificial intelligence. Machine learning explores the construction and study of algorithms that can learn from and make predictions on data.
– Wikipedia (https://en.wikipedia.org/wiki/Machine_learning)
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 5ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
BBM problem:• For a fixed Budget for budget period (month)
– With a group of Campaigns (Budget Group)
• Make Daily Budget last whole Day
• Maximum Average CPC per day limit
• Fairly compete Campaigns based on value of Clicks (conversions)
• Maximize Clicks (conversions)
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 6ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
BBM Theory
C B
AAA
minCPC
0 2 4 6 8 1 00
5
1 0
1 5
2 0
2 5
3 0
3 5
C P C
Clicksday
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 7ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
Using Machine Learning in solving BBM problem
Setting Daily Budget
Setting Bid every 30 minutes
Anomaly detection
Day of week % of spend allocation
Managing Mobile bidding
Allocation of Budget across Publishers (AdWords, Bing, Yahoo!Japan,…)
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 8ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
BBM Spend - Jun 2015
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 9ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
BBM Constraint Obedience - Jun 2015
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 10ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
Proving Machine Learning works:
20,000 Campaigns in AdWords• 12,000 on BBM• 8,000 not on BBM
June 2015
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 11ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
BBM Search - Daily Budget spend
Case (Spent Daily Budget %) times (Not BBM %)BBM 3.6BBM (3.7+ grade) 4.0
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 12ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
BBM Search performance – Daily Budget Spent caseCase Imp Share IS LTB IS LTR CPC Avg. Pos.Not BBM 47.28% 32.49% 29.14% $6.46 2.306BBM 55.42% 18.43% 26.13% $4.04 2.417BBM (3.7+ grade) 54.99% 15.84% 20.23% $2.95 2.484
Case Imp Share % IS LTB % IS LTR % CPC % Avg. Pos. %BBM 17.2% -43.3% -10.3% -37.5% -4.8%BBM (3.7+ grade) 16.3% -51.2% -30.6% -54.4% -7.2%
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 13ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
BBM Search performance – Daily Budget not Spent caseCase Imp Share IS LTR CPC Avg. Pos.Not BBM 64.51% 35.49% $4.29 2.50BBM 77.18% 22.82% $4.05 2.22BBM (3.7+ grade) 73.41% 26.59% $2.91 2.35
Case Imp Share % IS LTR % CPC % Avg. Pos. %BBM 19.6% -35.7% -5.7% 11.1%BBM (3.7+ grade) 13.8% -25.1% -32.3% 6.4%
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 14ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
Daily Budget Spend (%)
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 15ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
Distributions of Daily Budget spent (%) - Human
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 16ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
Distributions of Daily Budget spent (%) – BBM no Skynet
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 17ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
Distributions of Daily Budget spent (%) – BBM with Skynet
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 18ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
Day of Week % Spend Machine Learning
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 19ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
Gamification
Rewards good behavior
Enhances inner commitment
1. Autonomy
2. Mastery
3. Connection
BRIGHT IDEASSTART HERE.
©2015 All rights reserved 20ACQUISIO 2015 SUMMIT MONTREAL – JULY 29+30 #ACQsummit
Acquisio and Machine Learning
Big Data is Different
Continuous discovery of key relationships for Predictions
Fine grain refinement of Predictions
Cornerstone of Gamification
Anomaly detection
Cross Publisher optimization
BRIGHT IDEASSTART HERE.
2015 SUMMIT MONTREAL | JULY 29+30
Thank you!
Bryan M. Minor, Ph.D.Chief Scientist, [email protected]@BryanMinorPhD