olly downs-sic-2011

MoneyballHow buying by “On Base Percentage”

rather than “RBIs” drives results in Display Advertising

Olly DownsConsulting Chief Scientist, AdReady

1

THE ART OF WINNING AN UNFAIR GAME

2002 New York Yankees 2002 Oakland Athletics

2

THE ART OF WINNING AN UNFAIR GAME

$125,000,000

$41,000,000

2002 New York Yankees 2002 Oakland Athletics

2

Baseball Digital Advertising

GoalRuns

(more than opponent)Actions

(more than alternative)

Asset Players Media / Data

Budget Player Salaries Media Purchased

Constraint9 Innings(27 outs) Campaign Duration

WHY?BASEBALL & DIGITAL ADVERTISING

3

Baseball Digital Advertising

How to Evaluate RBI (runs-batted-in)OBP (on-base-percentage)

eCPC / eCPA

The Premium Strategy A-Rod Home Pages / Sponsorships

The Non-Obvious Strategy

Find Undervalued Talent/Use Different Metrics

Find Undervalued, Non-Obvious Sites/Segments

Pick More Mature Players or Identify Commonalities of Younger Players with

Known Players

Intelligently Initialize and/or Quickly Adapt/Optimize Campaign Parameters

WHY?BASEBALL & DIGITAL ADVERTISING

4

MONEYBALLSTRATEGY

5

• We might want to buy runs, but we benefit from players who get on base

• Purchase Established Players based on different metrics – OBP vs. HR• Selectively draft College Baseball players rather than High School Players – stable statistics• Buying High School Players, look for similarities with mature players who have the right metrics

MONEYBALLSTRATEGY

5

• We might want to buy runs, but we benefit from players who get on base

• Purchase Established Players based on different metrics – OBP vs. HR• Selectively draft College Baseball players rather than High School Players – stable statistics• Buying High School Players, look for similarities with mature players who have the right metrics

• It’s been 9 years since the Oakland A’s incredible season

• Moneyball principles have garnered significant adoption across MLB• BUT inefficiencies still exist

MONEYBALLSTRATEGY

5

THE “EFFICIENT FRONTIER”BASEBALL

Salary Offered

Sign

ing

Prob

abilit

y

0%

100%

50%

6

THE “EFFICIENT FRONTIER”BASEBALL

What at least one other team is willing to pay

Salary Offered

Sign

ing

Prob

abilit

y

0%

100%

50%

6

THE “EFFICIENT FRONTIER”BASEBALL What at least one other

team is willing to pay

Salary Offered

Sign

ing

Prob

abilit

y

0%

100%

50%

6

EFFICIENT FRONTIERHIGH-COST, HIGH PERFORMANCE PLAYERS

7


• Top 10 players• By 2011 Salary

Players by Salary Team Salary

Alex Rodriguez New York Yankees $32,000,000Vernon Wells Los Angeles Angels $26,642,857CC Sabathia New York Yankees $24,285,714Mark Teixeira New York Yankees $23,125,000Todd Helton Colorado Rockies $20,275,000Miguel Cabrera Detroit Tigers $20,000,000Roy Halladay Philadelphia Phillies $20,000,000Ryan Howard Philadelphia Phillies $20,000,000Alfonso Soriano Chicago Cubs $19,000,000Carlos Lee Houston Astros $19,000,000

7



Players by AVG Team Salary

Skip Schumaker St Louis Cardinals $2,750,000Jimmy Rollins Philadelphia Phillies $8,500,000Chase Utley Philadelphia Phillies $15,285,714Jorge Posada New York Yankees $13,100,000Brett Gardner New York Yankees $529,500Ryan Braun Milwaukee Brewers $4,287,500Kelly Shoppach Tampa Bay Rays $3,000,000David Freese St Louis Cardinals $416,000Chris Young Arizona Diamondbacks $5,200,000Magglio Ordonez Detroit Tigers $10,000,000

• By 2011 Batting Average

7



Players by HR Team Salary

Jose Bautista Toronto Blue Jays $8,000,000Matt Kemp Los Angeles Dodgers $7,100,000Mark Reynolds Baltimore Orioles $5,333,333Dan Uggla Atlanta Braves $9,146,942Mike Stanton Florida Marlins $416,000Jacoby Ellsbury Boston Red Sox $2,400,000Jay Bruce Cincinnati Reds $2,791,667Michael Morse Washington Nationals $1,050,000Paul Konerko Chicago White Sox $12,000,000Troy Tulowitzki Colorado Rockies $5,500,000


• By 2011 Home Runs

7



Players by OBP Team Salary




• By On Base Percentage

7



Players by OBP Team Salary




• By On Base Percentage

• Optimizing these metrics and combinations of them at all costs is great if youdon’t have a constrained budget…

7

EFFICIENT FRONTIERHIGH-PERFORMANCE PER DOLLAR PLAYERS

8


• Now let’s look at the Top 10 Players per $ against the same metrics• Top 10 batters Per $• By 2011 Batting Average

Players by AVG/$ Team 2011 Salary

David Freese St Louis Cardinals $416,000Ryan Roberts Arizona Diamondbacks $423,500Brett Gardner New York Yankees $529,500Jason Jaramillo Pittsburgh Pirates $422,500Jason Donald Cleveland Indians $423,200Don Kelly Detroit Tigers $423,000Lucas Duda New York Mets $414,000Starlin Castro Chicago Cubs $440,000Emilio Bonifacio Florida Marlins $425,000Jason Bourgeois Houston Astros $423,000

8



Players by HR/$ Team 2011 Salary

Mark Trumbo Los Angeles Angels $414,000Mike Stanton Florida Marlins $416,000Andrew McCutchen Pittsburgh Pirates $452,500Neil Walker Pittsburgh Pirates $437,000Carlos Santana Cleveland Indians $416,600J.P. Arencibia Toronto Blue Jays $417,400Freddie Freeman Atlanta Braves $414,000Gaby Sanchez Florida Marlins $431,000Logan Morrison Florida Marlins $414,000Danny Valencia Minnesota Twins $437,500


8



Players by OBP/$ Team 2011 Salary

David Freese St Louis Cardinals $416,000Allen Craig St Louis Cardinals $414,000A.J. Ellis Los Angeles Dodgers $421,000Lucas Duda New York Mets $414,000Jason Donald Cleveland Indians $423,200Dexter Fowler Colorado Rockies $424,000Mike Stanton Florida Marlins $416,000Emilio Bonifacio Florida Marlins $425,000Jason Jaramillo Pittsburgh Pirates $422,500Carlos Santana Cleveland Indians $416,600


• By 2011 On Base Percentage

8

DAVID FREESE

“David Freese completed a startling night of comebacks with a home run leading off the bottom of the 11th inning to beat Texas 10-9 on Thursday, and suddenly fans all over got something they have waited a long time to see: Game 7 in the World Series."

2011 World Series MVP......and Top Moneyball Pick in the Major Leagues!

9

http://msn.foxsports.com/mlb/player/david-freese/610661

http://msn.foxsports.com/mlb/player/david-freese/610661

EFFICIENT FRONTIERCOST-EFFECTIVE WAYS TO LOAD THE BASES

10


• On-Base Percentage is still uncorrelated with Salary!!

0

0.1

0.2

0.3

0.4

OBP

$250,000

$750,000$1,500,000

$3,500,000$7,500,000

$15,000,000$26,000,000

A-Rod

Salary

10


0

0.1

0.3

0.4

0.5

OBP/$M

M

$250,000

$750,000

$1,500,000$3,500,000

$7,500,000$15,000,000

$26,000,000A-Rod

Salary

• Which means it’s clear where to put your money to maximize OBP per $MM!

10

EFFICIENT FRONTIERPLAYER PERFORMANCE PER $

• Looking for the hitter to deliver the RBI’s

• There’s a pretty good valuation of home-run hitters

0

44

88

131

175

RBI/$M

M

$500,000$750,000

$1,500,000$3,500,000

$7,500,000$15,000,000

$23,000,000A-Rod

Salary

11

MONEYADSTHE STRATEGY

12

MONEYADSTHE STRATEGY

BaseballWe might want to buy runs, but we benefit from players who get on base

Digital AdvertisingWe want to drive consumer action, but we can benefit by growing our remarketable audience

• Purchase Established Players based on different metrics – OBP vs. HR

• Target campaigns to optimize audience acquisition rather than actions, which demands a lower premium per impression

• Selectively draft College Baseball players rather than High School Players – stable statistics

• Find pockets of inventory that we know from historical data will deliver high-quality click-based audience at low cost

• Buying High School Players, look for commonalities with the background of mature players who now have good “different metrics”

• Buy previously unseen media based on commonality with seen media that performs well against our new target metrics

12

0

0.4

0.8

1.1

1.5

Delivery=

Spend/Budget

$0.00$0.20

$0.40$0.60

$0.80$1.00

$1.30$1.60

CPC Bid($)

2nd Price Auction Model Google KW CPC Delivery

THE “EFFICIENT FRONTIER”DISPLAY ADVERTISING - DELIVERY - WINNING MEDIA

13

0

0.4

0.8

1.1

1.5

Delivery=

Spend/Budget

$0.00$0.20

$0.40$0.60

$0.80$1.00

$1.30$1.60

CPC Bid($)

2nd Price Auction Model Google KW CPC Delivery

THE “EFFICIENT FRONTIER”DISPLAY ADVERTISING - DELIVERY - WINNING MEDIA

Diminishing Returns

13

THE EFFICIENT FRONTIERDISPLAY ADVERTISING - PERFORMANCE

0

1.00

2.00

3.00

4.00

1/CPA

Perf

$0.00 $0.50 $1.00 $1.50 $2.00 $2.50 $3.00 $3.50 $4.00$4.50$5.00$5.50$6.00

Bid, CPC ($)

14


0

1.00

2.00

3.00

4.00

1/CPA

Perf

$0.00 $0.50 $1.00 $1.50 $2.00 $2.50 $3.00 $3.50 $4.00$4.50$5.00$5.50$6.00

Bid, CPC ($)

At low bids, we win low quality

impressions that have low CTR and

convert poorly

14


0

1.00

2.00

3.00

4.00

1/CPA

Perf

$0.00 $0.50 $1.00 $1.50 $2.00 $2.50 $3.00 $3.50 $4.00$4.50$5.00$5.50$6.00

Bid, CPC ($)

At low bids, we win low quality

impressions that have low CTR and

convert poorly

At high bids, we buy premium impressions that result in high

CPC and CPA

14

EFFICIENT FRONTIERFINDING POCKETS OF HIGH PERFORMANCE EVEN IN UNSEEN MEDIA

15

• We use predictive models to discover statistically similar media to media not previously run


15


• This allows us to identify optimal combinations of media sources, budgets to apply and bidding delivery and performance curves


15


• This allows us to identify optimal combinations of media sources, budgets to apply and bidding delivery and performance curves

• At each node of the tree we have both spend, initial bid, delivery and performance models


15

EXAMPLEADVERTISING EMPLOYING THE STRATEGIES

• These models are estimated dynamically and used to learn quickly

0

0.38

0.75

1.13

1.50

1 2 3 4 5 6 7 8 9 10 11 12 13$0

$75.00

$150.00

$225.00

$300.00

$69.15

$276.45

$165.98

$127.89

Del

iver

y=Sp

end/

Budg

et

Data Points

CPA

($)

Delivery CPA

Commonality-Based Intelligent Initial Bid

(Bid=$2.20)

16



0

0.38

0.75

1.13

1.50

1 2 3 4 5 6 7 8 9 10 11 12 13$0

$75.00

$150.00

$225.00

$300.00

$69.15

$276.45

$165.98

$127.89

Del

iver

y=Sp

end/

Budg

et

Data Points

CPA

($)

Delivery CPA


(Bid=$2.20)0

0.4

0.8

1.1

1.5

Delivery=

Spend/Budget

CPC Bid($)

16



0

0.38

0.75

1.13

1.50

1 2 3 4 5 6 7 8 9 10 11 12 13$0

$75.00

$150.00

$225.00

$300.00

$69.15

$276.45

$165.98

$127.89

Del

iver

y=Sp

end/

Budg

et

Data Points

CPA

($)

Delivery CPA


(Bid=$2.20)

First Bid Optimization (Optimize Delivery Bid=$3.85)

16



0

0.38

0.75

1.13

1.50

1 2 3 4 5 6 7 8 9 10 11 12 13$0

$75.00

$150.00

$225.00

$300.00

$69.15

$276.45

$165.98

$127.89

Del

iver

y=Sp

end/

Budg

et

Data Points

CPA

($)

Delivery CPA


(Bid=$2.20)


0

0.4

0.8

1.1

1.5

Delivery=

Spend/Budget

CPC Bid($)

16



0

0.38

0.75

1.13

1.50

1 2 3 4 5 6 7 8 9 10 11 12 13$0

$75.00

$150.00

$225.00

$300.00

$69.15

$276.45

$165.98

$127.89

Del

iver

y=Sp

end/

Budg

et

Data Points

CPA

($)

Delivery CPA


(Bid=$2.20)


0

1.50

3.00

4.50

6.00

1/CPA

Perf

Bid, CPC ($)

16



0

0.38

0.75

1.13

1.50

1 2 3 4 5 6 7 8 9 10 11 12 13$0

$75.00

$150.00

$225.00

$300.00

$69.15

$276.45

$165.98

$127.89

Del

iver

y=Sp

end/

Budg

et

Data Points

CPA

($)

Delivery CPA


(Bid=$2.20)


Second Bid Optimization(Optimize Performance

/Delivery Trade-Off, Bid=$2.00)

16

MONEYADSFROM BASEBALL TO DISPLAY ADVERTISING

17


• Batter Up!

17


• Batter Up!

• Get on base!

7 ways!

17


• Batter Up!

• Get on base!

• Batter Up!

17


• Batter Up!

• Get on base!

• Batter Up!• Get on base!

17


• Batter Up!

• Get on base!


• Get the Slugger up!!...

17


• Batter Up!

• Get on base!


• Get the Slugger up!!...• ...and run them in! !!!

17


18


• Construct the media plan...• ...to efficiently acquire retargetable

audience

18



audience• Keyword Campaigns - optimized for click-

generation

• Site Targeted Campaigns - targeted to placements optimized to get clicks for advertiszer

Many ways!

18




generation


• Look-a-like Users/Behavioral Segments

• Social Media and Search

18




generation




• Retarget!

18




generation




• Retarget!• ..and convert the audience!

!!!

18




generation





19




generation





Targeted Campaign

Impressions

Clicks

Actions

Driving traffic through to one-time conversion.

Treat first-time and returning visitors the same

Conventional Approach

19




generation





Driving Traffic -> Building Retargeting Pool

Retargeting established cookie pool, with message contextualized

to previous site visits, to drive conversion

Impressions

Clicks/Cookies

Actions

Broad Audience Acquisition Followed by

Retargeting

MoneyAds Approach

19

EXAMPLECONVENTIONAL APPROACH

Retargeting Campaign on current cookie pool

Impressions

Clicks

Actions

Site Content Campaign

eCPM: $1.50eCPC: $1.09eCPA: $184.92 eCPM: $2.92

eCPC: $2.36eCPA: $73.02

20

EXAMPLEMONEYADS APPROACH

21


Retargeting campaign on expanded cookie pool

Impressions

Clicks/Cookies

Actions

eCPM: $2.92eCPC: $2.36eCPA: $73.02Conversion Rate: 3.23%

Audience Acquisition Campaign

eCPM: $1.50eCPC: $1.09

21


Retargeting campaign on expanded cookie pool

Impressions

Clicks/Cookies

Actions

eCPM: $2.92eCPC: $2.36eCPA: $73.02Conversion Rate: 3.23%

Audience Acquisition Campaign

eCPM: $1.50eCPC: $1.09

Overall Performance

eCPA =CPCacquisition +CPCretargeting

Retargeting Conversion Rate= $106.95

21

TAKEAWAYS

22

1 - Focus on metrics that count towards a more balanced view of marketing success - value all the methods for getting on base.

TAKEAWAYS

22


2 - Be measured about what you buy - buy college rather than high-school players.

TAKEAWAYS

22



3 - Diversify your talent - not everyone can afford to buy A-Rod!

TAKEAWAYS

22



3 - Diversify your talent - not everyone can afford to buy A-Rod!

4 - You don’t need to spend a lot to be successful - use focused strategies and robust data.

TAKEAWAYS

22

THANK [email protected]

@mathandporsches

23

mailto:[email protected]

mailto:[email protected]

olly downs-sic-2011

Technology