research © 2008 yahoo! statistical challenges in online advertising deepak agarwal deepayan...

Statistical Challenges in Online Advertising

Deepak Agarwal

Deepayan Chakrabarti(Yahoo! Research)

Online Advertising

• Multi-billion dollar industry, high growth

– $9.7B in 2006 (17% increase), total $150B

• Why this will continue?

– Broadband cheap, ubiquitous

– “Getting things done” easier on the internet

– Advertisers shifting dollars

• Why does it work?

– Massive scale, automated, low marginal cost

– Key: Monetize more and better, “learn from data”

– New discipline “Computational Advertising”

What is “Computational Advertising”?

New scientific sub-discipline, at the intersection of – Large scale search and text analysis

– Information retrieval

– Statistical modeling

– Machine learning

– Optimization

– Microeconomics

Online advertising: 6000 ft Overview

Ad Network

Content

Pick ads

Content Provider

Examples:Yahoo, Google,

MSN, RightMedia, …

Outline

• Background on online advertising

– Sponsored Search, Content Match, Display, Unified marketplace

• The Fundamental Problem

• Statistical sub-problems:

– Description

– Existing methods

– Challenges

Different flavors

Online Advertising

Revenue Models

Advertising Setting

CPM CPC CPA

Display Content Match

Sponsored Search

Ad exchanges

Revenue Models

CPM CPC CPA

Content

Pick ads

Cost Per iMpression

$Content Provider

Ad Network

Revenue Models

CPM CPC CPA

Content

Pick ads

Cost Per Click

$Content Provider

Ad Networkclick

Revenue Models

CPM CPC CPA

Content

Pick ads

Cost Per Action

$Content Provider

Ad Networkclick

Revenue Models

• Example: Suppose we show an ad N times on the same spot

• Under CPM: Revenue = N * CPM

• Under CPC: Revenue = N * CTR * CPC

CPM CPC CPA

Click-through Rate(probability of a click given an impression)

Depends on auction

mechanism

Auction Mechanism

• Revenue depends on type of auction– Generalized First-price:

• CPC = bid on clicked ad

– Generalized Second-price: • CPC = bid of ad below clicked ad (or the reserve price)

• CPC could be modified by additional factors

• [Optimal Auction Design in a Multi-Unit Environment: The Case of Sponsored Search Auctions] by Edelman+/2006

• [Internet Advertising and the Generalized Second Price Auction…] by Edelman+/2006

Revenue Models

• Example: Suppose we show an ad N times on the same spot

• Under CPM: Revenue = N * CPM

• Under CPC: Revenue = N * CTR * CPC

• Under CPA: Revenue = N * CTR * Conv. Rate * CPA

CPM CPC CPA

Conversion Rate(probability of a user conversion on the advertiser’s landing page

given a click)

Revenue Models

website traffic

website traffic +ad relevance

Revenue dependence

website traffic +ad relevance +landing page quality

Relevance to advertisers

Prices and Bids

Ease of picking ads

Background

Online Advertising

Revenue Models

Advertising Setting

CPM CPC CPA

Sponsored Search

Ad exchanges

Advertising Setting

Ad Network

Content

Pick ads

Content Provider

• What do you show the user?

• How does the user interact with the ad system?

Advertising Setting

Sponsored Search

Advertising Setting

Sponsored Search

Pick ads

Advertising Setting

• Graphical display ads

• Mostly for brand awareness

• Revenue model is typically CPM

Sponsored Search

Advertising Setting

Sponsored Search

Content match ad

Advertising Setting

Sponsored Search

Pick ads

Text ads

Match ads to the content

Advertising Setting

• The user intent is unclear

• Revenue model is typically CPC

• Query (webpage) is long and noisy

Sponsored Search

Advertising Setting

Sponsored Search

Search Query

Sponsored Search

Pick ads

Text ads

Search Query

Match ads to the query

Advertising Setting

• User “declares” his/her intention

• Click rates generally higher than for Content Match

• Revenue model is typically CPC (recently some CPA)

• Query is short and less noisy than Content Match

Sponsored Search

Summary

• Different revenue models

– Depends on the goal of the advertiser campaign

• Brand awareness

– Display advertising

– Pay per impression (CPM)

• Attracting users to advertised product

– Content Match, Sponsored Search

– Pay per click (CPC), Pay per action (CPA)

Background

Online Advertising

Revenue Models

Advertising Setting

CPM CPC CPA

Sponsored Search

Ad exchanges

Unified Marketplace

• Publishers, Ad-networks, advertisers participate together in a singe exchange

• Publishers put impressions in the exchange; advertisers/ad-networks bid for it

• CPM, CPC, CPA are all integrated into a single auction mechanism

Overview: The Open Exchange

Transparency and value

Has ad impression to sell --AUCTIONS

Bids $0.50Bids $0.75 via Network…

… which becomes $0.45 bid

Bids $0.65—WINS!

AdSenseAd.com

Bids $0.60

Unified scale: Expected CPM

• Campaigns are CPC, CPA, CPM

• They may all participate in an auction together

• Converting to a common denomination is a challenge

Outline

– Description

– Challenges

Outline

– Sponsored Search and Content Match

– Description

– Challenges

Display Advertising

• Main goal of advertisers: Brand Awareness

• Revenue Model: Primarily Cost per impression (CPM)

• Traditional Advertising Model:

1. Ads are targeted at particular demographics (user characteristics)

1. GM ads on Y! autos shown to “males above 55”

2. Mortgage ad shown to “everybody on Y! Front page”

2. Book a slot well in advance– “2M impressions in Jan next year”

– These future impressions must be guaranteed by the ad network

Display Advertising

• Fundamental Problem: Guarantee impressions to advertisers

Young US

FemaleY! Mail

1. Predict Supply:

• How many impressions will be available?

• Demographics overlap

2. Predict Demand:

• How much will advertisers want each demographic?

Display Advertising

Young US

FemaleY! Mail

1. Predict Supply

2. Predict Demand

3. Find the optimal allocation

• subject to supply and demand constraints

Display Advertising

1. Predict Supply

2. Predict Demand

3. Find the optimal allocation, subject to constraints

• Optimal in terms of what objective function?

Allocation through Optimization

• Optimal in terms of what objective function?

– E.g. Maximize value of remaining inventory • Cherry-picks valuable inventory, saves it for later

– Fairness• “Spreads the wealth” subject to constraints

sisupply demand

Example

Young US

FemaleY!

US & Y(2)

Supply Pools

DemandUS, Y, nFSupply = 2Price = 1

US, Y, FSupply = 3Price = 5

Supply Pools

How should we distribute impressions from the supply pools to satisfy this

demand?

Example (Cherry-picking)

• Cherry-picking: Fulfill demands at least cost

US & Y(2)

Supply Pools

DemandUS, Y, nFSupply = 2Price = 1

US, Y, FSupply = 3Price = 5

demand?

Example (Fairness)

• Cherry-picking: Fulfill demands at least cost

• Fairness:Equitable distribution of available supply pools US & Y

Supply Pools

DemandUS, Y, nFSupply = 2

Cost = 1

US, Y, FSupply = 3

Cost = 5

demand?

Objective functions

pool of Value:

poolfor inventory remaining :

Maximize

)~/log(

Minimize :function Objective

.V valueof

function decreasinglly monotonica becan general,In

allocation alproportion1 ..

SSjjjwkwjjjk

wxXdXwxxkj

Fairness""

Display Advertising

1. Predict Supply

2. Predict Demand

3. Find the optimal allocation, subject to constraints– Pick the right objective function

• Further issues:

– Risk Management: Supply and demand forecasts should have both mean and variance

– Forecast aggregation: Forecasts may be needed over multiple resolutions, in time and in demographics

Display Advertising

1. Predict Supply

2. Predict Demand

3. Find the optimal allocation, subject to constraints– Pick the right objective function

• Forecasting accuracy is critical!

– Overshoot under-delivery of impressions unhappy advertisers

– Undershoot loss in revenue

Outline

– Sponsored Search and Content Match

– Description

– Challenges

Sponsored Search and Content Match

• Given a query:

– Select the top-k ads to be shown on the k slots to maximize total expected revenue

• What is total expected revenue?

Example (Content Match)

Ad Position 1

Ad Position 2

Ad Position 3

Example (Content Match)

Reminder: Auction Mechanism

• Revenue depends on type of auction– Generalized First-price:

• CPC = bid on clicked ad

– Generalized Second-price: • CPC = bid of ad below clicked ad (or the reserve price)

• CPC could be modified by additional factors

• Total expected revenue = revenue obtained in a given time window

• [Optimal Auction Design in a Multi-Unit Environment: The Case of Sponsored Search Auctions] by Edelman+/2006

• [Internet Advertising and the Generalized Second Price Auction…] by Edelman+/2006

• Given a query:

• What affects the total revenue?

– Relevance of the ad to the query

– Bids on the ads

– User experience on the ad landing page (ad “quality”)

– Expected total revenue is some function of these.

• Given a query:

• Fundamental Problem:

– Estimate relevance of the ad to the query

Ad Relevance Computation

Overview

• Information Retrieval (IR)

– Techniques

– Challenges

• Machine Learning using Click Feedback

• Online Learning

IR-based ad matching

• “Why not use a search engine to match ads to context?”– Ads are the “documents”

– Context (user query or webpage content) is the “query”

• Three broad approaches:– Vector space models

– Probabilistic models

– Language models

• Open-source software is available:

– Lemur (www.lemurproject.org)

• Vector space models:

– Each word/phrase in the vocabulary is a separate dimension

– Each ad and query is a point in this vector space

– Example: cosine similarity

• Probabilistic models

• Language models

• Q1: How can we score the goodness of an ad for a context?

• Cosine similarity:

• Advantages:

– Simple and easy to interpret

– Normalizes for different ad and context lengths

Ad vectorQuery vector

• Vector space models

• Probabilistic models:

– Predict, for every (ad, query) pair, the probability that the ad is relevant to the query

– Example: Okapi BM25

• Language models

• Okapi BM25:

Term Frequency

Parameters

Norm. document

length

Inverse Document Frequency

Term Frequency

in query

• Okapi BM25:

• Advantages:– Different terms are weighted differently

– Tunable parameters

– Good performance

Term Frequency

Norm. document

length

Term Frequency

in query

• Vector space models

• Probabilistic models

• Language models:

– Ads and queries are generated by statistical models of how words are used in the language

– What statistical models can be used?

– How do we translate query and ad generation probabilities into relevance?

• What statistical models can be used?

– Bigram model

– Multinomial model• Given any ad or query, we can compute the parameter

setting most likely to have generated the document

Term Frequency

Term probability (model parameters)Total length

How do we translate query and ad generation probabilities into relevance?

Method 1

• Compute most likelyquery and ad params

• Generate ad usingquery params

• High probability high relevance

QueryQuery

params

Method 2

• Generate query usingad params

• High probability high relevance

QueryQuery

params

Method 3

• Compute KL-divergencebetween params

• Low KL-divergence high relevance

QueryQuery

params

• New methods to combine syntactic and semantic information

• For example, “A Semantic Approach to Contextual Advertising” by Broder+/SIGIR/2007– Words only provide syntactic clues

– Classify ads and queries into a common taxonomy

– Taxonomy matches provide semantic clues

Overview

– Techniques

– Challenges

• Online Learning

Challenges of IR-based ad matching

• Word matches might not always work

Woes of word matching

Extract Topical info

Increases coverage,more relevant match

• Word matches might not always work

• Works well for frequent words, what about rare words? Long tail, big revenue impact.– Remedy: Add more matching dimensions (phrase,…)

• Static, does not capture effect of external factors– E.g. high interest in basketball page due to an event;

dies off after the event

– Click feedback a powerful way of capturing such latent effects; difficult to do it through relevance only

• Relevance scores may not correspond to CTR; does not provide estimates of expected revenue

• Heterogeneous corpus (query, ads). Single tfidf scores not applicable.

• In content match, queries long and noisy

• Partial feedback does not work

– Not scalable

• Ads are small, relevance of landing page difficult to determine (video, image, text)

Machine Learning using Click Feedback

Overview

– Advantages and Challenges of Click Feedback

– Feature-based models• Description

• Case Studies

– Hierarchical Models

– Matrix Factorization and Collaborative Filtering

– Challenges and Open Problems

• Online Learning

Learning from Click Feedback

• Learning relevance from partial human-labeled training data

– Attractive but not scalable

• Users provide us direct feedback through ad clicks

– Low cost and automated learning mechanism

– Large amounts of feedback for big ad-networks

• Estimation problem:

– Estimate CTR = Pr(click| query, ad, user)

Learning from Clicks: Challenges

• Noisy labels– Clicks (unscrupulous users gaming the system)

– Negatives (not clear; I never click on ads )

• Sparseness– (query, ad) matrix has billions of cells; long tail

• Too few data points in large number of cells; MLE has high variance

• Goal is to learn the best cells, not all cells

• Dynamic and seasonal effects– CTRs evolve; subject to seasonal effects

• Summer, Halloween,..

• Palin ads popular yesterday, not today

Challenges continued

• Selection bias– We never showed watch ads on golf pages

• Positional bias, presentation bias– Same ad performs differently at different positions

• Slate bias– Performance of ad depends on other ads that were

displayed

Overview

• Case Studies

• Online Learning

Feature based approach

• Query, Ad characterized by features– Query: bag-of-words, phrases, topic,…

– Ads: bag-of-words, keywords, size,…

• Query feature vector: q

• Ad feature vector: a

• Pr(Click|Q,A) = f(q,a;θ)

• Example: Logistic regression– log-odds(Pr(Click|Q,A)) = q’ W a

– W estimated from data

Feature based models: Challenges

• Challenges– High dimensional, need to regularize (Priors)

– De-bias for positional and slate effects

– Negative events to be weighted appropriately

• Go through case studies reported in literature

Predicting Clicks: Estimating the Click-through rates of new ads: Richardson et al, WWW 2007

• Estimate CTR of new ads in Sponsored search

• Log-odds(CTR(ad)) = wifi(ad)

• Features used:

– Bid term CTRs of related ads (from other accounts)• CTRs of all other ads with keyword “camera”

– Appearance, attention, advertiser reputation, landing page quality, relevance of bid terms to ad, bag-of-words in ad.

• Does not capture interactions between (query, ad), main focus is to estimate CTR of new ads only

• Negative events down-weighted based on eye-tracking study

Combining relevance with Click Feedback, Chakrabarti et al, WWW 08

• Content Match application

• CTR estimation for arbitrary (page, ad) pairs

• Features :

– Bag-of-words in query, ads; relevance scores from IR

– Cross-product of words: Occurs in both page and ad

• Learn to predict click data using such features

• Prediction function amenable to WAND algorithm

– Helps with fast retrieval at serve time

Proposed Method

• A logistic regression method model for CTR

CTR Main effect for page

(how good is the page)

Main effect for ad

(how good is the ad)

Interaction effect

(words shared by page and ad)

Model parameters

Proposed Method

• Mp,w = tfp,w

• Ma,w = tfa,w

• Ip,a,w = tfp,w * tfa,w

• So, IR-based term frequency measures are taken into account

Proposed Method

• Two sources of complexity

– Adding in IR scores

– Word selection for efficient learning

Proposed Method

• How can IR scores fit into the model?

– What is the relationship between logit(pij) and cosine score?

– Quadratic relationship

Cosine scorelo

Proposed Method

• How can IR scores fit into the model?

• This quadratic relationship can be used in two ways

– Put in cosine and cosine2 as features

– Use it as a prior

Proposed Method

• Word selection

– Overall, nearly 110k words in corpus

– Learning parameters for each word would be:• Very expensive

• Require a huge amount of data

• Suffer from diminishing returns

– So we want to select ~1k top words which will have the most impact

Proposed Method

• Word selection

– Data based:• Define an interaction measure for each word

• Higher values for words which have higher-than-expected CTR when they occur on both page and ad

Experiments

Recall

25% lift in precision at 10% recall

Overview

• Case Studies

• Online Learning

Regelsen and Fain, 2006

• Estimate CTR of terms by “borrowing strength” at multiple resolutions

• Hierarchical clustering of related terms

– Clustering advertiser keyword matrix

• Estimating CTR at finer resolutions by using information at coarser resolutions

– Weighted average, more weight to finer resolutions

– Weights selected heuristically, no principled approach

Estimation in the “tail”

• A more principled approach to “Estimating Rates of Rare Events at Multiple Resolutions” [KDD/2007]

• Contextual Advertising

– Show an ad on a webpage (“impression”)

– Revenue is generated if a user clicks

– Problem: Estimate the click-through rate (CTR) of an ad on a page

• Most (ad, page) pairs have very few impressions, if any,

• and even fewer clicks

Severe data sparsity

Estimation in the “tail”

• Use an existing, well-understood hierarchy

– Categorize ads and webpages to leaves of the hierarchy

– CTR estimates of siblings are correlated

The hierarchy allows us to aggregate data

• Coarser resolutions

– provide reliable estimates for rare events

– which then influences estimation at finer resolutions

System overview

Retrospective data[URL, ad, isClicked]

Crawl URLs

Classify pages and ads

Rare event estimation using

hierarchy

a sample of URLs

Impute impressions, fix sampling bias

Sampling of webpages

• Naïve strategy: sample at random from the set of URLsSampling errors in impression volume AND click

volume

• Instead, we propose:

– Crawling all URLs with at least one click, and

– a sample of the remaining URLs

Variability is only in impression volume

Imputation of impression volume

Ad classes

sums to #impressions on ads of this ad class

[column constraint]

sums to ∑nij + K.∑mij

[row constraint]

sums toTotal impressions

(known)

#impressions = nij + mij + xij

Clicked pool

Sampled Non-clicked

Excess impressions(to be imputed)

Level 0

Level i

Page hierarchy Ad hierarchy

• Region= (page node, ad node)

• Region Hierarchy A cross-product of the page

hierarchy and the ad hierarchy

Page classes Ad classes

Region

sums to

[block constraint]

Level i

Level i+1

Imputing xij

Level i

Level i+1

Iterative Proportional Fitting [Darroch+/1972]

• Initialize xij = nij + mij

• Iteratively scale xij values to match row/col/block constraint

• Ordering of constraints: top-down, then bottom-up, and repeat

blockPage classes Ad classes

Imputation: Summary

• Given

– nij (impressions in clicked pool)

– mij (impressions in sampled non-clicked pool)

– # impressions on ads of each ad class in the ad hierarchy

• We get

– Estimated impression volume Ñij = nij + mij + xij

in each region ij of every level

System overview

Retrospective data[page, ad, isclicked]

Crawl Pages

Classify pages and ads

Rare event estimation using

hierarchy

a sample of pages

Impute impressions, fix sampling bias

Rare rate modeling

1. Freeman-Tukey transform: – yij = F-T(clicks and impressions at ij)

≈ transformed-CTR

– Variance stabilizing transformation: Var(y) is independent of E[y] needed in further modeling

SijSparent(ij)

Rare rate modeling

2. Generative Model (Tree-structured Markov Model)

yij yparent(ij)

covariates βij variance Vij

Unobserved “state”

variance Wij

Vparent(ij)

βparent(ij)

Wparent(ij)

Rare rate modeling

• Model fitting with a 2-pass Kalman filter:

– Filtering: Leaf to root

– Smoothing: Root to leaf

• Linear in thenumber of regions

Tree-structured Markov model

rVrWRootWRootSrpaSrwrWNrw

rwrpaSrS

rSd(r).rd

rVrSrdT

/on Depends :Smoothing

.0;)( indep );,0(~)(

smoothing) (requireregion per one effects, random: levelat covariatesfor t vector coefficien:)(

),)((~

Model Markov

/),Corr( ;)(

WWllWSVar

Scalable Model fitting Multi-resolution Kalman filter

1994) Rubin, and(Liu algorithm ECME :componets Variance

downtree , Uptree:steps Two

n computatio )3regionchildren O(# region,parent each At

regionsparent ofnumber on Dependsregions ofnumber in thelinear y"essentiall" Algorithm

2002) Cressie, and (Huang algorithmfilter Kalman -

smoothingfiltering

}r{S states ofPosterior

Multi-Resolution Kalman filter: Mathematical overview

parent from info usingchildren on info Update

parentfor available info recombine

children fromn informatio Combine

parentfor child ofon contributiCollect

);1)(),(( ;)(

equations; state Invert the

updatesBayesian standard using nodes leaf ofposterior Update

step Smoothing

:step Filtering

downtree

rdrdcorrrBrrSrBrpa

uptree

Experiments

• 503M impressions

• 7-level hierarchy of which the top 3 levels were used

• Zero clicks in

– 76% regions in level 2

– 95% regions in level 3

• Full dataset DFULL, and a 2/3 sample DSAMPLE

Experiments

• Estimate CTRs for all regions R in level 3 with zero clicks in DSAMPLE

• Some of these regions R>0 get clicks in DFULL

• A good model should predict higher CTRs for R>0 as against the other regions in R

Experiments

• We compared 4 models

– TS: our tree-structured model

– LM (level-mean): each level smoothed independently

– NS (no smoothing): CTR proportional to 1/Ñ

– Random: Assuming |R>0| is given, randomly predict the membership of R>0 out of R

Experiments

Enough impressions little “borrowing”

from siblings

Few impressions Estimates depend more on siblings

Related Work

• Multi-resolution modeling

– studied in time series modeling and spatial statistics [Openshaw+/79, Cressie/90, Chou+/94]

• Imputation

– studied in statistics [Darroch+/1972]

• Application of such models to estimation of such rare events (rates of ~10-3) is novel

Summary

• A method to estimate

– rates of extremely rare events

– at multiple resolutions

– under severe sparsity constraints

• The method has two parts

– Imputation incorporates hierarchy, fixes sampling bias

– Tree-structured generative model extremely fast parameter fitting

Overview

• Case Studies

• Online Learning

Collaborative Filtering

• Collaborative filtering– Similarity based methods

ijujiNj

ijuisrsr

Rating (CTR) for query u of ad i

Ad-ad similarity matrix

Local neighborhood of ad i

Collaborative Filtering

• Collaborative filtering– Similarity based methods

– Possible adaptation

– Challenges: • Learning similarity

• Simultaneously incorporating query and ad similarities

ijujiNj

ijuisrsr

);,()(odds-log

aNjqjqj

aNjqjqa

zfp θaq

Feature-based model

Collaborative filtering model

Matrix Factorization

• Matrix Factorization– Each query (ad) is a linear

combination of latent factors

– Solve for factors, under someregularization and constraints

kakqkqa

)()(odds-log θa;q,

Factor coefficients for query

Factor coefficients

for ad

Matrix Factorization

• Matrix Factorization

• Bi-clustering

– Predictive Discrete latent factor models, Agarwal and Merugu, KDD 07.

kakqkqa

)()(odds-log θa;q,

cluster ad:(a)cluster;Query :)(

);,()(odds-log)(),(

zaqfpaqqa

Overview

• Case Studies

• Online Learning

Challenges of Feature-based models

• Learns from clicks but still misses context in many instances as in relevance based approach

• Introducing features that are too granular makes it hard to learn CTR reliably

• Does not capture the dynamics of the system

• Training cost is high

• Slow prediction functions inadmissible due to latency constraints

• Other methods– Boosting, Neural nets, Decision Trees, Random Forests, ……

• Local models– Mixture of experts: Fit local, think global

• Hierarchical modeling with multiple trees– User interest, query, ad,..

– Each tree is different

– How to perform smoothing with multiple disparate trees?

A)Q,|click(A)Q,|P(clickk

• Combining cold start with warm start together main challenge in collaborative filtering based methods

• We believe, solving basic issues more challenging

– Positional bias

– Selection bias

– Correlation in ads on a slate

– Dynamic CTR; seasonal variations

Online learning

Overview

• Online Learning

Online learning for ad matching

• All previous approaches learn from historical data

• This has several drawbacks:

– Slow response to emerging patterns in the data• due to special events like elections, …

– Initial systemic biases are never corrected• If the system has never shown “sound system dock” ads

for the “iPod” query, it can never learn if this match is good

– System needs to be retrained periodically

• Solution: Combining exploitation with exploration

– Exploitation: Pick ads that are good according to current model

– Exploration: Pick ads that increase our knowledge about the entire space of ads

• Multi-armed bandits

– Background

– Applications to online advertising

Background: Bandits

Bandit “arms”

p1 p2 p3(unknown payoff

probabilities)

• “Pulling” arm i yields a reward:

• reward = 1 with probability pi (success)

• reward = 0 otherwise (failure)

Background: Bandits

• Goal: Pull arms sequentially so as to maximize the total expected reward

– Estimate payoff probabilities pi

– Bias the estimation process towards better arms

Bandit “arms”

p1 p2 p3(unknown payoff

probabilities)

Background: Bandits

• An algorithm to sequentially pick the arms is called a bandit policy

• Regret of a policy = how much extra payoff could be gained in expectation if the best arm is always pulled

– Of course, the best arm is not known to the policy

– Hence, the regret is the price of exploration

– Low regret implies that the policy quickly converges to the best arm

• What is the optimal policy?

Background: Bandits

• Which arm should be pulled next?– Not necessarily what looks best right now, since it might have

had a few lucky successes

– Seems to depend on some complicated function of the successes and failures of all arms

argmax g(s1, f1, s2, f2, …, sk, fk) ?

Number of successes

Number of failures

Background: Bandits

• Consider a bandit which

– has an infinite time horizon, but

– future rewards are geometrically discountedRtotal = R(1) + γ.R(2) + γ2.R(3) + … (0<γ<1)

• Theorem [Gittins/1979]: The optimal policy decouples and solves a bandit problem for each arm independently

argmax {g1(s1, f1), g2(s2, f2), …, gk(sk, fk)}

argmax g(s1, f1, s2, f2, …, sk, fk) ?

Background: Bandits

• Theorem [Gittins/1979]: The optimal policy decouples and solves a bandit problem for each arm independently

– Significantly reduces the dimension of the problem space

– Gives a minimum regret bound of O(log T)

– But, the optimal functions gi(si, fi) are hard to compute

– Need approximate methods…

Background: Bandits

Bandit Policy

1. Assign priority to each arm

2. “Pull” arm with max priority, and observe reward

3. Update priorities

Priority 1

Priority 2

Priority 3

Allocation

Estimation

Background: Bandits

• One common policy is UCB1 [Auer/2002]

Number of successes

Number of failures

Total number of observations

Number of observations of

Observed payoff

Factor representing uncertainty

Background: Bandits

• As total observations T becomes large:

– Observed payoff tends asymptotically towards the true payoff probability

– The system never completely “converges” to one best arm; only the rate of exploration tends to zero

Observed payoff

Background: Bandits

• Sub-optimal arms are pulled O(log T) times

• Hence, UCB1 has O(log T) regret

• This is the lowest possible regret

Observed payoff

– Background

Background: BanditsW

Bandit “arms”

~106 ads

~109 pages

Background: Bandits

Content Match = A matrix

• Each row is a bandit

• Each cell has an unknown CTR

One bandit

Unknown CTR

Background: Bandits

Why not simply apply a bandit policy directly to our problem?

• Convergence is too slow ~109 bandits, with ~106 arms per bandit

• Additional structure is available, that can help Taxonomies

Taxonomies for dimensionality reduction

Apparel Computers Travel

• Already exist

• Actively maintained

• Existing classifiers to map pages and ads to taxonomy nodes

Page/Ad

A bandit policy that uses this structure can be faster

Outline

Multi-level Bandit Policy for Content Match

• Experiments

• Summary

Multi-level Policy

Webpages

… …

……

classes

Consider only two levels

Multi-level Policy

ApparelCompu-

ters Travel

… …

……

Consider only two levels

Ad parent classes

Ad child classes

One bandit

Multi-level Policy

ApparelCompu-

ters Travel

… …

……

Key idea: CTRs in a block are homogeneous

Ad parent classes

One bandit

l Ad child classes

Multi-level Policy

• CTRs in a block are homogeneous

– Used in allocation (picking ad for each new page)

– Used in estimation (updating priorities after each observation)

Multi-level Policy

Used in allocation (picking ad for each new page)

– Used in estimation (updating priorities after each observation)

Multi-level Policy (Allocation)

Page classifier

• Classify webpage page class, parent page class

• Run bandit on ad parent classes pick one ad parent class

• Classify webpage page class, parent page class

• Run bandit on ad parent classes pick one ad parent class

• Run bandit among cells pick one ad class

• In general, continue from root to leaf final ad

Page classifier

Bandits at higher levels

• use aggregated information

• have fewer bandit arms Quickly figure out the best ad parent class

Page classifier

Multi-level Policy

Used in allocation (picking ad for each new page)

Used in estimation (updating priorities after each observation)

Multi-level Policy (Estimation)

– Observations from one cell also give information about others in the block

– How can we model this dependence?

• Shrinkage Model

Scell | CTRcell ~ Bin (Ncell, CTRcell)

CTRcell ~ Beta (Paramsblock)

# clicks in cell# impressions

in cell

All cells in a block come from the same distribution

• Intuitively, this leads to shrinkage of cell CTRs towards block CTRs

E[CTR] = α.Priorblock + (1-α).Scell/Ncell

Estimated CTR

Beta prior (“block CTR”)

Observed CTR

Experiments

20 nodes

221 nodes…

~7000 leaves

Taxonomy structure

We use these 2 levels

Depth 0

Depth 7

Depth 1

Depth 2

Experiments

• Data collected over a 1 day period

• Collected from only one server, under some other ad-matching rules (not our bandit)

• ~229M impressions

• CTR values have been linearly transformed for purposes of confidentiality

Experiments (Multi-level Policy)

Multi-level gives much higher #clicks

Number of pulls

Experiments (Multi-level Policy)

Multi-level gives much better Mean-Squared Error it has learnt more from its explorations

Number of pulls

Experiments (Shrinkage)

Number of pulls Number of pullsMea

ks without shrinkage

with shrinkage

Shrinkage improved Mean-Squared Error, but no gain in #clicks

Summary

• Taxonomies exist for many datasets

• They can be used for

– Dimensionality Reduction

– Multi-level bandit policy higher #clicks

– Better estimation via shrinkage models better MSE

– Background

Challenges and Open Problems

• Bandit policies typically assume stationarity

• But, sudden changes are the norm in the online advertising world:

– Ads may be suddenly removed when they run out of budget

– New ads are constantly added to the system

– The total number of ads is huge, and full exploration may be too costly

– Mortal multi-armed bandits [NIPS/2008]

Mortal Multi-armed Bandits

• Traditional bandit policies like UCB1 spend a large fraction of their initial pulls on exploration

– Hard-earned knowledge may be lost due to finite arm lifetimes

• Method 1 (Sampling):

– Pick a random sample from the set of available arms

– Run UCB1 on sample, until some fraction of arms in the sample are lost

– Pro: Quicker convergence, more exploitation

– Con: Best arm in the sample may be worse than best arm overall

– Pick sample size to control this tradeoff

• Traditional bandit policies like UCB1 spend a large fraction of their initial pulls on exploration

– Hard-earned knowledge may be lost due to finite arm lifetimes

• Method 2 (Payoff threshold):

– New bandit policy: If the observed payoff of any arm is higher than a threshold, pull it till it expires

– Pro: Good arms, once found, are exploited quickly

– Con: While exploiting good arms, the best arm may be starving and may expire without being found

– Pick threshold to control this tradeoff

• Challenges:

– Selecting the critical sample size or threshold correctly, for arbitrary payoff distributions

– What if even the payoff distribution is unknown?

Challenges and Open Problems

• Mortal multi-armed bandits

• What if the bandit policy has some information about the budget?

– The bandit policy can control which arms expire, and when

– “Handling Advertisements of Unknown Quality in Search Advertising” by Pandey+/NIPS/2006

• Combining budgets with extra knowledge of ad CTRs

– E.g., Using an ad taxonomy

• Using a bandit scheme to infer/correct an ad taxonomy

Conclusions

• We provided an introduction to Online Advertising

– Discussed the eco-system and various actors involved

– Discussed different flavors of online advertising• Sponsored Search, Content Match, Display Advertising

Conclusions

Online Advertising

Revenue Models

Advertising Setting

CPM CPC CPA

Sponsored Search

Ad exchanges

Conclusions

• Outlined associated statistical challenges

– Sponsored search, Content Match, Display

• We believe the following to be a technical roadmap

Offline Modeling Online ModelsTime series

Explore/Exploit

Multi-armed bandits

Regression, collaborative filtering, mixture of experts

Multi-resolution models

Selection bias Slate correlation

Noisy labels

Conclusions

• Offline Modeling– By far the best studied so far

– Not a careful study of selection bias, slate correlations, noisy labels. Good opportunity here

– More emphasis on matrix structure, goal is to estimate interactions

• Explore/Exploit– Some work using multi-armed bandits; long way to go

• Time series model to capture temporal aspects– Little work

• Holistic approach that combines all components in a principled way

research © 2008 yahoo! statistical challenges in online advertising deepak agarwal deepayan...

Documents

rupa chakrabarti - econstor.eu

yamacraw-yahoo falls backcountry route yahoo …...yahoo...

chakrabarti palki - jini network technology

deepayan chakrabarticikm 20021 f4: large scale automated...

graph mining: laws, generators and algorithms · graph...

arpe chakrabarti publication

review on pupils - neurological approach - by deepayan kar

rajesh chakrabarti bihar break through

deepayan founder

graph mining: laws, generators and...

chakrabarti 2000, puranas and bengal

1 tools for large graph mining thesis committee: christos...

basic electrical engineering by chakrabarti

bandits for taxonomies: a model-based approach sandeep...

dansatoarea srinwanti chakrabarti

cellular automata by devdutta chakrabarti

chakrabarti upcea 2011_rev_2_ (2)

features. yahoo! features my yahoo! flickr delicious yahoo!...

1 clustering applications at yahoo! deepayan chakrabarti...

nonparametric link prediction in dynamic graphs purnamrita...