developing real time bidding solutions with rtbkit webinar on demand

Open-Source RTB for Everyone

Developing Real Time Bidding Solutions with

Overview

● The Project● The Problems With RTB

o Systemo Selectiono Value

● RTB Competitive Landscape

● How RTBkit Addresses the Problems with RTB

A Little Bit About

A Little History

● Created by machine-learning and digital marketing company Datacratic

● Code base evolved from running RTB in production from 2011-2013

● Open sourced in Feb. 2013, with ongoing support from Datacratic

● Apache-style governance started Jan. 2014

Participation and Governance

Participation and Governance● Apache-style governance

o BDNFL - Benevolent Dictator Not for Life

o Councillorso Committers

● Outside contributions welcome● Github pull request workflow --

committers review and merge● Contributor guidelines● Users can become

Contributors● Contributors can become

Committers -- currently two outside Committers

Support, Community, Adoption

Support and Community● Free support from the

community and Datacratic● 230 active developers● 35 committers, 11 outside of

Datacratic● Users in 25+ countries● ~15 installations in prod: N.

America, Germany, France, Russia, Argentina, China

● Datacratic supports development, code review, governance and evolution

● Participation and contributions from Rubicon Project

Development Support

Development Support● Getting Started Guide● Developer Tutorials● Working integration test system

o mock Exchange and Ad Servero fixed-price Bidding Agent

● Example Code● Documentation● Ubuntu AMI● Google Group support● Pull request review and support

The Problems With RTB

SYSTEM VALUESELECTION

Provided by RTBkit Customized by User

General/Technical Specific/Business

Solves the RTB System Problem

SYSTEM SELECTION VALUE

ScaleSpeed

Distribution

Reliability

Addresses the RTB Selection Problem

ScaleSpeed Show user an ad?

What ad?

Distribution

Reliability

Provided by RTBkit Customized by User

Addresses the RTB Value Problem

ScaleSpeed

Distribution

Show user an ad?

What ad?

What is it worth?

What should I pay?Reliability

RTB Technology LandscapeSystem Pros Cons Degree of

Difficulty

Exchange / DSP UI

Easy to get started ● Manual, hard to scale● Lack of control over bidding

strategy and data

Intermediate Hosted Bidding

● More control over bidding strategy and use of data

● Don't have to do Ops

Strategy and use of data mediated by vendor and product features

Medium

Roll Your Own Bidder

Full control of all aspects of the system

Solely responsible for everything Hardest

● Benefit from core problems being solved

● Benefit from● community● Flexible customization

● Full control (optionally) but requires digging in

● Responsible for ops

Addresses the System ProblemHo

Architectural OverviewRTBkit Core

● Router● Banker● Post Auction Service● Service Monitor● Agent Configuration Service

Plugins● Exchange Connectors● AdServer Connector● Bidding Agent● Augmenter● Logger

Bidder Core Advantages● Core working bidder system● High-performance real-time components● Multiple data center support● Reliable global banker● Guaranteed response time to exchanges● High-performance parsing, routing, filtering,

logging and monitoring● Arbitrary use of custom business logic and

your data

Inventory Integration● Ships with 9 Exchange Connectors

o Rubicon Projecto AdXo FBXo AppNexuso Nexageo MoPubo GumGumo BidSwitch

● OpenRTB 2.1 compatible

Router ResponsibilitiesRouter Responsibilities

● Gets bid requests from Exchange Connector

● Uses Filters to filter eligible campaigns● Passes bid requests through Augmenter● Passes bid requests to Bidding Agents to

generate bid responses● Communicates with Banker to guarantee

no overspend● Guarantees timely response

Router ComponentsRTBkit

Exchange Connector

Static Filters

Augmentation Loop

Dynamic Filters

Auction Loop

Slave Banker

Master Banker

Bidding Agents

Augmenter

Post Auction Service

Agent Config

RTBkitRouter

Exchanges

Router Data FlowsRTBkitRouter

Exchanges Exchange Connector

Static Filters

Augmentation Loop

Dynamic Filters

Auction Loop

Slave Banker

Master Banker

Bidding Agents

Augmenter

Agent Config

Bid Request Lifecycle

RTBkitRouter

Exchange Connector

Static Filters

Dynamic FiltersAugmenter

Ad Server

Conversion Source

Bidding Agents

Ad Server Connector

Exchanges

RTBkit Data FlowsFive asynchronous data

flows flow through the Router:

● Bid request processing ● Banking updates● Event Matching● Notifying Bidding Agents

of Events● Filtering and Bidding

Agent configuration

RTBkitRouterExchange Connector

Static Filters

Augmentation Loop

Dynamic Filters

Auction Loop

Slave Banker

Master Banker

Bidding Agents

Augmenter

Agent Config

Exchanges

Ad Server and Conversion Integration

● Standard HTTP JSON connector for receiving Wins, Clicks and Conversions

● Passes all events to PAS for matching● Logs events and matched events

Post Auction Service● Clearinghouse for matching all bids to

Wins, Clicks and Conversions

● Current bottleneck, can process events in the hundreds / sec

● Recently improved: sharded hash tables, one thread per core

● More improvements on the near roadmapPost

Auction Service

Router Bids

Ad ServerConnector

Events

Shadow Account

Bidding Agents

LoggerMatched Events

Matched Events

Wins andInferred Losses

Banker Responsibilities and DesignBudget Management

● Single source of truth for budget● Authorizes and caps spendingDesigned for Reliability

● Designed for high-latency, low-bandwidth ● Insulates banker state by using "shadow

account" bookkeeping● Totals always go up -- you can always

reason about the relative timing of entries● Updates global state once per minute● Atomic, idempotent persistenceCurrency Support

● Strongly typed 64-bit int Currency, won't allow cross-currency conversions

● Automatic scaling conversions (e.g. CPM to micro-dollars)

Banker Account Types● Budget Account

o All budget for Account tree set in Master Banker Budget Account

o Cannot bid from this accounto Can transfer budget to child Spend

Accountso Only exists in the Master Banker

● Spend Accounto Must have a parent Budget Accounto Can bid from this accounto Can track spend directlyo Cannot have childreno Shadowed into separate processes

Budget Account

Spend Account

Master Banker

Spend Account

Account Hierarchies and Spend Tracking

● Spend accumulates from Children to Parents● Temporary bookkeeping for bids happens in

shadow accounts in separate processes● Shadows sync once per second

o Router shadow - tracks budget committed to pending bids

o PAS shadow - tracks budget to debit on Wins and to credit on Losses

BudgetA

SpendA:B

Master Banker

SpendA:C

Shadow Spend

Router Slave Banker

Shadow Spend

PAS Slave Banker

Shadow Spend

Banker APIs● REST API suitable for human reader and

outside tool integration● Also used by Router, Post Auction Service

and Bidding Agents● API wraps Account Type, Account

Hierarchy and Currency Pool concepts

Monitoring and Operations Tools● Extensive instrumentation● Lock-free, high-performance Carbon

logging library, with tunable sampling rate, one-second granularity and useful aggregate functions

● Can add any custom metrics● Operational dashboard included

Monitoring and Operations Tools● All standard and custom metrics

charted in graphite

Addresses the Selection ProblemHow

Filter Design and Features● Router passes bid requests through

Filtering pipeline● Bids must pass all filters to reach Agents● Thread safe● Useful primitives for custom filters● Predefined Agent and Creative filters● Designed to guarantee performance first,

be flexible and powerful second

Filter Levels● Agent Filters

o Control whether Agent bids on bid request

● Creative Filterso Control whether Creative eligible to

be the one returned in bid response

FormatLocation

ExchangeLanguage

Creative

Exchange

Location

Language

Segments

Hour of Week

Fold Position

User Partition

Filter Priorities and PerformanceFilter Priorities and Performance

● Execution order optimized for performance● Broad, inexpensive filters run first● Expensive filters run later (or not at all!)● Only fast filters on real-time thread● Static filters support batch processing● Filter matching tests match and retrieves

eligible creatives in one pass

FormatLocation

ExchangeLanguage

Creative

Exchange

Location

Language

Segments

Hour of Week

Fold Position

User Partition

Bid Request

Bid RequestBid

RequestBid Request

Augmenters: Your Logic and DataAugmenters: Filter Based on Your Logic

and Data● Bid Requests pass through Augmenter

after Filtering, before Bidding● Custom filtering based on combinations of

bid request fields, your data, etc.

Augmenter Implementation● Provides thread pool● Enforces 5ms timeout● Sync and async support● Ships with Redis Augmenter

● Separate config for each Bidding Agent● Can subscribe to other RTBkit data

streams to write datao e.g. - frequency cap Augmenter

subscribes to PAS MATCHEDWINs

Router

Bid Request

Wicked Fast DB TM

Thread Pool

Augmenter Impl.

Augmenter

Data Sink Callback

Augmented Bid

Request

Addresses the Value ProblemHow

Bidding Agent Configuration

● Bidding Agents configure the Router, PAS and Augmentero creatives per campaigno dynamic filterso augmenters and augmenter filterso ad markup

Custom Bidding Agent Implementation

● C++ or JavaScript● Programmatic configuration● Custom bidding logic based on

o bid attributeso a custom Win Cost Model to adjust

for desired margin and data costs● Custom Pacing logic supportedCustom Bidding

ConfigurationBidding Logic

Win Cost ModelPacing

Currency Helpers

Router

Bid Request1Bid Response

onWinonLossonNoBudgetonTooLateonDroppedBidonInvalidBid

onImpressiononClickonVisit 4

Augmenters: Your Logic and DataAugmenters: Modify Bid Requests Based

on Your Logic and Data● Allows you to augment the bid request,

adding any fields you want● Augmented fields are then available to the

Bidding Agents

Future Directions

Already Delivered in 2014● PAS horizontal scalability

Improved Packaging● HTTP Bidding Agent APIThis Year

● “RTBkit 2.0” support for HTTP plugin APIs for● Augmenter Data ● Loggers● Analytic Tools

● Support third-party marketplace for data, analytics, bid strategy, etc.

RTBkit Resources

Links● http://github.com/rtbkit● http://rtbkit.org● https://groups.google.com/a/rtbkit.

org/forum/#!forum/discuss

Developers Getting Started Guide● https://github.com/rtbkit/rtbkit/wiki/

Getting-Started

About Datacratic• Datacratic is an enterprise software company

that develops machine learning and artificial intelligence technology, which enables real-time machine based decisioning to be deployed into a wide range of applications.

• Datacratic’s products are currently optimized for Data Management Platforms (DMPs), Demand Side Platforms (DSPs), Agency Trading Desks (ATDs), E-Commerce web sites and others in the real-time marketing ecosystem.

Mark Weiss @marksweiss● Head of Customer Solutions at Datacratic

www.datacratic.com

opensource.datacratic.com/

We're hiring! datacratic.com/site/careers

You are Invited! RTBkit Meetup September 4th, London, UK

Who should attend?Developers or engineers with interested in Ad Technology & Real-Time bidding.

Hosted by:

Presenters:James Prudhomme, CEO Datacratic & Miguel Sempere, Data Engineer Affectv

developing real time bidding solutions with rtbkit webinar on demand

Technology

beating adwords bidding [webinar]

rtbkit meetup - developer spotlight, behind the scenes of...

chazen japan study tour - columbia business school ·...

bidding toolkit

philippine bidding documents - re-bidding of medical...2...

bidding document (national competitive bidding) (technical

bidding documents

bidding document - eproc.punjab.gov.pk · bidding document...

(e-bidding) (e-bidding) (e-bidding) m uu dauau dmo,ooo.oo...

bidding document

bidding document - punjab,...

bridge improvers lesson 1 review of nt bidding. improvers...

bidding en

health decisions webinar: aca and competitive bidding

nasb bidding

section 4 - bidding forms -...

software solutions for landscape professionals “ bidding...

welcome general steps to… bidding – electronically...

international bidding

table of contents a. bidding and contract ......04...