developing real time bidding solutions with rtbkit webinar on demand

Open-Source RTB for Everyone

Developing Real Time Bidding Solutions with

© Datacratic Inc. 2014 All Rights Reserved.

Overview

● The Project● The Problems With RTB

o Systemo Selectiono Value

● RTB Competitive Landscape

● How RTBkit Addresses the Problems with RTB

● Demo© Datacratic Inc. 2014 All Rights Reserved.

A Little Bit About

A Little History

● Created by machine-learning and digital marketing company Datacratic

● Code base evolved from running RTB in production from 2011-2013

● Open sourced in Feb. 2013, with ongoing support from Datacratic

● Apache-style governance started Jan. 2014


Participation and Governance

Participation and Governance● Apache-style governance

o BDNFL - Benevolent Dictator Not for Life

o Councillorso Committers

● Outside contributions welcome● Github pull request workflow --

committers review and merge● Contributor guidelines● Users can become

Contributors● Contributors can become

Committers -- currently two outside Committers


Support, Community, Adoption

Support and Community● Free support from the

community and Datacratic● 230 active developers● 35 committers, 11 outside of

Datacratic● Users in 25+ countries● ~15 installations in prod: N.

America, Germany, France, Russia, Argentina, China

● Datacratic supports development, code review, governance and evolution

● Participation and contributions from Rubicon Project


Development Support

Development Support● Getting Started Guide● Developer Tutorials● Working integration test system

o mock Exchange and Ad Servero fixed-price Bidding Agent

● Example Code● Documentation● Ubuntu AMI● Google Group support● Pull request review and support


The Problems With RTB

SYSTEM VALUESELECTION

Provided by RTBkit Customized by User

General/Technical Specific/Business


Solves the RTB System Problem

SYSTEM SELECTION VALUE

ScaleSpeed

Distribution

Reliability


Provided by RTBkit Customized by User© Datacratic Inc. 2014 All Rights Reserved.

Addresses the RTB Selection Problem


ScaleSpeed Show user an ad?

What ad?

Distribution

Reliability

Provided by RTBkit Customized by User



Addresses the RTB Value Problem


ScaleSpeed

Distribution

Show user an ad?

What ad?

What is it worth?

What should I pay?Reliability


Provided by RTBkit Customized by User© Datacratic Inc. 2014 All Rights Reserved.

RTB Technology LandscapeSystem Pros Cons Degree of

Difficulty

Exchange / DSP UI

Easy to get started ● Manual, hard to scale● Lack of control over bidding

strategy and data

Low

Intermediate Hosted Bidding

● More control over bidding strategy and use of data

● Don't have to do Ops

Strategy and use of data mediated by vendor and product features

Medium

Roll Your Own Bidder

Full control of all aspects of the system

Solely responsible for everything Hardest

● Benefit from core problems being solved

● Benefit from● community● Flexible customization

● Full control (optionally) but requires digging in

● Responsible for ops

Hard


Addresses the System ProblemHo

w

Architectural OverviewRTBkit Core

● Router● Banker● Post Auction Service● Service Monitor● Agent Configuration Service

Plugins● Exchange Connectors● AdServer Connector● Bidding Agent● Augmenter● Logger


Bidder Core Advantages● Core working bidder system● High-performance real-time components● Multiple data center support● Reliable global banker● Guaranteed response time to exchanges● High-performance parsing, routing, filtering,

logging and monitoring● Arbitrary use of custom business logic and

your data


Inventory Integration● Ships with 9 Exchange Connectors

o Rubicon Projecto AdXo FBXo AppNexuso Nexageo MoPubo GumGumo BidSwitch

● OpenRTB 2.1 compatible


Router ResponsibilitiesRouter Responsibilities

● Gets bid requests from Exchange Connector

● Uses Filters to filter eligible campaigns● Passes bid requests through Augmenter● Passes bid requests to Bidding Agents to

generate bid responses● Communicates with Banker to guarantee

no overspend● Guarantees timely response


Router ComponentsRTBkit

Exchange Connector

Static Filters

Augmentation Loop

Dynamic Filters

Auction Loop

Slave Banker

Master Banker

Bidding Agents

Augmenter

Post Auction Service

Agent Config

RTBkitRouter

Exchanges


Router Data FlowsRTBkitRouter

Exchanges Exchange Connector

Static Filters

Augmentation Loop

Dynamic Filters

Auction Loop

Slave Banker

Master Banker

Bidding Agents

Augmenter


Agent Config


Bid Request Lifecycle

RTBkitRouter

Exchange Connector

Static Filters

Dynamic FiltersAugmenter


Ad Server

Conversion Source

Bidding Agents

Ad Server Connector

Exchanges


RTBkit Data FlowsFive asynchronous data

flows flow through the Router:

● Bid request processing ● Banking updates● Event Matching● Notifying Bidding Agents

of Events● Filtering and Bidding

Agent configuration

RTBkitRouterExchange Connector

Static Filters

Augmentation Loop

Dynamic Filters

Auction Loop

Slave Banker

Master Banker

Bidding Agents

Augmenter


Agent Config

Exchanges


Ad Server and Conversion Integration

● Standard HTTP JSON connector for receiving Wins, Clicks and Conversions

● Passes all events to PAS for matching● Logs events and matched events


Post Auction Service● Clearinghouse for matching all bids to

Wins, Clicks and Conversions

● Current bottleneck, can process events in the hundreds / sec

● Recently improved: sharded hash tables, one thread per core

● More improvements on the near roadmapPost

Auction Service

Router Bids

Ad ServerConnector

Events

Shadow Account

Bidding Agents

LoggerMatched Events

Matched Events

Wins andInferred Losses


Banker Responsibilities and DesignBudget Management

● Single source of truth for budget● Authorizes and caps spendingDesigned for Reliability

● Designed for high-latency, low-bandwidth ● Insulates banker state by using "shadow

account" bookkeeping● Totals always go up -- you can always

reason about the relative timing of entries● Updates global state once per minute● Atomic, idempotent persistenceCurrency Support

● Strongly typed 64-bit int Currency, won't allow cross-currency conversions

● Automatic scaling conversions (e.g. CPM to micro-dollars)


Banker Account Types● Budget Account

o All budget for Account tree set in Master Banker Budget Account

o Cannot bid from this accounto Can transfer budget to child Spend

Accountso Only exists in the Master Banker

● Spend Accounto Must have a parent Budget Accounto Can bid from this accounto Can track spend directlyo Cannot have childreno Shadowed into separate processes

Budget Account

Spend Account

Master Banker

Spend Account


Account Hierarchies and Spend Tracking

● Spend accumulates from Children to Parents● Temporary bookkeeping for bids happens in

shadow accounts in separate processes● Shadows sync once per second

o Router shadow - tracks budget committed to pending bids

o PAS shadow - tracks budget to debit on Wins and to credit on Losses

BudgetA

SpendA:B

Master Banker

SpendA:C

Shadow Spend

A:B

Router Slave Banker

Shadow Spend

A:C

Shadow Spend

A:C

PAS Slave Banker

Shadow Spend

A:B


Banker APIs● REST API suitable for human reader and

outside tool integration● Also used by Router, Post Auction Service

and Bidding Agents● API wraps Account Type, Account

Hierarchy and Currency Pool concepts


Monitoring and Operations Tools● Extensive instrumentation● Lock-free, high-performance Carbon

logging library, with tunable sampling rate, one-second granularity and useful aggregate functions

● Can add any custom metrics● Operational dashboard included


Monitoring and Operations Tools● All standard and custom metrics

charted in graphite


Addresses the Selection ProblemHow

Filter Design and Features● Router passes bid requests through

Filtering pipeline● Bids must pass all filters to reach Agents● Thread safe● Useful primitives for custom filters● Predefined Agent and Creative filters● Designed to guarantee performance first,

be flexible and powerful second


Filter Levels● Agent Filters

o Control whether Agent bids on bid request

● Creative Filterso Control whether Creative eligible to

be the one returned in bid response

Agent

FormatLocation

ExchangeLanguage

Creative

Exchange

Location

Language

Host

URL

Segments

Hour of Week

Fold Position

User Partition


Filter Priorities and PerformanceFilter Priorities and Performance

● Execution order optimized for performance● Broad, inexpensive filters run first● Expensive filters run later (or not at all!)● Only fast filters on real-time thread● Static filters support batch processing● Filter matching tests match and retrieves

eligible creatives in one pass

FormatLocation

ExchangeLanguage

Creative

Agent

Exchange

Location

Language

Host

URL

Segments

Hour of Week

Fold Position

User Partition

Agent

Bid Request

Bid RequestBid

RequestBid Request


Augmenters: Your Logic and DataAugmenters: Filter Based on Your Logic

and Data● Bid Requests pass through Augmenter

after Filtering, before Bidding● Custom filtering based on combinations of

bid request fields, your data, etc.


Augmenter Implementation● Provides thread pool● Enforces 5ms timeout● Sync and async support● Ships with Redis Augmenter

● Separate config for each Bidding Agent● Can subscribe to other RTBkit data

streams to write datao e.g. - frequency cap Augmenter

subscribes to PAS MATCHEDWINs

Router

Bid Request

Wicked Fast DB TM

Thread Pool

Augmenter Impl.

Augmenter


Data Sink Callback

Augmented Bid

Request


Addresses the Value ProblemHow

Bidding Agent Configuration

● Bidding Agents configure the Router, PAS and Augmentero creatives per campaigno dynamic filterso augmenters and augmenter filterso ad markup


Custom Bidding Agent Implementation

● C++ or JavaScript● Programmatic configuration● Custom bidding logic based on

o bid attributeso a custom Win Cost Model to adjust

for desired margin and data costs● Custom Pacing logic supportedCustom Bidding

Agent

ConfigurationBidding Logic

Win Cost ModelPacing

Currency Helpers

Router

Bid Request1Bid Response

2

onWinonLossonNoBudgetonTooLateonDroppedBidonInvalidBid

3


onImpressiononClickonVisit 4


Augmenters: Your Logic and DataAugmenters: Modify Bid Requests Based

on Your Logic and Data● Allows you to augment the bid request,

adding any fields you want● Augmented fields are then available to the

Bidding Agents


Future Directions

Already Delivered in 2014● PAS horizontal scalability

Improved Packaging● HTTP Bidding Agent APIThis Year

● “RTBkit 2.0” support for HTTP plugin APIs for● Augmenter Data ● Loggers● Analytic Tools

● Support third-party marketplace for data, analytics, bid strategy, etc.


RTBkit Resources

Links● http://github.com/rtbkit● http://rtbkit.org● https://groups.google.com/a/rtbkit.

org/forum/#!forum/discuss

Developers Getting Started Guide● https://github.com/rtbkit/rtbkit/wiki/

Getting-Started


http://github.com/rtbkit

http://rtbkit.org/

https://groups.google.com/a/rtbkit.org/forum/

https://groups.google.com/a/rtbkit.org/forum/

https://github.com/rtbkit/rtbkit/wiki/Getting-Started

https://github.com/rtbkit/rtbkit/wiki/Getting-Started

About Datacratic• Datacratic is an enterprise software company

that develops machine learning and artificial intelligence technology, which enables real-time machine based decisioning to be deployed into a wide range of applications.

• Datacratic’s products are currently optimized for Data Management Platforms (DMPs), Demand Side Platforms (DSPs), Agency Trading Desks (ATDs), E-Commerce web sites and others in the real-time marketing ecosystem.

Mark Weiss @marksweiss● Head of Customer Solutions at Datacratic

www.datacratic.com

opensource.datacratic.com/

We're hiring! datacratic.com/site/careers


http://www.datacratic.com/

http://opensource.datacratic.com/

http://datacratic.com/site/careers

You are Invited! RTBkit Meetup September 4th, London, UK

Who should attend?Developers or engineers with interested in Ad Technology & Real-Time bidding.

Hosted by:

Presenters:James Prudhomme, CEO Datacratic & Miguel Sempere, Data Engineer Affectv