rtbkit introduction & best practices

Introduction & Best Practices

Open-Source RTB for Everyone

Copyright © 2014 Datacratic Inc. All rights reserved

Overview

• About the RTBkit Project

• Use Cases

• Best Practices

• What’s New in 2.0

• Resources

• Q&A


About the project


A Little History

• RTBKit was created by machine-learning and digital marketing company Datacratic

• Code base evolved from running RTB in production from 2011-present

• Open sourced in Feb. 2013, with ongoing support from Datacratic

• Apache-style governance started Jan. 2014


Participation and Governance

• Apache-style governance

• BDNFL - Benevolent Dictator

Not for Life

• Councillors

• Committers

• Outside contributions welcome

• Github pull request workflow --

committers review and merge

• Contributor guidelines

• Users can become Contributors

• Contributors can become

Committers -- currently two

outside Committers


Development Support

Development Support

• Getting Started Guide

• Developer Tutorials

• Working integration test system– mock Exchange and Ad Server

– fixed-price Bidding Agent

• Example Code

• Documentation

• Ubuntu AMI

• Google Group support

• Pull request review and support


Addresses the RTB Value Problem


SYSTEMSELECTIO

NVALUE

Scale

Speed

Distribution

Show user an ad?

What ad?

What is it worth?

What should I pay?Reliability

GENERAL/TECHNICA

L

SPECIFIC/BUSINESS

PROVIDED BY

RTBKIT

CUSTOMIZED BY

USER

RTB Technology Landscape

System Pros Cons Degree of Difficulty

Exchange / DSP UI

Easy to get started ● Manual, hard to scale● Lack of control over bidding strategy and

data

Low

Intermediate Hosted Bidding

● More control over bidding strategy and use of data

● Don't have to do Ops

Strategy and use of data mediated by vendor and product features

Medium

● Core systems and integration problems solved

● Benefit from community● Flexible customization● Flexible use of data

● Requires significant development● Responsible for operations of a high-

volume distributed system

High

Roll Your Own Bidder

Full control of all aspects of the system Solve all the hard systems problems yourself first, then build the rest of the system you need.

Highest


Use Cases & Best Practices


Use Cases

Use Case RTBkit Benefits

Startup DSP / Targeted Media Company

● Move directly to having your own bidder in first iteration of company technology● Full flexible use of proprietary data to pursue specialized strategy, differentiate and create IP

Established Small to Midsize DSP / Targeted Media Company

● Move from (perhaps multiple) external buying platforms to unified internal platform. Save on fees and ad ops personnel.

● Improved margins● Full flexible use of proprietary data to create additional value for customers

Larger Demand Side Platform / Hosted Service / Integrated Platform

● Real-world example: Rubicon Project● Get reliable, high-performance core bidder functionality without having to build it● Benefit from open source, open architecture and coming plugin ecosystem with RTBkit 2.0


Best Practices: Single Node Deployment

• Recommended single-node

installation on EC2

• Singletons and Real-Time

nodes:

– C3.8xlarge:

– 32 cores, 60GB RAM

– 200GB ESB

– 1000 provisioned IOPS

• Supports about $100,000

monthly media spend

• About $1200 / month non-

reserved, $700 reserved


Best Practices: Multi Node Deployment

• Recommended multi-node installation

on EC2

• Singletons and Real-Time nodes:

– C3.8xlarge:

– 32 cores, 60GB RAM

– 200GB ESB

– 1000 provisioned IOPS

• 1..N Real-Time Nodes can scale

horizontally

• HAProxy load balancing for real-time

nodes

• Aggregating Carbon and Graphite node:

– M3.xlarge

– 2 cores, 3.69GB RAM


Best Practices: Multi Node Multi-DC Deployment

• Singleton and Real-Time nodes

same as Multi-Node setup, per

Data Center

• One HAProxy load balancer for

real-time nodes per DC

• Local Carbon and Graphite

services on each host and one

global aggregating service for all

DCs


Additional RTBkit Best Practices

• Graphite

– {MAX_UPDATES_PER_SECOND} = 20

– Configure with memcached

– Recommended metrics to monitor:

• <installation>.<location>.*.dataLogger.*

• <installation>.<location>.*.postAuction.bidResult.WIN.messagesReceived

• <installation>.<location>.*.postAuction.bidResult.WIN.notInSubmitted

• <installation>.<location>.*.postAuction.delivery.CLICK.delivered

• <installation>.<location>.*.router.bid

• <installation>.<location>.*.router.*.auctionNetworkLatencyMs.mean

• <installation>.<location>.*.router.*.auctionReceived

• <installation>.<location>.*.router.*.auctionStart

• <installation>.<location>.*.router.*.auctionTotalTimeMs.mean

• <installation>.<location>.*.router.monitor.systemInSlowMode

• Redis

– Configure to run as a service

– Configure to dump every 60 seconds to

near-term in memory store and set up a cron

to persist that to DB or long-term storage

every N minutes

• Delete old logs periodically


Additional AWS Best Practices

• Provisioned IOPS for real-time

endpoints and Carbon / Graphite

hosts

• SSD-backed EBS volumes

• HVM (hardware virtual machine)

instance types, not PV

(paravirtual) for real-time

endpoints


2.0 Roadmap Preview


The Big Story in 2.0… Plugins!

• Why Plugins? We want to support

an ecosystem of revenue opportunities

on RTBkit.

• Design supports this goal

– Plugins identified by URL

– Can have multiple versions

– Support multiple RTBkit versions

– Support multiple RTBkit installations

– Requests can be authenticated

– Vendors of public plugins can set up direct

billing


Public Plugin Ecosystem

• Major goal of RTBkit 2.0

• RTBkit users can leverage

third-party Plugins

• Public Plugin requirements:

– Multitenant – one plugin can serve

many RTBkit instances

– Multi-location – one logical plugin

can be deployed in multiple locations

– Authentication and Security

– Per-call metering for billing

– Support multiple RTBkit versions

RTBkit

Plugin

Plugin

Plugin

Plugin

Plugin

Plugin Plugin

Plugin

Plugin Plugin

Plugin

PluginPlugin

Plugin

Plugin

RTBkit

RTBkit

Plugin

Plugin


Plugins – Ecosystem Business Use Cases

• RTBkit users benefit and Vendors benefit:

– Bidder Plugins – campaign management and

bidding logic

– Bid Optimization

– First-party Data in bidding logic

– Per-bid bid price modifier

– First-party Data for attribution

– Third-party Data

– Custom Attribution

– Offline Analytics

– Real-time Analytics

– Real-time Audience Modeling


2.0 Roadmap and Progress

Already Delivered in 2014

• HTTP Bidding Agent APIs – Precursor to full

2.0 API

Up Next – Q4 2014

• First API open source code releases – written

in Go not C++

• RTBkit updates to be compatible with new

APIs

Up Next – Q1 2015

• Data and Logger HTTP APIs for Data and

Analytics Plugins


RTBkit Resources

RTBkit Links

• http://github.com/rtbkit

• http://rtbkit.org

• https://groups.google.com/a/rtbkit.org/forum/#!forum/d

iscuss

• https://github.com/rtbkit/rtbkit/wiki/Getting-Started

Introduction to RTBkit

• http://www.slideshare.net/Datacratic/a-technical-

introduction-to-rtbkit

• http://www.slideshare.net/Datacratic/a-technical-

introduction-to-rt-bkit-edit-45minute?related=1

RTBkit 2.0 Presentation

• http://www.slideshare.net/Datacratic/rtbkit-20-

roadmap-preiew


RTBkit & Rubicon Project Deliver Scalable Ad Tech

Infrastructure & Extensible Bidding Stack

• Rubicon Project is a leading technology company automating the buying and selling of advertising.

• Rubicon Project & RTBkit have combined forces to deliver a scalable ad tech infrastructure,

and makes it easy for media buyers to deploy their own proprietary bidding stack with the open

and extensible architecture of RTBkit.

• The OpenDSP team is a contributor to the RTBkit open source project and active member

of the RTBkit community.


Join the Alpha Program

• There are limited seats available in the alpha program. Contact the OpenDSP team at

[email protected] to get started.

About Datacratic


Datacratic is an enterprise software

company that develops machine

learning

and artificial intelligence technology,

which enables real-time machine based

decisioning to be deployed into

a wide range of applications.Datacratic’s products are currently optimized for:

• Data Management Platforms (DMPs)

• Demand Side Platforms (DSPs)

• Agency Trading Desks (ATDs)

• E-Commerce web sites

• Others in the real-time marketing ecosystem.

Mark Weiss @marksweiss

Head of Customer Solutions at Datacratic

www.datacratic.com

opensource.datacratic.com/

We're hiring! datacratic.com/site/careers

http://www.datacratic.com

http://opensource.datacratic.com/

http://datacratic.com/site/careers

Host an RTBkit Meetup!


Q & A


rtbkit introduction & best practices

Software