tweaking open source

Slide 1

Tweaking Open-Sourcea case-study

Nelson Gomes ([email protected])Team Leader11th of November 2011

Introduction

OpenX Components

Improvements Introduced

Overall Architecture

Report Server

Problems Found

Links

Q&A

Talk Index

Introduction Online Advertisement

Online advertisement coupes with delivering ads;

Placing ads in sites is a complex process:

Obtain all ads electable for a placeholder;

Exclude ads with business limitations like capping;

Assure that the ads are beying presented to the target audience;

Assure the advertiser goals are being met;

Account the ads delivered;

Deliver the right ad format;


Examples


Examples


Examples


Examples

Introduction OpenX

Open Source advertising server;

Licensed under GNU General Public License;

Project forked from phpAds developed by Tobias Ratschiller in 1998;

Was called phpAdsNew, OpenAds and finally OpenX;

Features:

Has a web based GUI;

Extendable plugins architecture;

Serves ads throught JS and Iframes calls mainly;

Introduction OpenX

Support technologies:

PHP;

MySQL;

Web Server (Apache, Nginx);

Optional memcached usage;

Filesystem to serve ad content;

Introduction OpenDisplay

Starting from OpenX 1.8.5 version, SAPO OpenDisplay project began;

A four-person team started in April 2010 to analyse and improve OpenX capabilities to ensure entire SAPO's ad serving network;

In August 2010, OpenDisplay started to serve a major website, while development was undergoing;

In February 2011, SAPO began migrating it's ad serving network in a process that took about 3 months to complete;

Today OpenDisplay serves the entire SAPO's ad network;

Introduction OpenDisplay

OpenDisplay serves ads for several media:

Internet;

Mobile Internet;

Mobile Applications;

TV set-top boxes;

Connected TV's;

In the near future we'll be serving bulk campaigns for other media;

I'll try to tell in this presentation this endeavour steps and quirks;

OpenDisplay Components FrontendThis component is responsible to serve all ad formats;

No data processing is done here due to performance besides adserving itself;

The adserving is done using munged PHP scripts for performance;

Plugins are included in a on demand basis;

Database queries are cached;

So it's all about ad serving decision making;

OpenDisplay Components BackendComprises data feature processing;

Web based GUI for campaign and ad management;

Ad serving statistics;

Reporting;

Batch processing of ad delivery data for use by the frontends;

OpenDisplay Components TasksMaintenance Priority Engine (MPE)

Determines witch campaigns to serve given their priorities;

Calculates ad serving probabilites given it's probabilities and corrects them when underperforming or overperforming;

Maintenance Statistics Engine (MSE)

Processes ad serving numbers;

Starts and stops campaigns;

Improvements Introduced - General

Added reusable segmentation rules;

This way a rule can be reused in several campaigns;

Added compound segmentation rules;

Segmentation rules engine was rewritten, cause the previous segmentation system was inadequate;

Added the concept of Orders;

Sometimes a customer has several goals to different sites;

The concept of order allows to place several campaigns with different goals in a single customer order;


Added Zone Groups;

Instead of selecting placeholders one a at a time we can associate several at once;

Imagine that a Run of Network (RON) campaign for all MREC (300x250) placeholders would need to be associated to all placeholders one by one;

Added revenue-share acounting;

For ads served on pages with third-party content;

This way, revenue can be shared with third-party content providers;


OpenDisplay went through a security audit by SAPO's security team and several issues were solved;

Backoffice:

UI session cookies are now only delivered over SSL;

Session id generation function wasn't good enought and could be easily guessed. This correction minimized session hijacking;

New user profiles were added, and entity access was reviewed;

Some user profiles were changed to read-only, like advertisers and sites;


Ads uploaded into the ad server are stored in a folder and served upon;

At first look there is no problem with this, but over time in some systems this can cause inode exhaustion;

So to prevent this, and speed up file retrieval we improved upload component to distribute the files in a two-level folder hierarchy;

OpenX can use a content farm to deliver ads, so we use this feature from the start;


Traffic forecast:

OpenX doesn't have a traffic forecast engine, instead it uses an average of ads served;

We developed two alternative forecast algorithms using Python;

This forecast is critical for a couple of reasons:Inventory selling;

Correct impression allocation for campaigns, specially due to targetting rules;


Traffic forecast example:


Added data logging and analysis:

We started to summarize delivery properties to allow us to calculate precise segmentation delivery probabilies;

Using these numbers in combination with traffic forecast we can estimate the inventory for each campaign and it's overall probability of delivery;

Also, this information is useful to commercial purposes:Knowing the market is a very valuable information;

We are currently migrating some of this data to Hbase that reduces data, making it usable;


Restructured VAST 1.0 system and upgraded it to 2.0;

Video Ad Serving Template (VAST) standard from Interactive Advertising Bureau;

Delivers video ads (pre, mid and postrolls);

Delivers overlays;

We also added a new type of ad that allows us to serve SAPO text ads has images;

This virtual ad type works has a proxy to a different ad system, combining two different ad systems;

Probably the first time an ad system combined them;


OpenDisplay (...) http://(...)/video.flv


Flash ads are a major problem in some systems that don't support Flash;

iPhones and iPads for example;

To assure these ads are at all times visible we added automatic Flash ad image generation to ads upload via Backend;

This way, even if a Flash ad doesn't have a fallback image, we generate one automatically;

This was accomplished using GNU's gnash in combination with xvfb-run that provides a virtual X Window System for gnash to run;


Future developments will include bulk campaigns;These campaigns differ from regular campaigns cause we know the characteristics of the audience in advance;

Splitting audiance in sets with the same features we can process an entire set within the LP solver at once minimizing the number of variables;

So we can optimize the revenue using linear programming solutions;We will use GLPK (GNU Linear Programming Kit) has a solver to obtain an optimal solution;

This way we can provide a solution that maximizes a campaign's revenue;


GLPK sample problem:

# Giapetto's problem, maximizing Giapetto's profitvar x1 >=0; /* soldier worths 3 */var x2 >=0; /* train worths 2 */

/* Objective function */maximize z: 3*x1 + 2*x2; // maximize Giapetto's profit

/* Constraints */s.t. Finishing : 2*x1 + x2

tweaking open source

Technology