aws summit benelux 2013 - media and online advertising on aws
TRANSCRIPT
![Page 1: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/1.jpg)
Media and Online Advertising
on AWS
Jan Borch – AWS Solutions Architect
![Page 2: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/2.jpg)
Media Application on AWS
Music streaming
![Page 3: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/3.jpg)
![Page 4: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/4.jpg)
Media Application on AWS
Video streaming
![Page 5: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/5.jpg)
![Page 6: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/6.jpg)
Media Application on AWS
Digital publishing
![Page 7: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/7.jpg)
![Page 8: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/8.jpg)
Media Application on AWS
Media sharing
![Page 9: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/9.jpg)
Martijn Bakker
Chief Engineer,
WeTransfer
![Page 10: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/10.jpg)
Martijn Bakker, Chief Engineer, WeTransfer
AWS Summit Benelux 2013
![Page 11: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/11.jpg)
![Page 12: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/12.jpg)
WeTransfer
• Send big files anywhere
• Up to 2GB per transfer
• Files are stored for 7 days
• Beautiful backgrounds
(50/50 split between ads & art)
2
![Page 13: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/13.jpg)
WeTransfer
• WeTransfer Plus: € 120 per year
• Up to 5GB per transfer
• Store up to 50GB permanently
• Your own backgrounds (no ads)
![Page 14: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/14.jpg)
WeTransfer
• 1.8 Million transfers per day
• Downloads: 25 Gigabit per second
• 1.5 Million requests per hour (site + API)
• Over a Petabyte of storage used on S3
(Peak measurements - July 2013)
![Page 15: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/15.jpg)
Company
• 15 wonderful, dedicated people
• Founded & based in Amsterdam
• Originated from Oy Communications
![Page 16: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/16.jpg)
Origins
• Purely a functional tool
• Design company needs to send big files to clients
• FTP & friends “technical”, “confusing”
![Page 17: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/17.jpg)
Early development
• OyTransfer + advertising = WeTransfer
• Goals:
• beautiful
• easy to use
• secure
![Page 18: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/18.jpg)
Growth
• Double the transfers every
3 months
• Previous hoster:
• could not match growth
tempo adequately
• hardware-based platform;
adds maintenance & development costs
![Page 19: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/19.jpg)
Growth
• Our 3 necessities:
• Storage
• Support
• Scalability
![Page 20: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/20.jpg)
Growth
• AWS: on-demand, available
right away
• Initial migration:
• development time: 1 month
• using S3 through EC2 instances
![Page 21: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/21.jpg)
The new WeTransfer
• Built for & with AWS: uses RDS, EC2, S3,
CloudWatch, DynamoDB, Route53, ElastiCache
• Ruby + HTML5 + JavaScript (frontend)
• Backend tailored around S3
• Launched January of this year
![Page 22: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/22.jpg)
WeTransfer and S3
• Virtually unlimited storage capacity
• Redundancy: always available
• Fast, and cheap compared to similar offers
• Dramatically less costs
![Page 23: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/23.jpg)
WeTransfer and S3
• Uses the multipart upload mechanism (where possible)
• Resumable uploads
• Uploads go directly to S3
thanks to CORS support
• Worker instances to process
uploaded content
![Page 24: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/24.jpg)
WeTransfer and S3
• Secure upload / download, and encryption
• Regionalized: storage facilities all over the world to
ensure proper speeds to end users
• No maintenance
![Page 25: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/25.jpg)
So why S3?
• Fast & flexible
• Almost no time spent on maintenance
• Virtually limitless capacity at the tips of your fingers
![Page 26: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/26.jpg)
https://wetransfer.com/jobs
![Page 27: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/27.jpg)
Online Advertising on AWS
YOUR
AD
HERE
![Page 28: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/28.jpg)
Common challenge for Advertising Platforms ...
![Page 29: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/29.jpg)
... device and media fragmentation ...
TXT
![Page 30: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/30.jpg)
... scaled to millions of users
![Page 31: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/31.jpg)
503 Service Temporarily Unavailable
The server is temporarily unable
to service your request due to
maintenance downtime or capacity
problems. Please try again later.
![Page 32: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/32.jpg)
503 Service Temporarily Unavailable
The server is temporarily unable
to service your request due to
maintenance downtime or capacity
problems. Please try again later.
![Page 33: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/33.jpg)
Maintain availability from one server…
![Page 34: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/34.jpg)
…to thousands
![Page 35: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/35.jpg)
Let's take a journey ...
![Page 36: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/36.jpg)
Let's take a journey ...
Store
![Page 37: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/37.jpg)
AWS S3 Storage for the Internet
![Page 38: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/38.jpg)
STATIC FILES REPOSITORY
AMAZON S3
MEDIA AD SERVED TO USER
![Page 39: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/39.jpg)
Let's take a journey ...
Store
![Page 40: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/40.jpg)
Let's take a journey ...
Transform
![Page 41: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/41.jpg)
AWS Elastic
Transcoder Video transcoding in the cloud
![Page 42: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/42.jpg)
VIDEO FILES REPOSITORY
AMAZON S3
MEDIA AD SERVED TO USER
AMAZON ELASTIC
TRANSCODER
![Page 43: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/43.jpg)
Let's take a journey ...
Transform
![Page 44: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/44.jpg)
Let's take a journey ...
Deliver
![Page 45: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/45.jpg)
AWS CloudFront Web service for content delivery
![Page 46: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/46.jpg)
Dallas(2)
St.Louis
Miami
Jacksonville Los Angeles (2)
Palo Alto
Seattle
Ashburn(2)
Newark New York (2)
Dublin
London(2)
Amsterdam Stockholm
Frankfurt(2) Paris(2)
Singapore(2)
Hong Kong
Tokyo
Sao Paulo
South Bend
San Jose Osaka
Milan
Sydney
Reach a global audience Reach a global audience
![Page 47: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/47.jpg)
CONTENT DELIVERY NETWORK
AMAZON CLOUDFRONT
IMPRESSION LOGS
![Page 48: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/48.jpg)
Simple HLS video streaming architecture
In-house content
publication server
Source Video
Assets in S3
S3
![Page 49: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/49.jpg)
Simple HLS video streaming architecture
In-house content
publication server
Source Video
Assets in S3
Video
transcoded into
HLS
S3 Elastic Transcoder
![Page 50: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/50.jpg)
Simple HLS video streaming architecture
In-house content
publication server
Source Video
Assets in S3
Video
transcoded into
HLS
Edge Delivery
using CloudFront
Stockholm
NY
CloudFront S3 Elastic Transcoder
![Page 51: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/51.jpg)
AWS CLI
aws s3 cp video.avi s3://mybucket/video
aws elastictranscoder create-job
--pipeline-id 1379510897399-mxjrif
--input '{"Key":"video/video.avi"}'
--outputs '[{"Key":"sample","PresetId":"1234-123", ...}]'
![Page 52: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/52.jpg)
Let's take a journey ...
Deliver
![Page 53: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/53.jpg)
Let's take a journey ...
Match
![Page 54: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/54.jpg)
AWS EC2 Resizable compute capacity in the cloud
![Page 55: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/55.jpg)
cc2.8xlarge
Virtual core: 32 - 2 x Intel Xeon
Memory: 60,5 GiB
I/O performance: 10 Gbit
Virtual core: 1
Memory: 1.7 GiB
I/O performance: Moderate
m1.small cr1.8xlarge
Virtual core: 32 - 2 x Intel Xeon
Memory: 240 GiB
I/O performance: 10 Gbit
SSD Instance store: 240 GB
cr1.8xlarge
Virtual core: 16
Memory: 60.5 GiB
I/O performance: 10 Gbit
SSD Instance store: 2 x 1TB
cr1.8xlarge
Virtual core: 16
Memory: 117 GiB
I/O performance: 10 Gbit
Instance store: 24 x 2TB
EC2 instance types
![Page 56: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/56.jpg)
Amazon Route 53 Highly available and scalable Domain Name System
Extremely reliable and cost effective
Feature Details
Global Supported from AWS global edge locations for fast and reliable domain name resolution
Scalable Automatically scales based upon query volumes
Latency based
routing
Supports resolution of endpoints based upon latency, enabling multi-region application delivery
Integrated Integrates with other AWS services allowing Route 53 to front load balancers, S3 and EC2
Reach a global audience
![Page 57: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/57.jpg)
Link to Ad Resource
AMAZON EC2 +
AUTOSCALING
Ad Servers
AMAZON ELB
![Page 58: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/58.jpg)
AWS DYNAMODB fast & fully managed
NoSQL database service
![Page 59: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/59.jpg)
AMAZON DYNAMODB
PROFILES DATABASE
![Page 60: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/60.jpg)
ad-id advertiser max-price imps to
deliver
imps
delivered
1 AAA 100 50000 1200
2 BBB 150 30000 2500
user-id attribute1 attribute2 attribute3 attribute4
A XXX XXX XXX XXX
B YYY YYY YYY YYY
not many
rows
so many
rows
frequent
update
(near realtime)
batch manner update
Ads
Profiles(user-cookie)
Very general table structure
![Page 61: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/61.jpg)
Let's take a journey ...
Match
![Page 62: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/62.jpg)
Let's take a journey ...
Capture
![Page 63: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/63.jpg)
Click-through Servers
AMAZON EC2 +
AUTOSCALING
![Page 64: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/64.jpg)
AWS OPSWORKS INTEGRATED APPLICATION
MANAGEMENT
![Page 65: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/65.jpg)
Stack
![Page 66: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/66.jpg)
Layer Stack
![Page 67: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/67.jpg)
Instances Layer Stack
![Page 68: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/68.jpg)
Scale Instances Layer Stack
![Page 69: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/69.jpg)
Agent on each
EC2 instance OpsWorks talks with
The heart of the service
![Page 70: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/70.jpg)
Instance lifecycle and configuration hooks
![Page 71: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/71.jpg)
Cookbooks
script "install_composer" do
interpreter "bash"
user "root"
cwd
"#{node[:deploy][:myphotoapp][:deploy_to]}/
current"
code <<-EOH
curl -s https://getcomposer.org/installer
| php
php composer.phar install
EOH
end
Amazon S3
Git repository
![Page 72: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/72.jpg)
Let's take a journey ...
Capture
![Page 73: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/73.jpg)
Let's take a journey ...
Report
![Page 74: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/74.jpg)
CLICK-THROUGH LOG FILES
AMAZON S3
![Page 75: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/75.jpg)
AMAZON ELASTIC MAP
REDUCE
CLICK-THROUGH LOG FILES
![Page 76: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/76.jpg)
Data Growth
GB
TB
PB
Data Growth
![Page 77: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/77.jpg)
Data Growth
Data Growth
Server Logs
Click Analysis
Impression logs
![Page 78: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/78.jpg)
Sampling
Big Data
Time to process
Inflexible
Complexities of Big Data
![Page 79: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/79.jpg)
Sampling
Big Data
Inflexible
Complexities of Big Data
Elastic Map Reduce &
Redshift
![Page 80: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/80.jpg)
Sampling
Big Data
Complexities of Big Data
“Queryable”
Elastic Map Reduce &
Redshift
![Page 81: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/81.jpg)
Big Data
Complexities of Big Data
“Queryable” All Data
Elastic Map Reduce &
Redshift
![Page 82: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/82.jpg)
Data Insight
Turning Data into Information
![Page 83: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/83.jpg)
Data Insight
Elastic
MapReduce
Turning Data into information
Redshift
![Page 84: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/84.jpg)
AWS Elastic Map Reduce
Process vast amounts of data using Hadoop
![Page 85: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/85.jpg)
AWS Redshift Fast, fully managed, petabyte-scale data
warehouse service
![Page 86: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/86.jpg)
Let's take a journey ...
Store
Transform
Deliver
Match
Capture
Report
![Page 87: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/87.jpg)
Amazon Web Services
![Page 88: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/88.jpg)
Garry Turkington,
CTO,
Improve Digital
![Page 89: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/89.jpg)
PRESENTATION
TYPE
IMPROVE DIGI
Amazon Web
Services at
Improve Digital
IMPROVE
DIGITAL
26 September 2013 Garry Turkington CTO
@garryimprove
![Page 90: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/90.jpg)
• Cloud-based Real Time Advertising
Technology
• Focus on the premium publisher / media
owner
• Integrations with thousands of Demand
Partners
• Decision driven by Real Time Data
• Offices in UK, NL, DE & ES
• +100 Premium Publishers
IMPROVE
DIGITAL
ABOUT US
![Page 91: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/91.jpg)
Use AWS in conjunction with dedicated physical
infrastructure
2 sides to the story
• Front end: serving of ads to end-users
• Back-end: Data processing and dev/test
Use of AWS
![Page 92: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/92.jpg)
• Fleet of ad servers running mostly on EC2
• Ad serving process is computationally expensive and has strict time constraints
• Need ability to spin up additional instances based on demand: horizontally scalable system
• Place ad servers in different regions to reduce serving latency; big benefit of EC2 over physical kit
• Grow fleets in different regions separately
Serving ads
![Page 93: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/93.jpg)
• S3/Glacier used for policy-driven data retention
• S3 is the starting point for AWS and on-premises data processing jobs
• S3 used as a shared storage space between distributed components
• VPC used to integrate AWS and on-premises flows and systems
• Automated deployment of dev/test software into VPC EC2 has been great
Backend systems
![Page 94: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/94.jpg)
As a startup it was almost a no-brainer
• Didn't want/need overhead of own physical infrastructure
• Pricing model with hugely reduced (or zero) up-front cost is an easy sell
• Coordinating the ability to quickly grow the ad server fleet is *hard* with a physical data centre
• As a more mature company the above still apply
• In addition our needs have also matured from the lower level "give us servers and storage"
Why do we use AWS?
![Page 95: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/95.jpg)
Lessons learned:
It works! Service integration is often ridiculously easy: pull S3 data into EMR, set
up auto-scaling etc
Geographic data locality -- helps with compliance
Automatic cost reductions does wonders for corporate acceptance
Continuous evolution of the services means that they suddenly can be
a great fit
![Page 96: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/96.jpg)
Lessons learned:
It works in its own way
Need to understand exactly what each service offers
Need to design for fault tolerance; instances can fail at high scale
Had to work hard to get our network to integrate with VPC
Can't save you from yourself; poor design is poor design
Would still love to see another region in EU
![Page 97: AWS Summit Benelux 2013 - Media and Online Advertising on AWS](https://reader031.vdocument.in/reader031/viewer/2022021814/58f9bf60760da32f4b8b47b2/html5/thumbnails/97.jpg)
• Growth means more of all the above
• Want to re-evaluate services that weren't a great fit for us in the past (RDS, DynamoDB)
• Believe we can use data processing services (Elastic MapReduce in particular) alongside on-premises systems
• Looking to Cloud Formation/ Elastic Beanstalk and Opsworks to extend automation much further
The future