devops oxford- devops + bigdata @ realtime

DevOps + BigData @ RealTime

DevOps+(DevOps in a startup)

DevOps+● Fixing the electrical outage

DevOps+● Fixing the electrical outage● Cycling to Maplin for an extension lead● Dealing with the sewage from the blocked toilet

BigData● Also known as - Data● BigData is writing Doom or Elite for a 32k machine:

BigData● Or just because you can’t quite fit it on a 39” rotating disk:

@RealTime● Can you survive this ?

The Challenge - The 2014 (Football) World Cup

Analyse every Tweet issued during World Cup games in the 2014 competition

The Challenge - The 2014 (Football) World Cup

The Challenge - The 2014 (Football) World Cup

How busy was it expected to be ?

The Challenge - The 2014 (Football) World Cup

How busy was it expected to be ?● Predicted 1.2M Tweets per hour● Average time for our analysis is 100ms - lengthy texts up to 150ms● Our analysis time increases logarithmically with sentence count and sentence length so needed to avoid pathological cases● 150ms x 1.2M = 180k sec = 3000 minutes = between 33 and 50 hours

The Solution - AWS

● Lots of EC2 instances● 3 Availability Zones● ELB

The Solution - Unattended Build

● Cloud Formation● Puppet● Route53● ELB● 2 min

The Challenge - The 2014 (Football) World Cup

How busy was it ?● Predicted peak 1.2M Tweets per hour● Actual peak 2.5M Tweets per hour● Whole tournament avg 17M per day for 30 days● Peak 33M in one day

The Challenge - The 2014 (Football) World Cup

How busy was it ?● Predicted peak 1.2M Tweets per hour● Actual peak 2.5M Tweets per hour● Whole tournament avg 17M per day for 30 days● Peak 33M in one day

The Challenge - The 2014 (Football) World Cup

How busy was it ?● Predicted peak 1.2M Tweets per hour● Actual peak 2.5M Tweets per hour● Whole tournament avg 17M per day for 30 days● Peak 33M in one day

● A Long Time Ago In A Galaxy Far Far Away● Carhire3000 began - we had 6 servers● Building a new server took a week● 8 years later:-

○ 2 acquisitions○ Booking.com○ $996 million○ 600+ servers○ DevOps team of 5

● Booking.com methodology - Unattended build in 5 minutes● Control Room - Monitoring health of website

● Control Room - Columnar Databases

● Columnar Databases● Fire & Forget UDP events● What do you monitor ?● Experiments & Z-score● DB Queries < 2 seconds● Hot event data in MySQL - Hands Off !● 5 minutes later moved to columnar storage database