Mozilla’s HybridContinuous Integration

@armenzghttp://armenzg.blogspot.ca

@MozRelenghttp://planet.mozilla.org/releng

Once upon a time…

THE END

Fast growth will cause your

infrastructure painand lots of costs.

How much have we saved?

Roughly 60-70%

• No in-house cost• Higher EC2 usage

Term definitionsQuick intro to Mozilla’s CI

Inter data-center issuesAWS costs

TERMS

• Push – Commit – Code changes• VPN – Virtual Private Network• AWS – Amazon Web Services

•EC2 – Elastic Compute Cloud•EBS – Elastic Block Store•S3 – Simple Storage Service

• Buildbot

Mozilla’s products:•Non-for-profit & users first•Desktop, Android, Thunderbird & Firefox OS•~1/2 billion Firefox desktop users•50M-100M Android installs•90+ locales•Firefox OS phones in 13+ countries

Mozilla’s CI

1M+ build jobs10M+ test jobs90+ wall-time

years

Every year*:

* Back of the envelope

Mozilla’s CI

Mozilla’s CI

Mozilla’s CI

RelEng’s CI:•2 in-house data-centers•2 Amazon regions•VPC•VPN link between them•Internet data transfers through SSL

Mozilla’s CI

INTER DATACENTER ISSUES

• Loss of stateful connections• Link capacity

Note: Fixing these also brings costs down

http://atlee.ca/blog/posts/aws-networks-and-burning-trees.html

AWS COSTS

• Spot vs On-Demand instances• Shorten job’s wall time• Smarter bidding• Less EBS

http://rail.merail.ca/posts/ec2-spot-instances-experiments.htmlhttp://vps.glek.net/graphite

Spot vs On-Demand instances

http://atlee.ca/blog/posts/initial-jacuzzi-results.html

Shorten job’s wall time

Incremental builds

http://rail.merail.ca/posts/ec2-spot-instances-experiments.htmlhttp://vps.glek.net/graphite

Spot vs On-Demand instances

https://bugzilla.mozilla.org/show_bug.cgi?id=972562

Smarter bidding

- 75% less on-demand75% less on-demand- More instance typesMore instance types- Choose the right Choose the right

regionregion- Pricing analysisPricing analysis

Less EBS

EBS cost us 20% of all costs

What we have covered

• Analyze inter data-center flows• Smart and flexible bidding• Reduce wall time• Reduce disk usage

@armenzg - @MozReleng