continuously serving the oss community with continuous integration and delivery, by akshay karle and...
DESCRIPTION
Snap CI now enables public GitHub repositories to do Continuous Integration(CI) and Continuous Delivery(CD) for free. When doing CD, the goal is to automate the process of deployment and build software in a way that can be deployed to production anytime. To achieve this, we need to figure out a way to deploy the application that doesn't interrupt the users using the system. Learning about this process will allow you to reduce the frictions and delays for deploying new features and bug fixes to production. Your application needs to change in a way that understands this process. You will learn about application and deployment pipelines and the different ways to deploy. In this talk we will touch upon some common deployment pipelines. We will cover in detail about how we changed Snap to do deployments that didn't affect our customers, the challenges we faced and some patterns that helped us overcome them. What made this challenging was the data migrations and long running builds of our customers that made it difficult to figure out when to cut over.TRANSCRIPT
Continuously serving the OSS community with
Continuous Integration and Delivery
Akshay Karle !Fernando Júnior
“How long would it take your organization to deploy a change that involves just one single line of code?”
Mary and Tom Poppendieck
“(…) build software in such a way that the software can be released to
production at any time.” Martin Fowler
CONTINUOUS INTEGRATION
CONTINUOUS INTEGRATION
CONTINUOUS INTEGRATION
CONTINUOUS DELIVERY
CONTINUOUS DELIVERY
!
!
▫︎ Continuous Integration
!
▫︎ Automated tasks
!
▫︎ Repeatable/reliable process
THERE’S MORE…
CONTINUOUS DELIVERY
CONTINUOUS DELIVERY
DEPLOYING YOUR APPLICATION
!
!
▫︎ Database migrations
!
▫︎ Infrastructure update
!
▫︎ Restarting services
ZERO DOWNTIME DEPLOYMENTS
Deployment process of your application has got to be transparent for end users
WHO WE ARE?
Developer, Snap CI
/nandopaf
/fernando-alves
/nand0paf
Developer, Snap CI
/akshay_karle
/akshaykarle
/akshay_ka
WHAT WE DO?
!
!
!
!
▫︎Continuous Integration for repositories on GitHub
▫︎ SaaS
▫︎Helps you do Continuous Delivery
▫︎ Free for open source projects
SNAP INSIDE-OUT
LIFECYCLE OF A BUILD
LIFECYCLE OF A BUILD
LIFECYCLE OF A BUILD
LIFECYCLE OF A BUILD
head repository commit
LIFECYCLE OF A BUILD
head repository commit
LIFECYCLE OF A BUILD
head repository commit
DATABASE
LIFECYCLE OF A BUILD
head repository commit
DATABASE
LIFECYCLE OF A BUILD
head repository commit
DATABASE
Babysitters
LIFECYCLE OF A BUILD
head repository commit
DATABASE
Babysitters
LIFECYCLE OF A BUILD
head repository commit
DATABASE
Babysitters
LIFECYCLE OF A BUILD
head repository commit
DATABASE
LIFE CYCLE
!
▫︎ Prepare the container
▫︎ Starts database
▫︎ Sets PATH
▫︎ Download artifacts
▫︎ Git clone
▫︎ Runs each pipeline stage
▫︎ Upload artifacts
DATABASE
Babysitters
LIFECYCLE OF A BUILD
head repository commit
DATABASE
Babysitters
LIFECYCLE OF A BUILD
head repository commit
DATABASE
ARCHITECTURE OVERVIEW
L B
ARCHITECTURE OVERVIEW
L B
Database
web server
ARCHITECTURE OVERVIEW
!
!
▫︎ Rails app fronted by apache
!
▫︎ Receives the hooks from GitHub
Database
web server
ARCHITECTURE OVERVIEW
L B
web server
Database
ARCHITECTURE OVERVIEW
L B
Build Server
web server
Database
ARCHITECTURE OVERVIEW
!
▫︎ Background jobs
▫︎ Babysitters
▫︎ Build Queue
▫︎ Artifacts
!
▫︎ OpenVZ Containers
▫︎ Virtual machines where the pipeline runs
Build Server
ARCHITECTURE OVERVIEW
L B
Build Server
web server
Database
OUR DEPLOYMENT
!
▫︎ Do have automated scripts
!
▫︎ Deployment pipeline
!
▫︎ 1-click deploy
▫︎ Sort of…
Babysitters
DATABASE
L B
Build Server
Database
web server
OUR DEPLOYMENT
!
▫︎ Wait for all builds to finish
!
▫︎ Put app on maintenance mode
▫︎ No new requests picked up
!
▫︎ Deploy and wait for migrations
Babysitters
DATABASE
L B
Database
VZHOSTBuild Server
web server
ZERO DOWNTIME DEPLOYMENTS
BLUE-GREEN DEPLOYMENT
Web Server App Server Database
BLUE-GREEN DEPLOYMENT
Web Server App Server Database
App ServerApp ServerWeb Server Database
BLUE-GREEN DEPLOYMENT
Web Server App Server Database
App ServerApp ServerWeb Server Database
BLUE-GREEN DEPLOYMENT
App ServerApp ServerWeb Server Database
BLUE-GREEN DEPLOYMENTS FOR SNAP
VZHOSTBuild Server
web server
VZHOSTBuild Server
web server
L B DATABASE
BLUE-GREEN DEPLOYMENTS FOR SNAP
VZHOSTBuild Server
web server
VZHOSTBuild Server
web server
L B DATABASE
CHALLENGES IN SNAP
Long running builds
Database migrations
CHALLENGES IN SNAP
Long running builds
LONG RUNNING BUILDS
VZHOSTBuild Server
web server
VZHOSTBuild Server
web server
L B DATABASE
LONG RUNNING BUILDS
VZHOSTBuild Server
web server
VZHOSTBuild Server
web server
L B DATABASE
LONG RUNNING BUILDS
!
▫︎ Builds should continue to run
!
▫︎ Artifacts should continue to be served
COLOR AWARE BABYSITTERS
Build Server
Green Stack
Build Server
Blue Stack
COLOR AWARE BABYSITTERS
Build Server
Green Stack
Build Server
Blue Stack
web server
BUILD LIFECYCLE
head repository commit
DATABASEDATABASE
L B
BUILD LIFECYCLE
head repository commit
DATABASEDATABASE
L B
BUILD LIFECYCLE
head repository commit
DATABASEDATABASE
L B
THE SWITCH
L B
THE SWITCH
L B
THE SWITCH
L B
THE SWITCH
L B
THE SWITCH
L B
THE SWITCH
L B
THE SWITCH
L B
CHALLENGES IN SNAP
Long running builds
Database migrations
CHALLENGES IN SNAP
Database migrations
DATABASE MIGRATIONS
PipelinesCounterDuration
App v1.0
App v1.0
L B
DATABASE MIGRATIONS
PipelinesCounterDuration
App v1.0
App v1.0
L B
Upgrading to v2.0, migrating
DATABASE MIGRATIONS
PipelinesCounter
App v2.0
App v1.0
L B
Cannot have destructive migrations
MIGRATIONS IN PHASES
PipelinesCounterDuration
App v1.0
App v1.0
L B
MIGRATIONS IN PHASES
PipelinesCounterDuration
App v1.0
App v1.0
Upgrading to v2.0
L B
MIGRATIONS IN PHASES
PipelinesCounterDuration
App v2.0
App v1.0
L B
MIGRATIONS IN PHASES
PipelinesCounterDuration
App v2.0
App v1.0
L B
MIGRATIONS IN PHASES
PipelinesCounterDuration
App v2.0
App v1.0
L B
MIGRATIONS IN PHASES
PipelinesCounterDuration
App v2.0
App v1.0Upgrading to v3.0
L B
MIGRATIONS IN PHASES
App v2.0
App v3.0
PipelinesCounter
L B
MIGRATIONS IN PHASES
App v2.0
App v3.0
PipelinesCounter
L B
LESSONS LEARNT
Automation is beauty
Watch out your data
Things will go wrong
THANK YOUQuestions?
https://snap-ci.com