automate ci/cd with rancher

CI/CD with RancherHow we automate container orchestration in our environments at Kloeckner-i using Gitlab, Docker and Rancher

Original google slides version: https://goo.gl/YiMQkJ

https://goo.gl/YiMQkJ

Nick Thomas

Developer

Code Fairy

Working at Kloeckner-i

github.com/pangdudu

[email protected]

What’s this talk about?

Topics

I. IntroductionII. Container Orchestration

III. CI/CD with RancherIV. Setup storage management in

Rancher

Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher

source: medium

.com

Kloeckner-i

● Kloeckner-i, a corporate-startup hybrid based in Berlin

● We’re building the next generation of tools for the steel industry

● To spend as much time as possible developing, we need to automate everything else

source: pdssheetmetal.co.uk


Automate all the things!

What we want to automate:

● Checking our code style● Testing our code● Creating QA environments and

deploying our code to them● Running integration tests● Deploying our code to production

systems● Scaling our systems if we need

to


What we’re building - Microservices

Platform

Frontendfrontend.DOMAIN

...

API Proxyapi.DOMAIN

Backend Magic(Services)

ERP (SAP)

IDPIdentityProvider

idp.DOMAIN

Account Producer

Contract Producer

Order Producer

Con

nect

or A

PI

Central processing system(Multiple queues)

Consumer

Consumer

Webfrontend.DOMAIN


Container Orchestration

● Tools that organizations use to manage containers at scale (and in production)

● May include features such as container:○ Upgrades○ Scaling○ Multi Host Networking○ Storage management○ Health checks○ ...


source: quora.com

So why Rancher?

Candidates

● Amazon ECS● Google Container Engine● Azure Container Service● Apache Mesos● Kubernetes● Docker Swarm● CF Diego● CoreOS Fleet● Puppet, Bash, ...

Reasons I chose Rancher

● Wanted no vendor lock-in (self hostable and open-source)

● Something developers knowing docker-compose understand

● Docker Swarm (1.12) not ready● Mesos and Kubernetes more complicated● Github: 4K, Issues: open 1K, closed 4K● Powerful GUI● Just feels right


CI/CD with Gitlab, Docker, Rancher

Current stack

● Hosting: Google Compute Engine● Automation: Gitlab

○ Run tests: docker runner○ Host docker images: docker registry

● Orchestration: Rancher

source: lakehub.co.ke


Docker development workflow

1. Build Dockerfile2. Run container3. Build docker-compose.yml4. Run container with dependencies


Gitlab Dockerception

1. Build .gitlab-ci.ymla. Lintb. Testc. Build docker image

2. Push to repository3. Gitlab runs your jobs4. Build docker containers within

docker containers with docker


.gitlab-ci.yml


Gitlab Dockerception - Demo


http://www.youtube.com/watch?v=_MoHz2eop5o

Rancher - Another layer on top

Source: rancher.com


Rancher Features

● Rancher structure○ Environments○ Stacks and Services○ Catalog

● Discoverable API● Easy to setup

Source: rancher.com


Rancher - Demo


http://www.youtube.com/watch?v=K3raYllbriQ

SHIA - Just do it!

● Connecting Gitlab and Rancher● Rubygem SHIA● Deploy stack to environment● Deploy full stack to a “QA” environment● Don’t let your dreams be dreams …● ��

SHIA stands for Seamless Hive Integration Agent (it’s really hard to find buzzwords for that acronym )


SHIA - Deploying Services

● Create Environment● Add hosts on GCE via docker-machine● Add users to environment● Process stack config● Create description (branch:commit-hash)● Create or update stack● One command: shia -e linuxcon deploy


SHIA - Deploying QA Environments

● Convention “@env_xx” in git branch name● Deploys stack that is to be tested with current branch● Deploys the rest of the stack with master branch● Multiple stacks can be deployed to the same environment with different

branches● One command: shia -b ${CI_BUILD_REF_NAME} deploy_all


SHIA - Deploying Services - Demo


http://www.youtube.com/watch?v=k6iiCYY7f_c

SHIA - Deploying QA Environments - Demo


http://www.youtube.com/watch?v=hNH1VlbMxHs

SHIA - What we’re working on

● Automate GCE networking configuration○ Create GCE LB○ Configure DNS (currently using myra cloud)

● Seed data efficiently in QA environments● Run full stack integration tests in QA environments


Storage Management with Rancher


Storage options with Docker

● Default : No Data Persistence● Data Volumes : Container Persistence● Data Only Container : Container

Persistence● Host Mapped Volume : Container

Persistence● Host Mapped Volume, backed by Shared

Storage : Host Persistence● Convoy Volume Plugin : Host Persistence

(of Storage Host)


Convoy-NFS features


● Docker Volume Driver Plugin● Backend for different storage options

○ Device Mapper○ NFS/EFS/VFS

● Snapshots, Backups, Restore● Single Go binary● Can be deployed independently of Rancher

Convoy-NFS

Sou

rce:

ranc

her.c

om


Setting up convoy-nfs


http://www.youtube.com/watch?v=ycCKckE0bEM

Where to get SHIA

Kloeckner-i Github

https://github.com/kloeckner-i/shia

Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher / Fin



Introduction / Orchestration / CI/CD with Rancher / Storage management in Rancher / Fin

It’s easy. JUST DO IT!

Original google slides version: https://goo.gl/YiMQkJ

https://goo.gl/YiMQkJ

automate ci/cd with rancher

Technology