code, ci, infrastructure - the gophers way

Code, CI, Infrastructure - the gophers way

Alex BaitovSenior backend developer

2gis.com

About 2GISInternational tech company, developing software for convenient city living

2GIS City Information Service (most successful product)

● combines a map with a city directory

● used by over 40 million people

● covers over 300 cities in 9 different countries

info.2gis.com/index_en.html

Content

1. Application written in golang

2. Ordinary infrastructure

3. Unified infrastructure

4. Gitlab CI as a glue

5. Benefits of using one language

How did we come

to golang in production?

Problem of losing packages

Applicationbackend server

Mainstatisticsserver

Asynchronous reverse proxy with buffer

Applicationbackend server

Mainstatisticsserver

Asynchronous reverse proxy

with bufferon golang

What infrastructure

to choose?

Team infrastructures

Monolith

Microservice

Team infrastructures

Monolith

Microservice

Team infrastructures

Ordinary infrastructure

Monolith

Microservice

Team infrastructures

Ordinary infrastructure

Monolith

Microservice

Team infrastructures

Ordinary infrastructure

Monolith

Microservice

Team infrastructures

Ordinary infrastructure

Unified infrastructure

Monolith

Microservice

Team infrastructures

Ordinary infrastructure

Unified infrastructure

Monolith

Microservice

Microservice (other teams)

Why did we switch

to another infrastructure?

Software development life cycle

SDLC

PlanningPlanning

Develop-ment

Develop-ment

TestingTestingDeliveryDelivery

Mainte-nance

Mainte-nance

Infrastructure

is the core

of SDLC

SDLC: substages

PlanningPlanning

Develop-ment

Develop-ment

TestingTestingDeliveryDelivery

Mainte-nance

Mainte-nance

Application

SDLC

SDLC: substages

PlanningPlanning

Develop-ment

Develop-ment

TestingTestingDeliveryDelivery

Mainte-nance

Mainte-nance

Application

Unit

Functional

SDLC

SDLC: substages

PlanningPlanning

Develop-ment

Develop-ment

TestingTestingDeliveryDelivery

Mainte-nance

Mainte-nance

Application

UnitDeploy

Configuring Functional

SDLC

SDLC: substages

PlanningPlanning

Develop-ment

Develop-ment

TestingTestingDeliveryDelivery

Mainte-nance

Mainte-nance

Application

UnitDeploy

Configuring Functional

Metrics and Alerting

Logging

Monitoring

Init andkeeping alive

SDLC

SDLC: substage tools

PlanningPlanning

Develop-ment

Develop-ment

TestingTestingDeliveryDelivery

Mainte-nance

Mainte-nance

Application

UnitDeploy

Configuring Functional

Metrics and Alerting

Logging

Monitoring

Init andkeeping alive

SDLC

jenkins / rundeck / heat templates in openstack

chef / ansible

zabbix / sensu

application sends by protocol / beaver

kibana

upstart / systemd / monit

openstack / own testing frameworks

test modules

framework /package manager / standards

Disadvantages of lots of tools

● High entry barrier

Disadvantages of lots of tools

● High entry barrier

● Simultaneous studying of all this tools takes a long time

Disadvantages of lots of tools

● High entry barrier

● Simultaneous studying of all this tools takes a long time

● Difficult integration

Disadvantages of lots of tools

● High entry barrier

● Simultaneous studying of all this tools takes a long time

● Difficult integration

● Specific debugging of each tool

Disadvantages of lots of tools

● High entry barrier

● Simultaneous studying of all this tools takes a long time

● Difficult integration

● Specific debugging of each tool

● Human factor (sometimes forget something)

What if you need to look inside a tools or

write a custom module for it?

You need to know tool’s language

SDLC: substage tools languages

PlanningPlanning

Develop-ment

Develop-ment

TestingTestingDeliveryDelivery

Mainte-nance

Mainte-nance

Application

UnitDeploy

Configuring Functional

Metrics and Alerting

Logging

Monitoring

Init andkeeping alive

SDLC

java / heat templates yaml

ruby / python

c, php / ruby

application language / python

javascript

c

php / scala

php / scala / go

php / scala

Disadvantages of different languages of lots of tools

● Different error reporting for each language

Disadvantages of different languages of lots of tools

● Different error reporting for each language

● Harder understanding of how the tool works

Disadvantages of different languages of lots of tools

● Different error reporting for each language

● Harder understanding of how the tool works

● Studying different languages to write custom modules

Advantages of this approach

● Wider knowledge about familiar tools

● Familiar tools help to fast build of minimum viable product (MVP)

Disadvantages of this approach

● Too much support for one team

● Too much tools for building microservices

● Long everyday releases

● Expensive development process

How to improve it?

Unified infrastructure

● DEIS - platform as a service that deploys and scales containers

● Docker - pack, ship and run any application as a lightweight, portable,

self-sufficient container

● Prometheus - metrics and alerting system

● Grafana - metrics visualization

Unified infrastructure

● DEIS - platform as a service that deploys and scales containers

● Docker - pack, ship and run any application as a lightweight, portable,

self-sufficient container

● Prometheus - metrics and alerting system

● Grafana - metrics visualization

All of them are:

● open-source● written on go

Unified infrastructure

● DEIS -> DEIS 2 ( Workflow / Kubernetes )

● Docker - pack, ship and run any application as a lightweight, portable,

self-sufficient container

● Prometheus - metrics and alerting system

● Grafana - metrics visualization

All of them are:

● open-source● written on go

How do we use mostly golang

infrastructure in production?

SDLC: substage tools

PlanningPlanning

Develop-ment

Develop-ment

TestingTestingDeliveryDelivery

Mainte-nance

Mainte-nance

Application

SDLC

native /framework /vendoring / standards

SDLC: substage tools

PlanningPlanning

Develop-ment

Develop-ment

TestingTestingDeliveryDelivery

Mainte-nance

Mainte-nance

Application

Unit

Functional

SDLC

docker-compose / test library

test library

native /framework /vendoring / standards

SDLC: substage tools

PlanningPlanning

Develop-ment

Develop-ment

TestingTestingDeliveryDelivery

Mainte-nance

Mainte-nance

Application

UnitDeploy

Configuring Functional

SDLC

gitlab-ci / deis

gitlab-ci / deis (via environment variables)

docker-compose / test library

test library

native /framework /vendoring / standards

SDLC: substage tools

PlanningPlanning

Develop-ment

Develop-ment

TestingTestingDeliveryDelivery

Mainte-nance

Mainte-nance

Application

UnitDeploy

Configuring Functional

Metrics and Alerting

Logging

Monitoring

Init andkeeping alive

SDLC

gitlab-ci / deis

gitlab-ci / deis (via environment variables)

prometheus

deis captures application output

deis (via checking health)

kibana / grafana

docker-compose / test library

test library

native /framework /vendoring / standards

SDLC: substage tools languages

PlanningPlanning

Develop-ment

Develop-ment

TestingTestingDeliveryDelivery

Mainte-nance

Mainte-nance

Application

UnitDeploy

Configuring Functional

Metrics and Alerting

Logging

Monitoring

Init andkeeping alive

SDLC

go / gitlab-ci yaml

go / gitlab-ci yaml

go

go

go / javascript

go

go

go

go

This infrastructureis also used for

Scala, Node.js and C#applications

Comparison of SDLCs

Ordinary infrastructure

Tools (quantity)

Golang infrastructure

Languages (quantity)

15 + 8

7 3

How do we use it together?

Gitlab workflow

● 2 shared runners with docker: 16 GB, 32 CPU

● Gitlab-ci job isolation with docker or docker-compose

● Latest branch docker images of application - in entire

docker registry

● That allows to rollback

● Job parallelisation

Gitlab pipeline

Everything in code

● Application● Tests

Everything in code

● Application● Tests● Deploy● Configuration

Everything in code

● Application● Tests● Deploy● Configuration● Metrics● Alerts rules

Everything in code

● Application● Tests● Deploy● Configuration● Metrics● Alerts rules

Review everything via merge requestsEasy to resolve problem

What are benefits

from using golang

in application and infrastructure?

Benefits of language coincidence

First official tool client goes on its native language: go

Benefits of language coincidence

First official tool client goes on its native language: go

Easy to read and understand tools logic

Benefits of language coincidence

First official tool client goes on its native language: go

Easy to read and understand tools logic

Easy to contribute in infrastructure

Benefits of language coincidence

First official tool client goes on its native language: go

Easy to read and understand tools logic

Easy to contribute in infrastructure

Concentrate on one language

Benefits of language coincidence

First official tool client goes on its native language: go

Easy to read and understand tools logic

Easy to contribute in infrastructure

Concentrate on one language

Fast maintenance reaction

Benefits of language coincidence

First official tool client goes on its native language: go

Easy to read and understand tools logic

Easy to contribute in infrastructure

Concentrate on one language

Fast maintenance reaction

Fast mvp

Conclusion

Everything in code

Mostly one language

Golang is good for infrastructure tools

Golang is production proved

Alex Baitov al.baitov@2gis.ru

alex.baitov@gmail.com

slides & links : goo.gl / LKPmvi

2gis.ru

Thank you!

Extras

DockerAn open-source project that automates the deployment of Linux applications inside software containers

Also use:

● Docker-compose● Docker registry

DeisAn open-source platform as a service (PaaS) that makes it easy to deploy and manage applications on your own servers.

Deis builds upon Docker and CoreOS to provide a lightweight PaaS with a Heroku-inspired workflow.

Built-in:

● Logging● Scaling● Routing● Fault-tolerance

PrometheusAn open-source metrics and alerting system:

● Implements a highly dimensional model data● Flexible querying language● Built-in visualisation● Federation system● Push or pull data mining● Alert manager● Many client libraries