you don’t need dtap + backbase implementation - amsterdam 17-12-2015

You don’t need DTAPDevOps and Continuous Delivery at Backbase

DevOps Amsterdam meetup, 17-12-2015

Backbase, Amsterdam

Pavel Chunyayev

@PavelChunyayev

Amsterdam

Levi9 HQAmsterdam – 2005

25 people

Novi Sad

SerbiaNovi Sad – 2005

320+ people

Zrenjanin

SerbiaZrenjanin– 2014

30+ people

Iasi

RomaniaIasi – 200780+ people

Kiev

UkraineKiev – 2008130+ people

@PavelChunyayev

Electronic

Retail

Digital

Marketing

Traffic and

Transport

Software

Services

@PavelChunyayev

Customer satisfaction 2015

• 4th year great results in Outsourcing Performance study

• 2015: 100% customer recommendation scoreExcellent trust score

@PavelChunyayev

Forester Wave: Omni channel Banking Solutions Q3‘15

@PavelChunyayev

Gartner Magic Quadrant for Horizontal Portals 2014

@PavelChunyayev

Mark Rutteopens Backbaseoffice in Atlanta

@PavelChunyayev

Top 10 companies in NL with best work-life balance

@PavelChunyayev

About me

• 11 years of IT experience

• Lived and worked in Ukraine and Estonia

• Moved a year ago to the Netherlands

• Learning Dutch

• Love cycling

• De Holandse 100• https://www.dehollandse100.nl/actie/pavel-chunyayev

Continuous Delivery

Safely, rapidly and predictably deliver new features to production

Quality > Speed

DTAP

@PavelChunyayev

Journey across environments

Development Testing Acceptance Production

@PavelChunyayev

DTAP is history

• It was hard to create infrastructure -> we had to reuse it

• Manually provisioned environments• Took a lot of time

• Ticket to system administrator

• Nightmare for both parties• End up with golden images

• Fixed set of environments• Name them

• Configure once

• Maybe even automate deployment

@PavelChunyayev

DTAP is evil

• Configuration drift• Environments have nothing to do with production

• With time they move more and more apart

• Dirty environments• Undocumented, forgotten hacks

• No one can recreate environment as no one knows how to configure them

• Testing is complicated• Different bugs on different environments

Downwards spiral

You don’t need DTAP

Immutable infrastructure

Immutable infrastructureImmutable delivery

Easy on-demand environments

@PavelChunyayev

Change in paradigm

• Change in paradigm of the way software is built, shipped and managed

• No more static infrastructure

• Provision and create infrastructure just in time when it’s needed

• And then throw it away

Create • Use • Dispose

@PavelChunyayev

Never change again

• Never touch or change the running system. • No configuration changes

• No complicated update processed

• Any change = new deploy

• Create infrastructure right before deployment

• You always start from the scratch• Identical process

• No more dirty environments

• Does it matter how environment is called if the deployment process is identical?

@PavelChunyayev

Variability

• Variability is enemy

• Manufacturing: Six sigma

• How to deliver environments at scale?

@PavelChunyayev

Prerequisites

• Cloud • Manage infrastructure lifecycle as code / through API

• Docker containers

• Automate the setup and deployment of your application

@PavelChunyayev

Continuous Delivery

• Immutable infrastructure enables CD

Build • Test • Deploy

• When you want to change - go through the whole pipeline

@PavelChunyayev

Infrastructure is integral part of application

• Application is delivered as a set of immutable virtual machines or containers.

• Any change in the application is actually a change of infrastructure

• Exactly the same process to deploy any version of the application

• MTTR is predictable • It’s time required for whole pipeline execution

• Gives incentive to optimize the process in order to speed it up

@PavelChunyayev

Immutable infrastructure applies to

• Stateless applications• State is stored outside (DB)

• Even more relevant for microservices

• Pushing of responsibility to someone else

• Databases

• Separate DBMS from data

• There are solutions for this (e.g. Flocker)

@PavelChunyayev

Deliverables

• Server images (AMIs) or containers should be built using automated CI/CD process• Base image + application = deliverable

• Or the configuration process and deployment scripts are applied over and over again

@PavelChunyayev

Deliverables

• Create your own images, don't reuse someone else's

• Images must be versioned and all history must be saved

• Implement automated testing of infrastructure

• Create building blocks

• Test deployments without any fear• Deployments are automated

• Identical deployment procedures to all environments

• Green/blue, canary and rolling deployments

@PavelChunyayev

Immutable infrastructure

• Get infrastructure exactly when it’s needed

• Dispose immediately after use

• Never regret

Continuous Delivery at BackbaseAnd immutable infrastructure

@PavelChunyayev

Backbase

• Research and Development

• Professional Services

Customer Experience Platform

Launchpad Mobile SDK

Digital Banking Services

Digital Banking Platform

@PavelChunyayev

CXP Architecture

@PavelChunyayev

Publishing

• Multiserver configuration is needed to test publishing

Editorial Staging Live

@PavelChunyayev

Dual deployment model

• Cloud Native first

• But bank are not ready

• Dual deployment model• Containers/cloud – modern

• Enterprise style

@PavelChunyayev

Teams structure

• Squads• Team is divided into squads

• Each squad is responsible for one or more components

• Components• Components = microservices

• >10 BE components

• Dozens of FE components

• Continuous Delivery squad is a Platform team

@PavelChunyayev

Platform team

• Now consists of 4 people, 5th starting in January

• You can be part of this team

• Several generations of platform• We are in the middle of introduction 4th generation

@PavelChunyayev

Generations

1. Infrastructure as a service MVP

2. All configurations for CXP 5.6 and testing at scale

3. Docker and microservices support

4. Unified way of delivering infrastructure (being rolled out)

@PavelChunyayev

Development environments

• Docker containers = building blocks• Docker Machine

• Docker Compose

• No need to use virtual machines

• No need to use jetty and h2

• No need to start whole environment

@PavelChunyayev

Building blocks

• Ansible roles

• Packer

• Infrastructure testing• Before – Starting full environments

• Now – Testing roles in isolation• Test-kitchen

• Serverspec

@PavelChunyayev

Delivery pipelines

git Build Version Test Release

Component 1

git Build Version Test Release

Component 2

git Build Version Test Release

Component 3

Assemble CXP

Test Release

342

128

473

921

@PavelChunyayev

Pipelines as code

• Pipelines as code• Changes -> same process as code changes

• First stage is enforcing specifications

• GoCD + Gomatic (updated)• https://github.com/Backbase/gomatic

@PavelChunyayev

Platform

• Services to manage environments lifecycle

• The same way for everyone

• Autoconfig• AWS

• Autodocker• Docker

@PavelChunyayev

Autodocker

• Swarm cluster• Interface for docker environments provisioning

• Limitations• One host for networking

• Memory checks

• Destruction after TTL

• Configuration is a docker compose file

• Start exact version of your component and released version of all other components

@PavelChunyayev

Autoconfig

• AWS on-demand multi-server environments• Any application version• Any configuration option

• JSON (Form data) with parameters{

"server": "jboss",

"database": "oracle",

"universal_collection": "true",

"htts": "true"

}

@PavelChunyayev

Autoconfig APIs

• GET /api/stacks - List stacks available for provisioning

• GET /api/environments - List all currently provisioned environments

• POST /api/stacks/<stack_name> - Provision specified stack

• DELETE /api/environments/<environment_id> - Destroy environment with specified id

• DELETE /api/environments/all - Destroy all environments

@PavelChunyayev

Autodocker APIs

• POST /api/docker - create new docker environment

• GET /api/docker/<env-name> - get info about specified environment

• GET /api/docker - get info about docker environments provisioned by the user Optional parameters:• all_users = True • uptime (value in hours) • owner (AD username)

• DELETE /api/docker/<env-name> - delete specified docker environment

• DELETE /api/docker - remove docker environments provisioned by the user Optional parameters:• all_users = True • uptime (value in hours)• owner (AD username)

@PavelChunyayev

Quality

• Build the quality in the process

• Move tests to the left

• Testing pyramid

• No more manual testing

• Ideal testing framework • Manage parallelism

• Create environments

M

E2E

Contract

Component

Integration

Unit

@PavelChunyayev

Releasing

• Banks don’t want frequent releases

• 3 stage release process• Rolling

• In the effort to not release too infrequently

• Technical previews• At the end of every sprint

• Marketing releases• When product manager decides to release

@PavelChunyayev

Culture

• No blaming, no fingerpointing

• Failing the build is a learning opportunity

• We failed a lot• Several iterations of JS deployments

• Visibility

• Shared responsibility• Squad – for component delivery

• Tribe – for product delivery

@PavelChunyayev

Key takeaways

• Forget about DTAP

• Believe in immutable infrastructure

• Create repeatable and reliable process for releasing software

• Build quality in

• Improve continuously

Any questions?

pavel@levi9.com