pimp your pipeline - central configuration management - jens saade

Pimp Your Pipeline

Central Configuration Management

Jens Saade, youngculture April 25th 2015, Iasi CodeCamp Romania

ReferencesAbout youngculture Group

Jens Saade

Head Innovation & Technology E-Commerce Engineering Background (Retail, PIM, E-Commerce)Located in Zürich, CH

Dev Process Architecture Innova2on

Ini2a2ves

Central Configuration Repository

Appliance in Continuous Delivery

Automation & Configuration

Intro youngculture E-Commerce

Intro: E-Commerce

youngculture ECOM team delivers highly integrated E-Commerce Solutions based on the Hybris Platform.

Customized B2C Storefront Showcase

B2C & B2B Shops and PIM Systems

Delivering good quality in a highly complex project environment is only possible by aligning team, processes and infrastructure environments.

Con2nuous Delivery

Configura2on Mgmt

Test Automa2on

Cloud Services

Automated Provisioning

Workspace (Confluence,

Prac2ce Meetups

Integrated Communica2on

(HipChat)

System Monitoring

HW/OS (NewRelic, Nagios)

Applica2on Monitoring

Logs (Elas2cSearch/Kibana, Splunk)

Performance (Dynatrace),

Delivery Environment

EnvironmentsCAM

Collaborate

AutomateMonitor

TeamsSystems

Projects

Automation

Continuous Delivery

Familiar with the Concepts? Tried applying it in own Company?

Part of your Culture?

AUTOMATION is hard work ...expensive to establish...and can get pretty complex

... reduces monkey work... speeds up the dev & delivery processes

... calms down stress level ...and raises confidence and quality.

Release anytime, reproducible

Create Hybris

Projects, Extension

and Addons

SetupContinously

Build deployable

artifacts

BuildAssess

Software by Automated

Testing

Continuous Delivery Overview

Install for testing,

acceptance and

production

Deploy

> Commit Stage > Acceptance Stage > Deployment Stage

Feedback Feedback Feedback

Establishing a Delivery Pipeline is a joint team effort! It involves lots of disciplines and evolves constantly.

CustomersDevelopers Product Owner

Scrum Master

Delivery Process: Automation is Key

Test Execution EnvJenkinsCI

Artifactory Repository

GitRepository

Hybris Node

YC Deploy Client

Apache HTTPdSelenium

GatlingJMeter

Test Exec Hybris Test

Commit Stage

Acceptance Stage

Deploy Stage

Continuous Delivery

Monitoring Node

Stage/Production

Hybris Node1

Hybris Cluster

Apache HTTPd

Varnish

Apache HTTPd

Varnish

DB Cluster

Load Balancer

ElasticSeach

Logstash

Kibana

Zabbix

External Monitoring

NewRelic

Hybris Node 2

YC Deploy Client

Hosting Environment

Maintenance team

One essential problem left:

Configurations

Ecosystem of Configurations

Versioned Software Artifact

Test specifc artifact

versions for regressionQA on

features

Create new Conf

PropertyShare his conf with

Prepare RolloutConf for tonight

Enable feature for customer

Req Eng

Introduce new concept, define triggers

B=truec=debug

7Env.properties

? Project Specific

Project.properties

436Advanced.properties

7Env.properties

? Project Specific

Project.properties

Node 1

7Env.properties

? Project Specific

Project.properties

Node 2

7Env.properties

? Project Specific

Project.properties

Node 3

1 Clustered Environment

Dev(s)QA SystemsProductionUAT/Stage...

Hybris Configuration Stack (Properties based)

(manual) Configuration

Central Configuration Management

Diversity: Support complex environment setups

We need good concepts how to configure environments nearly equivalent

but allow flexibility for individual deviations

„The most efficient way to manage configuration is to have a central service through which every application can get the configuration it needs.“

Jez Humble and Dave Farley „Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation“, Addison-Wesley 2010

Artifact

Generic Configuration

Machine

Java VM

Rolling out Configurations

Configurations get packaged and rolled out

JVMs are bootstrapped from packaged configurations

Machine

Java VM

Machine

Java VM

Machine

Java VM

Environment Configuration

Turning from distributing (and manually tweaking) local configurations ...

... to a centralized environment that hosts configurations for self-service.

Direction change: Push -> Pull

Artifact

Machine

Java VM

Rolling out Configurations, alternative

Rollout Software Artifact as usual, but keep it generic, just provide configuration IDs

Isolate configurations from the deployment package and provide it centrally from the CCR

Machine

Java VM

Machine

Java VM

Machine

Java VM

UUID UUID

Hierachical Key-Value Store

Configuration

Machine

Application

Client Wrapper

Central Configuration Repository (CCR) and Consuming Clients

Agent pulls configuration properties via REST / Json

Then injects properties to JVM System context

MachineMachineMachine Machine

Machine

MachineMachineMachineMachine

Machine

REST / Json

discoBit

Admin UI

CD & CCR Applied @YC

Configuration Hierarchies

Automated space

Shared space

UAT Environment

UAT node 1

UAT Node 2

QA: Nightly Build Test System

Personalized Conf Developer A

Personalized Conf Developer B

Req Eng

Continuous Delivery Cycle

Customized (env1)

Customized (env2)

Machine

Java VM

discoBit Java Agent (UUID4)

System Context Injection via Java Agent

discoBit

Agent pulls configuration properties via REST / Json

Then injects properties to JVM System context

Automated space

Shared space

jsaade@MacBook-Pro:~/hybris/hybris-commerce-suite-5.4.0.0/hybris/bin/platform$ ./hybrisserver.sh Launching a JVM...Apr 20, 2015 1:42:33 PM com.v3rticle.oss.discobit.client.bootstrap.DiscobitSettings <init>INFORMATION: [discobit] configuring server from system: http://127.0.0.1:8089

Apr 20, 2015 1:42:33 PM com.youngculture.ecom.hybris.discobit.HybrisDiscobitAgent executeINFORMATION: Reading discobit configuration from http://127.0.0.1:8089::bb2835fb-466c-4143-8081-ffcd60c49467

Apr 20, 2015 1:42:33 PM com.v3rticle.oss.discobit.client.bootstrap.DiscobitSettings <init>INFORMATION: [discobit] configuring server from system: http://127.0.0.1:8089Apr 20, 2015 1:42:33 PM com.v3rticle.oss.discobit.client.bootstrap.DiscobitSettings <init>

INFORMATION: [discobit] configuring server from system: http://127.0.0.1:8089Apr 20, 2015 1:42:33 PM com.v3rticle.oss.discobit.client.DiscobitConnector authenticate

INFORMATION: [discobit] authentication response: HTTP/1.1 200 OKApr 20, 2015 1:42:33 PM com.v3rticle.oss.discobit.client.DiscobitConnector authenticateINFORMATION: [discobit] authentication cookie: [[version: 0][name: JSESSIONID][value: 65452FC130FE2F87E8655BF05D5F0A90][domain: 127.0.0.1][path: /][expiry: null]]Apr 20, 2015 1:42:34 PM com.youngculture.ecom.hybris.discobit.HybrisDiscobitAgent executeINFORMATION: injected 677 properties to system contextWrapperManager: Initializing...

Apr 20, 2015 1:42:39 PM org.apache.tomcat.util.digester.SetPropertiesRule beginWARNUNG: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'disableURLRewriting' to 'true' did not find a matching property.Apr 20, 2015 1:42:40 PM org.apache.tomcat.util.digester.SetPropertiesRule beginWARNUNG: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'disableURLRewriting' to 'true' did not find a matching property.Apr 20, 2015 1:42:40 PM org.apache.coyote.AbstractProtocol init

INFORMATION: Initializing ProtocolHandler ["http-bio-9001"]Apr 20, 2015 1:42:40 PM org.apache.coyote.AbstractProtocol init

INFORMATION: Initializing ProtocolHandler ["http-bio-9002"]Apr 20, 2015 1:42:45 PM org.apache.coyote.AbstractProtocol initINFORMATION: Initializing ProtocolHandler ["ajp-bio-8009"]

Apr 20, 2015 1:42:45 PM org.apache.catalina.startup.Catalina loadINFORMATION: Initialization processed in 5834 ms

Apr 20, 2015 1:42:45 PM org.apache.catalina.mbeans.JmxRemoteLifecycleListener createServerINFORMATION: The JMX Remote Listener has configured the registry on port 9003 and the server on port 9004 for the Platform serverApr 20, 2015 1:42:45 PM org.apache.catalina.core.StandardService startInternal

INFORMATION: Starting service Catalina

Key Take Aways

Key Take Aways •  Simple, unified approach to avoid wide-spread

configuration and their mutation

•  Language & product agnostic

•  Enabling other teams and colleagues to take controlled influence on configs in general

•  Needs a clear strategy concerning delivery cycle, security and infrastructure

Usecases •  Configure applications in hosting enviroments,

test farms, desktops

•  Developers share configurations while creating a solution

•  QA configures their testing systems individually

•  Dark launching of software changes

•  Feature toggling for indidual usage profiles

•  Central administration

•  Create multi-level configuration inheritance

•  Turn from a push-based model to a consumer pull-based self service

•  Allow fine grained access to (parts) of configurations for different stakeholder roles

•  Change properties without rollouts (trigger reload events instead)

Cons (yes there are)

•  Accessibilty from environment to CCR needs to be resilient, fallbacks needed (e.g. local caching).

•  Configuring during runtime can have unforseen side effects in your application (check lifecycle).

•  Administrative access to configurations needs to be well thought.

Where to go next? Potential Extensions

•  Tracking changes possible (who did what and when)

•  Add versioning by storing configs in an SCM for rollbacks ore regression testing

•  Stage changes for different rollout scenarios

•  Add hierarchical config structures like YAML

Questions?

Thanks! Mail: j.saade@youngculture.com

Twitter: @jenssaade

youngculture.com

Hybris.com

discoBit.com

Appendix

Continuous Delivery Hosting

QA Testing

Jenkins CI

Embedding into Continuous Delivery Flow

discoBit

Application Instance 2

UUID 2

Application Instance n

UUID nConfig

Repository

REST API

Artifactory

Distributable Artifact

Acceptance Stage

Deploy Stage

deploybuild configure

Commit Stage

env.properties

project.properties

advanced.properties

Continuous Delivery

QA Testing

Source (git)

Jenkins

Stacked Configurations in the Central Config Repo

Commit Stage

env.properties

discoBit

Acceptance Stage

Deploy Stage

deploybuild configure

Automated space

Shared space

Customized (env2)

Customized (env1)

Devs’ Machine

project.properties

advanced.properties

pimp your pipeline - central configuration management - jens saade

project specic236 project

delivery pipeline

dev delivery processes

complex project environment

hybris platform

yc youngculture ecom

joint team eort

hard work

Engineering

pimp my mac reloaded

being a pimp

pimp my widget!

pimp my life

pimp my business

pimp my java lavajug

pimp mylesson oseboldt

pimp my slide

pimp my community

pimp my pacifica

pimp my site

pimp my slides

pimp my plone

pimp my mike 2

pimp your linkedin profile

the pimp game

pimp up your profile

pimp my salad

pimp my front

pimp your portfolio legally