pimp your pipeline - central configuration management - jens saade
TRANSCRIPT
Pimp Your Pipeline
Central Configuration Management
Jens Saade, youngculture April 25th 2015, Iasi CodeCamp Romania
ReferencesAbout youngculture Group
Jens Saade
Head Innovation & Technology E-Commerce Engineering Background (Retail, PIM, E-Commerce)Located in Zürich, CH
Dev Process Architecture Innova2on
Ini2a2ves
Central Configuration Repository
Appliance in Continuous Delivery
Automation & Configuration
Intro youngculture E-Commerce
Intro: E-Commerce
@YC
youngculture ECOM team delivers highly integrated E-Commerce Solutions based on the Hybris Platform.
Customized B2C Storefront Showcase
B2C & B2B Shops and PIM Systems
Delivering good quality in a highly complex project environment is only possible by aligning team, processes and infrastructure environments.
Con2nuous Delivery
Configura2on Mgmt
Test Automa2on
Cloud Services
Automated Provisioning
Workspace (Confluence,
JIRA)
Prac2ce Meetups
Integrated Communica2on
(HipChat)
System Monitoring
HW/OS (NewRelic, Nagios)
Applica2on Monitoring
Logs (Elas2cSearch/Kibana, Splunk)
Performance (Dynatrace),
Delivery Environment
EnvironmentsCAM
Collaborate
AutomateMonitor
TeamsSystems
Projects
Automation
Continuous Delivery
Familiar with the Concepts? Tried applying it in own Company?
Part of your Culture?
AUTOMATION is hard work ...expensive to establish...and can get pretty complex
BUT
... reduces monkey work... speeds up the dev & delivery processes
... calms down stress level ...and raises confidence and quality.
Release anytime, reproducible
Create Hybris
Projects, Extension
and Addons
SetupContinously
Build deployable
artifacts
BuildAssess
Software by Automated
Testing
Test
Continuous Delivery Overview
Install for testing,
acceptance and
production
Deploy
> Commit Stage > Acceptance Stage > Deployment Stage
Feedback Feedback Feedback
Establishing a Delivery Pipeline is a joint team effort! It involves lots of disciplines and evolves constantly.
CustomersDevelopers Product Owner
Scrum Master
Delivery Process: Automation is Key
Test Execution EnvJenkinsCI
Artifactory Repository
GitRepository
Hybris Node
YC Deploy Client
Apache HTTPdSelenium
GatlingJMeter
Test Exec Hybris Test
Commit Stage
Acceptance Stage
Deploy Stage
Continuous Delivery
Monitoring Node
Stage/Production
Hybris Node1
Hybris Cluster
Apache HTTPd
Varnish
Apache HTTPd
Varnish
DB Cluster
Load Balancer
ElasticSeach
Logstash
Kibana
Zabbix
External Monitoring
NewRelic
Hybris Node 2
YC Deploy Client
YC Deploy Client
Hosting Environment
Team
sIn
frast
ruct
ures
Maintenance team
One essential problem left:
Configurations
Ecosystem of Configurations
Versioned Software Artifact
Dev 1
Dev 2
Ops 1
Ops2
QA
Test specifc artifact
versions for regressionQA on
features
Create new Conf
PropertyShare his conf with
Dev 2
Prepare RolloutConf for tonight
Enable feature for customer
Req Eng
Introduce new concept, define triggers
A=1
B=truec=debug
7Env.properties
? Project Specific
236
Project.properties
436Advanced.properties
7Env.properties
? Project Specific
236
Project.properties
436Advanced.properties
Node 1
7Env.properties
? Project Specific
236
Project.properties
436Advanced.properties
Node 2
7Env.properties
? Project Specific
236
Project.properties
436Advanced.properties
Node 3
1 Clustered Environment
Dev(s)QA SystemsProductionUAT/Stage...
Hybris Configuration Stack (Properties based)
(manual) Configuration
Central Configuration Management
Diversity: Support complex environment setups
!=
We need good concepts how to configure environments nearly equivalent
but allow flexibility for individual deviations
„The most efficient way to manage configuration is to have a central service through which every application can get the configuration it needs.“
Jez Humble and Dave Farley „Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation“, Addison-Wesley 2010
Artifact
Generic Configuration
Machine
Java VM
Rolling out Configurations
Configurations get packaged and rolled out
JVMs are bootstrapped from packaged configurations
Machine
Java VM
Machine
Java VM
Machine
Java VM
Environment Configuration
Turning from distributing (and manually tweaking) local configurations ...
... to a centralized environment that hosts configurations for self-service.
Direction change: Push -> Pull
Artifact
Machine
Java VM
Rolling out Configurations, alternative
Rollout Software Artifact as usual, but keep it generic, just provide configuration IDs
Isolate configurations from the deployment package and provide it centrally from the CCR
Machine
Java VM
Machine
Java VM
Machine
Java VM
UUID
Generic Configuration
CCR
Environment Configuration
UUID
UUID UUID
UUID
UUID
Generic Configuration
Environment Configuration
Hierachical Key-Value Store
Configuration
Configuration
UUID3
UUID4
Machine
Application
Client Wrapper
Central Configuration Repository (CCR) and Consuming Clients
Agent pulls configuration properties via REST / Json
Then injects properties to JVM System context
MachineMachineMachine Machine
Machine
Machine
Machine
MachineMachineMachineMachine
Machine
Machine
Machine
Machine
Machine
REST / Json
discoBit
API
Admin UI
CCR
CD & CCR Applied @YC
Configuration Hierarchies
Automated space
UUID1
Shared space
UUID2
UAT Environment
UUID7
UAT node 1
UUID8
UAT Node 2
UUID9
QA: Nightly Build Test System
UUID6
Personalized Conf Developer A
UUID3
Personalized Conf Developer B
UUID5
Ops 1
Ops2
QA
Dev 1
Dev 2
Req Eng
Continuous Delivery Cycle
Customized (env1)
UUID3
Customized (env2)
UUID4
Machine
Java VM
discoBit Java Agent (UUID4)
System Context Injection via Java Agent
discoBit
Agent pulls configuration properties via REST / Json
Then injects properties to JVM System context
Automated space
UUID1
Shared space
UUID2
jsaade@MacBook-Pro:~/hybris/hybris-commerce-suite-5.4.0.0/hybris/bin/platform$ ./hybrisserver.sh Launching a JVM...Apr 20, 2015 1:42:33 PM com.v3rticle.oss.discobit.client.bootstrap.DiscobitSettings <init>INFORMATION: [discobit] configuring server from system: http://127.0.0.1:8089
Apr 20, 2015 1:42:33 PM com.youngculture.ecom.hybris.discobit.HybrisDiscobitAgent executeINFORMATION: Reading discobit configuration from http://127.0.0.1:8089::bb2835fb-466c-4143-8081-ffcd60c49467
Apr 20, 2015 1:42:33 PM com.v3rticle.oss.discobit.client.bootstrap.DiscobitSettings <init>INFORMATION: [discobit] configuring server from system: http://127.0.0.1:8089Apr 20, 2015 1:42:33 PM com.v3rticle.oss.discobit.client.bootstrap.DiscobitSettings <init>
INFORMATION: [discobit] configuring server from system: http://127.0.0.1:8089Apr 20, 2015 1:42:33 PM com.v3rticle.oss.discobit.client.DiscobitConnector authenticate
INFORMATION: [discobit] authentication response: HTTP/1.1 200 OKApr 20, 2015 1:42:33 PM com.v3rticle.oss.discobit.client.DiscobitConnector authenticateINFORMATION: [discobit] authentication cookie: [[version: 0][name: JSESSIONID][value: 65452FC130FE2F87E8655BF05D5F0A90][domain: 127.0.0.1][path: /][expiry: null]]Apr 20, 2015 1:42:34 PM com.youngculture.ecom.hybris.discobit.HybrisDiscobitAgent executeINFORMATION: injected 677 properties to system contextWrapperManager: Initializing...
Apr 20, 2015 1:42:39 PM org.apache.tomcat.util.digester.SetPropertiesRule beginWARNUNG: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'disableURLRewriting' to 'true' did not find a matching property.Apr 20, 2015 1:42:40 PM org.apache.tomcat.util.digester.SetPropertiesRule beginWARNUNG: [SetPropertiesRule]{Server/Service/Engine/Host/Context} Setting property 'disableURLRewriting' to 'true' did not find a matching property.Apr 20, 2015 1:42:40 PM org.apache.coyote.AbstractProtocol init
INFORMATION: Initializing ProtocolHandler ["http-bio-9001"]Apr 20, 2015 1:42:40 PM org.apache.coyote.AbstractProtocol init
INFORMATION: Initializing ProtocolHandler ["http-bio-9002"]Apr 20, 2015 1:42:45 PM org.apache.coyote.AbstractProtocol initINFORMATION: Initializing ProtocolHandler ["ajp-bio-8009"]
Apr 20, 2015 1:42:45 PM org.apache.catalina.startup.Catalina loadINFORMATION: Initialization processed in 5834 ms
Apr 20, 2015 1:42:45 PM org.apache.catalina.mbeans.JmxRemoteLifecycleListener createServerINFORMATION: The JMX Remote Listener has configured the registry on port 9003 and the server on port 9004 for the Platform serverApr 20, 2015 1:42:45 PM org.apache.catalina.core.StandardService startInternal
INFORMATION: Starting service Catalina
Key Take Aways
Key Take Aways • Simple, unified approach to avoid wide-spread
configuration and their mutation
• Language & product agnostic
• Enabling other teams and colleagues to take controlled influence on configs in general
• Needs a clear strategy concerning delivery cycle, security and infrastructure
Usecases • Configure applications in hosting enviroments,
test farms, desktops
• Developers share configurations while creating a solution
• QA configures their testing systems individually
• Dark launching of software changes
• Feature toggling for indidual usage profiles
Pros
• Central administration
• Create multi-level configuration inheritance
• Turn from a push-based model to a consumer pull-based self service
• Allow fine grained access to (parts) of configurations for different stakeholder roles
• Change properties without rollouts (trigger reload events instead)
+++
+
+
Cons (yes there are)
• Accessibilty from environment to CCR needs to be resilient, fallbacks needed (e.g. local caching).
• Configuring during runtime can have unforseen side effects in your application (check lifecycle).
• Administrative access to configurations needs to be well thought.
-‐
-‐
-‐
Where to go next? Potential Extensions
• Tracking changes possible (who did what and when)
• Add versioning by storing configs in an SCM for rollbacks ore regression testing
• Stage changes for different rollout scenarios
• Add hierarchical config structures like YAML
Questions?
Appendix
Continuous Delivery Hosting
QA Testing
Jenkins CI
Embedding into Continuous Delivery Flow
discoBit
Application Instance 2
UUID 2
Application Instance 1
UUID1
Application Instance n
UUID nConfig
Repository
REST API
Artifactory
Distributable Artifact
Acceptance Stage
Deploy Stage
deploybuild configure
UI
1
2
3
Commit Stage
env.properties
project.properties
advanced.properties
Continuous Delivery
QA Testing
Source (git)
Jenkins
Stacked Configurations in the Central Config Repo
Commit Stage
env.properties
discoBit
Application Instance 1
UUID3
Acceptance Stage
Deploy Stage
deploybuild configure
Automated space
UUID1
Shared space
<extends>
UUID2
UUID1
<extends>
Customized (env2)
Customized (env1)
UUID3
UUID4
Devs’ Machine
Application Instance 3
UUID4
Application Instance 2
UUID3
1
2
project.properties
advanced.properties
merge