Achieving Continuous Delivery - Micro Services

- Vikram Gadang

Agenda● Starting point● Observations and lessons learned● Architecting for CD● Build pipeline strategy● Testing strategy● Deployment strategy

State of affairs● Manual process (QAR) - Creating deployable artifacts

● Manual build process to QA & Prod

● Manual regression tests followed up by actual feature tests

● Fix regression bugs and/or feature bugs and repeat build

● Move build artifact to stage env and repeat manual testing

● Install the build to production during off-peak hours (2am - 4am)

Initial Attempt● Use Jenkins as CI server to create a build pipeline

● Manually triggered build, test & deploy

● Write test cases that tested functionality end-to-end

ObservationsLong build & test time● Most testing involve integration testing

● Tests not very reliable due to backend dependencies

ObservationsMonolithic architecture● Large and complex codebase

● Hard to scale & deploy

● Long-term commitment to a technology

ObservationsCode coverage by tests.. 30% or less● High cyclomatic complexity. 10 or more per function (too many

if/then/else, switch, try/catch statements in code)

● Entangled code

● Complex dependency mesh

Micro Services Architecture

Monolithic architecture puts all functionality in one process

Microservices architecture breaks down functionality into many processes

A B C D E

A

A

B

B

C

C

E

E

D

A B

C D

E

B

Monolithic Architecture

Browser

Tomcat

WAR

StoreFront UI

Catalog Service

Profile Service

Billing Service

Database

Remote Services

Micro Services Architecture

StoreFront UI

Catalog Service

Profile Service

Billing Service

Database

Remote Service

Micro Services - Pros

● Divide and Conquer. Break down complex problems into smaller parts that can be solved independently

● Small codebase. Small enough to fit in your head● Enables frequent deployments● Enables polyglot programming & persistence● Enables independent scalability● No long-term commitment required. Rewrite over

maintain

Micro Services - Hard to do

● Dealing with complexity of the system as a whole● Monitoring● Deployment in a container or outside a container● Avoiding duplication - Hint: move common code to

libraries● Service discovery● Simple != Easy

Basic Requirements for CD● Make Dev, QA, Stage & Prod environments similar● Zero-downtime deployments● Testing and deployment are integral parts of delivery● Build quality in● Automate everything

○ Build○ Unit tests○ Integration tests○ Code coverage and quality analysis○ Deployment○ Acceptance tests

Basic Requirements for CD● Ability to scale out individual components● Stateless systems● Convention over configuration● Commit early and often. Test early and fail fast● All triggers are data-driven. (Measure everything)

Delivery Pipeline

Implementing the pipeline● Started slow, added components over time

● Build → Test → DeployDev → Test → DeployQA → Test → DeployStage → Test → Deploy Prod → Test

● Continuously improve the pipeline

Delivery Pipeline

Commit

Build & Unit Test

Integration Test

Deploy DevAcceptance Test

Publish Artifact

QA

Deploy Dev

Acceptance Test

Staging

Deploy Dev

Acceptance Test

Deploy Dev

Acceptance Test

Build & Unit Test

Integration Test

Acceptance Test

Publish Artifact

Deploy QA Acceptance Test

Deploy Stage

Acceptance Test

Production

Deploy Prod

Acceptance Test

Code Analysis

Quality Gate

Pipeline Implementation

● Jenkins - CI Server

● Gradle - Build tasks

● Nexus - Repository

Pipeline View (Jenkins)

Technology Stack● Centos 6.4+

● JVM 7+

● Tomcat 7+

● Spring Boot

● Mongo DB

● MySQL DB

Source Code Strategy● Minimize pre-requisites to get started

○ Only JDK 7+

● Start with a walking skeleton○ Metrics and health stats○ Unit, integration and acceptance tests○ Security○ Logging○ Build and deployment scripts

Source Code Strategy● Incremental development with frequent commits

● All commits go into master and are continuously integrated and tested leading to faster feedback

● Developers pick each others changes immediately

● Branch for release, branch for feature more often leads to merge/integration hell at end of sprints

● Feature flags to enable/disable features

● Write less code

More code = more testspublic void doPost(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {

StringBuffer jb = new StringBuffer(); String line = null; try { BufferedReader reader = request.getReader(); while ((line = reader.readLine()) != null) jb.append(line); } catch (Exception e) { /*report an error*/ }

try { JSONObject jsonObject = JSONObject.fromObject(jb.toString()); } catch (ParseException e) { // crash and burn throw new IOException("Error parsing JSON request string"); }

// Work with the data ...

// Return response HistoryResponse historyresp = historyService.writeHistory(history); JSONObject respjson = convertToJson(historyresp); response.setContentType("application/json"); response.getWriter().write(respjson.toString()); }

Less code is better@RequestMapping(value="/history", method=POST)public HistoryResponse writeHistory(@RequestBody History history ) {

return historyService.writeHistory(history);

}

Testing strategy● Unit testing in 5 mins or less - fast feedback

● Integration testing in data access and remote access layer

● Acceptance tests after unit & integration tests

● Automated code review & quality analysis.

● Builds fails if any of above fail

Testing● JUnit - Java Unit & Integration Tests

● Mockito - Mocking framework

● QUnit - Javascript unit tests

● Selenium - Acceptance & Web tests

● JMeter - Acceptance & Performance

Testing strategy

Real world scenario

Code Quality Strategy● Enforce minimum code coverage by tests

○ Overall coverage (line & branch) 90% or more○ New code coverage (line & branch) 90% or more

● Enforce low cyclomatic complexity (brain overload & testability)○ 10 per method. (if, while, do, for, &&, ||, ?:, catch, switch, case, return,

throw)

● Enforce code quality○ Error Handling○ Style problems○ Ignoring conventions or skipping unit tests

Code Quality Tools● Jacoco - Java code coverage

● Saga - JS code coverage

● PMD - Code analysis

● FindBugs - Potential issue finder

● JDepend - Code complexity

● SonarQube - Ties it all together

SagaJS Code Coverage

Measure Code Quality (Jenkins)

Measure Code Quality (SonarQube)

Measure Code Quality

Measure Code Coverage

Measure Code Quality

Deployment Strategy● Automate deployment using Chef● Systems built for zero downtime deployment● Rollback is just another deployment by a single click● Do not make any changes directly in production

environment● Emergency releases go through standard deployment

pipeline

Deployment Tools

● Jenkins - CI Server

● Nexus - Repository

● Chef - Deployment

Monitoring● New Relic - Monitoring platform

● Http/JMX monitoring - Health and metrics

● Java Melody - Collection of stats

● Elastic Search/Logstash/Kibana - Logs collection

ARGUS/TOGA

Monitoring (New Relic)

API DocumentationAutomate API documentation with Swagger UI https://github.com/wordnik/swagger-ui

API Documentation

Takeaways● If it hurts, do it more often

● CD forces you to do the right thing

● You cannot do it without automating the build, test, deploy and release process

● Measure everything. Make no assumptions

● Things go better if dev and ops are friends

How long would it take your organization to deploy a change that involved just one single line of code?

- Mary & Tom Poppendieck

How long would it take your organization to deploy a change that involved just one single line of code? Do you deploy changes at this pace on a repeatable, reliable basis?

- Mary & Tom Poppendieck

References● http://martinfowler.com/books/continuousDelivery.html

● http://martinfowler.com/articles/microservices.html

● http://microservices.io/patterns/microservices.html

● http://www.infoq.com/presentations/cd-gradle-jenkins

● https://speakerdeck.com/bmuschko/building-a-continuous-delivery-

pipeline-with-gradle-and-jenkins

● http://projects.spring.io/spring-boot

● http://12factor.net

Questions?

vikram.gadang@teamaol.com