speed up continuous delivery with bigdata analytics

#JenkinsWorld

Speed up Continuous Deliverywith BigData Analytics

Luca Milanesio

@gitenterprise @gerritreview #JenkinsWorld

• Founded in the UK• HQ in London, Offices in the UK and Italy• Committed to OpenSource

About GerritForge


• Co-founder and Director of GerritForge • over 20 years in Agile

Development and ALM• Contributor to many

OpenSource projects• Mind and hands of

@gitenterprise and @gerritreview Twitter feeds

About Luca Milanesio


NOT a lecture on Continuous Delivery

NOT a lecture on Code ReviewNOT a lecture on BigData

What NOT to expect?


SO WHAT?#!@$!


SHOW a Continuous Delivery scenario


LEARNtogether from its problems


EXPERIMENTa new approach using BigData


DISCUSSresults together and iterate


Continuous Delivery Pipeline

Push Build

Unit Tests

Push Build

Unit Tests

Automated

Tests

Push Build

Unit Tests

Automated

Tests

Beta Users

Tests

User's feedback Publish to all Users

Failed

Failed


Code Review & CD Validation

A1

refs/for/master

master

C2 C2*

A1 C2 (local)master

C2*

push

Verified: -1(build failed)

push

Verified: +1(build OK)

Reviewed: +2

CI Build: OK

C2*

CI Build: OK

amend


Git/Gerrit (Code Review)

Components

Jenkins (Continuous Integration & Delivery)


DEMO

See the pipeline in action


We use micro-services–100+ Components–200+ Repositories–300+ People and 100,000+ of review comments

–Cross-components [implicit] dependencies–Lots of build chains–10,000+ of builds

Now feel the pain :-(


Problem #1 : Tons of logs slows us down

"Build was so fast a week ago … why is it getting so slow now?"

100Ks of logs are tough for Jenkins


"It was working until 5 mins ago … then someone changed something"

Rings any bell?

Problem #2 : Ripple effect


In order to deliver Feature ABC I NEED

Ver. N of Service-A with Ver. M of Service-B and

Ver. K of Service C

Problem #3 : Dependencies


Release small changes is GOODRelease often every change is GOOD

BUTHow long it takes the "Big Picture" and End-to-

End?Where is my bottleneck?

Problem #4 : Cycle time


Problems #1 + #2 + #3 + #4 = DevOps meltdown

Problem #6 : all together


• Collect all review events• Collect all logs• Channel them to a central store• Crunch and Crunch continuously•Never delete• Process, inspect and learn

BigData to the rescue


Continuous Delivery – Analytics Pipeline

Data Aggregatio

n Jobs


Continuous Delivery – Analytics Dimensions

Projects Commits

People Reviews System Metrics


Continuous Delivery – Tools Dimensions

SCM / Code Review

CI / CD


DEMO

Continuous Delivery Analytics


•Collect more data (System, JVM, Review scores)•Real-time statistics•Events correlation•Feedback loop to Code-Review•Production data harvest•More?

Future directions


Want to know more?

www.gerritforge.com/contact


• GerritForge's blogwww.gitenterprise.me• Gerrit Code Review

www.gerritcodereview.com• Jenkins

www.jenkins.io• RabbitMQ

rabbitmq.com

Resources

• Apache Flumeflume.apache.org• Hadoop HDFS

hadoop.apache.org• Apache Spark

spark.apache.org• Kibana

github.com/elastic/kibana

speed up continuous delivery with bigdata analytics

Technology