building a modern enterprise soa at linkedin
TRANSCRIPT
![Page 1: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/1.jpg)
Building a Modern Enterprise SOA at LinkedIn
![Page 2: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/2.jpg)
Agenda
Building Code at LinkedIn
Product Development with Multiproduct
Build Automation with Gradle
A Peek at the Future
©2013 LinkedIn Corporation. All Rights Reserved. 2
![Page 3: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/3.jpg)
Building Code at LinkedIn
©2013 LinkedIn Corporation. All Rights Reserved. 3
![Page 4: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/4.jpg)
In the beginning there was Network
Single, relatively homogenous code base
– Build from source, little dependency management
Java, Spring, Ant, …
JavaScript, HTML, JSPs, CSS, …
JARs, WARs, Jetty, Tomcat, …
©2013 LinkedIn Corporation. All Rights Reserved. 4
![Page 5: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/5.jpg)
Then everything went exponential
Number of developers, programming languages, build
systems, frameworks, lines of code, servers, users, page views, …
… and pretty much everything else
©2013 LinkedIn Corporation. All Rights Reserved. 5
![Page 6: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/6.jpg)
![Page 7: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/7.jpg)
Hypergrowth isn’t all fun and games
Acquisitions came in with their own technology and processes
Releases became more and more painful
Productivity and stability suffered
©2013 LinkedIn Corporation. All Rights Reserved. 7
![Page 8: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/8.jpg)
Scaling a software development organization
Requires sensible code and dependency management
– Source code APIs
– Service APIs
– Versioned dependencies
Performance is king
– Iterative improvements
Divide and conquer
– Split and isolate failures
©2013 LinkedIn Corporation. All Rights Reserved. 8
![Page 9: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/9.jpg)
Code Isolation
![Page 10: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/10.jpg)
Network Trunk Development
All development in Network shifted to trunk
– No branches, no merging
Continuous releases from trunk
– Deploy multiple times per day
Work started on break up and clean up
– Migrating build logic to Gradle
©2013 LinkedIn Corporation. All Rights Reserved. 10
![Page 11: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/11.jpg)
Product Development with Multiproduct
©2013 LinkedIn Corporation. All Rights Reserved.
![Page 12: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/12.jpg)
Traditional Software Development
Use a well-established technology stack
– Homogeneity => Simplicity
To adopt a new technology:
– Requires “out of the box” thinking and effort
– Do a proof-of-concept implementation
– Present to decision makers to demonstrate ROI => get approval
Slow-moving by design
– New technology integration expensive
– Top-down management decisions used as barrier
©2013 LinkedIn Corporation. All Rights Reserved. 12
![Page 13: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/13.jpg)
LinkedIn Software Development
We don’t ever want to be in the box
– Technical experimentation and diversity encouraged
– Living on the bleeding edge, often defining it
The Need for Speed
– Pace of iteration
– Automation (the human is slow)
– Continuous delivery
Build versus buy
©2013 LinkedIn Corporation. All Rights Reserved. 13
![Page 14: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/14.jpg)
Chaos Theory
![Page 15: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/15.jpg)
Multiproduct
Toolset that is architected for a heterogeneous technology world
Agnostic to version control, build system and programming stack
– Future-proof
Abstracts common software tasks
– For example: “build”, “test”, “release”
Provide a default implementation, allow users to override
– i.e. Gradle w/ LI plug-ins
©2013 LinkedIn Corporation. All Rights Reserved. 15
![Page 16: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/16.jpg)
Key Concepts
Elevate tooling from artifact to product level
Metadata ties the tooling together
– Ivy
– Version and Build specification
Pluggable implementation of subsystems
Version management
Continuous automated delivery
©2013 LinkedIn Corporation. All Rights Reserved. 16
![Page 17: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/17.jpg)
©2013 LinkedIn Corporation. All Rights Reserved. 17
Source
ControlBuild System Deployment
![Page 18: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/18.jpg)
Version Management
End-of-life dates
– Graceful deprecation and upgrades
Push version upgrades to consumers
Dependency Reports
– What products depends on me
– What products do I depend on
©2013 LinkedIn Corporation. All Rights Reserved. 18
![Page 19: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/19.jpg)
Push My Upgrade
©2013 LinkedIn Corporation. All Rights Reserved. 19
![Page 20: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/20.jpg)
Tracking Upgrades
©2013 LinkedIn Corporation. All Rights Reserved. 20
![Page 21: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/21.jpg)
Continuous Delivery in Multiproduct
Automated pipeline triggered on developer change
– No other developer action needed
Publishing 10,000+ artifacts per day for 300+ products
– Mean time for a good commit: ~10 minutes
– Mean time counting failures: ~25 minutes
©2013 LinkedIn Corporation. All Rights Reserved. 21
![Page 22: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/22.jpg)
Continuous Delivery
©2013 LinkedIn Corporation. All Rights Reserved. 22
![Page 23: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/23.jpg)
©2013 LinkedIn Corporation. All Rights Reserved. 23
![Page 24: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/24.jpg)
Build Automation with Gradle
©2013 LinkedIn Corporation. All Rights Reserved. 24
![Page 25: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/25.jpg)
Why LinkedIn uses Gradle
Dependency resolution engine
Rich plug-in system w/ real programming language
– DSL has high learning curve, but powerful
Visions align
– Automation
– Continuous delivery
©2013 LinkedIn Corporation. All Rights Reserved. 25
![Page 26: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/26.jpg)
LinkedIn Gradle plug-ins
Customize built-in plug-ins for LinkedIn’s environment
– i.e. Java, Scala, War, FindBugs, Cobertura
Add custom artifact types
– For example database patches, static content, and Hadoop workflows
Create metadata for publishing and deployment tooling to consume
Elevate concepts from artifact to product level
©2013 LinkedIn Corporation. All Rights Reserved. 26
![Page 27: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/27.jpg)
Dependency Graph powered by Gradle
©2013 LinkedIn Corporation. All Rights Reserved. 27
![Page 28: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/28.jpg)
©2013 LinkedIn Corporation. All Rights Reserved. 28
End-of-life enforcement
![Page 29: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/29.jpg)
Our own gradlew: ligradle
Our own custom Gradle wrapper
Provisions Gradle and plug-ins
– Allows each product to define versions it uses
Provides lifecycle management for Gradle and plug-ins
– End-of-life
Usage data
– Used to track usage and discover problems
©2013 LinkedIn Corporation. All Rights Reserved. 29
![Page 30: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/30.jpg)
Usage Data
©2013 LinkedIn Corporation. All Rights Reserved. 30
![Page 31: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/31.jpg)
Source vs Binary Dependencies
Source offers incremental updates and flexibility intra-product
Binary offers stability and speed inter-product
There’s no right answer, but there are plenty of wrong answers!
©2013 LinkedIn Corporation. All Rights Reserved. 31
![Page 32: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/32.jpg)
Network migration to Gradle
3,600 build.xml files to convert
– Many of them with custom logic
300 developers to train
Performance targets
– 2x speed-up for clean builds
– 5x speed-up for incremental builds
©2013 LinkedIn Corporation. All Rights Reserved. 32
![Page 33: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/33.jpg)
Proof of Concept
Migrated 1,100 modules
Tested single large build versus isolated
segments
– Single large build simpler
Scale feasible
– Requires scalability and performance
work in Gradle core
©2013 LinkedIn Corporation. All Rights Reserved. 33
![Page 34: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/34.jpg)
Gradle features required for migration
Configuration on demand
– Only configure the task graph you need
Refactored cache logic for performance
– Task history
– Dependency descriptors
Candidate performance improvements
– Parallel configuration
– Daemon stores project model
– Daemon performs continuous up-to-date checks
©2013 LinkedIn Corporation. All Rights Reserved. 34
![Page 35: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/35.jpg)
![Page 36: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/36.jpg)
Daemon Heap Usage
©2013 LinkedIn Corporation. All Rights Reserved. 36
![Page 37: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/37.jpg)
Project Timeline – 1 year
Q1: Proof-of-concept and prep work
Q2: Implementation
Q3: Roll-out
Q4: Clean-up
©2013 LinkedIn Corporation. All Rights Reserved. 37
![Page 38: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/38.jpg)
A Peek at the Future
©2013 LinkedIn Corporation. All Rights Reserved. 38
![Page 39: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/39.jpg)
Gradle Features
Faster
– Use the daemon effectively in development and CI
– Intra-project parallel execution
More scalable
– Heap usage
Ease of use
– IDE integration
©2013 LinkedIn Corporation. All Rights Reserved. 39
![Page 40: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/40.jpg)
More Multiproduct Intelligence
Analytics
– Common exceptions
– Usage and error patterns
Verification suite
– CheckStyle, FindBugs and Cobertura
Automation and Integration
Ease of use
©2013 LinkedIn Corporation. All Rights Reserved. 40
![Page 41: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/41.jpg)
Distributed Build Automation
Distribute build and testing on a cluster
Automatic provisioning
Artifact sharing at scale
©2013 LinkedIn Corporation. All Rights Reserved. 41
![Page 42: Building a Modern Enterprise SOA at LinkedIn](https://reader036.vdocument.in/reader036/viewer/2022081401/5a6d00667f8b9aff418b4979/html5/thumbnails/42.jpg)
©2013 LinkedIn Corporation. All Rights Reserved. 42