@mockitoguy@LinkedInEng

You deserve great tools!Commit-to-production pipeline at LinkedIn

Agenda:1. Continuous delivery in the Open Source - 5min2. linkedin.com - 3 releases per day, 3 hours time from

commit to production - 15min3. Commit to Production pipeline - 20min4. Q&A - 5min

Szczepan Faber● Originally from Poland, in US since 2015● Creator of Mockito framework in 2007● Core engineer of Gradle in 2011-2015● Tech Lead at LinkedIn Development Tools

team since 2015● @mockitoguy on Twitter● Author on LinkedIn

@mockitoguy@LinkedInEng

1.Continuous Delivery in the Open Source, lessons

learned from Mockito

@mockitoguy@LinkedInEng

● Project started in 2007● Easy manual releases

2008 - Mockito 1.0

@mockitoguy@LinkedInEng

Mockito popularity

@mockitoguy@LinkedInEng

2014: Mockito & Continuous Delivery

● Every merged pull request lands new version in Maven Central

@mockitoguy@LinkedInEng

The bright side of CD in OSS

● Productivity – zero release overhead● Happy users – get features faster● Faster debugging – quickly identify

bad version (MTTR)● Sustainability – release & stay alive● No waste - no unreleased code● Quality – self-enforced craftsmanship

of every change● Thriving, engaged community –

contributions are released quickly

@mockitoguy@LinkedInEng

Feedback from community

● Quality anxiety● What version to use?● Dependency Management cost

@mockitoguy@LinkedInEng

http://shipkit.org

● Mockito uses Shipkit● Toolkit for shipping it

@mockitoguy@LinkedInEng

And also:○ Rapid innovation○ Competitive advantage

● Productivity● Happy users● Faster debugging● Sustainability● No unreleased code● Quality

Continuous Delivery in the enterprise

@mockitoguy@LinkedInEng

2.linkedin.com - 3 releases per day, 3 hour time from

commit to production

@mockitoguy@LinkedInEng

Years before 2015

● monthly releases● feature rush● branch hell● manual verification

@mockitoguy@LinkedInEng

2015 at LinkedIn: 3x3● 3 hours max time from commit to production● 3 releases per day

@mockitoguy@LinkedInEng

Continuous delivery is hard!(what do we do if something is hard?)

● Flaky test is worse than no test● Production-grade tests and infrastructure● Improving Open Source test frameworks● Need for speed – going parallel and

distributed● Master branch always green

@mockitoguy@LinkedInEng

3x3 & engineering culture @LI● Engineering culture shift● It’s not about technology (but it plays a role)● Read more on LinkedIn eng blog

@mockitoguy@LinkedInEng

3.Commit to Production Pipeline

@mockitoguy@LinkedInEng

Several years ago at LinkedIn

● Monolithic codebase● We tried, but it didn’t scale● Decision to break free from Mono

@mockitoguy@LinkedInEng

Monorepo vs. Multirepo

● It’s not about how you keep the code but how you ship it

● Monorepo vs. Multirepo● Monolith vs. Federation

@mockitoguy@LinkedInEng

Monolith vs. Federation

Monolith - central government● All software components are developed, tested and

published at the same time● No versioning between internal components● Unified versions of external dependencies

Federation - autonomy within ecosystem● Every software component owns its release cadence● Individual versioning of software components● Freedom to choose dependency version

@mockitoguy@LinkedInEng

Federated software components

● A codebase at LinkedIn is called “Multiproduct” (MP)

● Commit to Production in MP is standardized

● Easy contributing to codebases (GitHub analogy)

@mockitoguy@LinkedInEng

Commit to Production in Multiproduct● Code change● Code review

(strong code ownership)● Pre/post push validation● Depender testing● New version

Library:● Update consumers● Deprecate/end-of-life

previous versions

Application:● Staging● Canary● Ramp-up features● Remove experiment

@mockitoguy@LinkedInEng

THANK YOU! Talks at JavaOne ‘17You deserve great tools!!!● Now: Commit-to-production pipeline at LinkedIn

○ This talk. Hope you liked it :)

● Tomorrow: Play on Gradle - Midtier behind linkedin.com○ Lessons learned from driving disruptive change in large org○ Replacing the heart of Play Framework with Gradle at scale

● Tomorrow: Mockito 2 Cleaner Tests and Enhanced Productivity○ Do you want to write cleaner tests faster?

@mockitoguy@LinkedInEng

Thank you!!! Questions?Commit-to-production pipeline at LinkedIn

● Productivity● Happy users● Faster debugging● Sustainability● No unreleased code● Quality● Rapid innovation http://shipkit.org

Toolkit for shipping it used by Mockito

3x3 Continuous delivery at LinkedIn:● 3 hours commit to production● 3 releases per day● Standardized workflow● Commit-to-production pipeline

Szczepan Faber● Originally from Poland, in US since

2015● Creator of Mockito framework in 2007● Core engineer of Gradle in

2011-2015● Tech Lead at LinkedIn Development

Tools team since 2015● @mockitoguy on Twitter● Author on LinkedIn

● Tomorrow: Play on Gradle - Midtier behind linkedin.com● Tomorrow: Mockito 2 Cleaner Tests and Enhanced Productivity