russ barnett, chief architect john esser, director of ... · problems with large batches...
TRANSCRIPT
![Page 1: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/1.jpg)
Architecting for Continuous Delivery
Russ Barnett, Chief Architect John Esser, Director of Engineering Productivity
Ancestry.com
![Page 2: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/2.jpg)
Background
2
![Page 3: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/3.jpg)
Growth at Ancestry.com
• Subscribers have doubled in the past 3 years – (~1M to > 2M)
• Page views have doubled in the past 3 years – (~25M/day to ~50M/day)
• Development head count has tripled – (100 to 300)
• Feature throughput has dramatically increased
3
![Page 4: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/4.jpg)
Continuous Delivery
is consistently and reliably
releasing business value increments
fast
through automated build, test, configuration and deployment.
What is the value of going fast?
A LOT!
![Page 5: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/5.jpg)
Evolution to Continuous Delivery
Agile Boot Up (Scrum)
Enterprise Agile Framework
Agile Architecture Standards
Continuous Delivery Adoption
Future – “Agile v2”
Two year period (2010 – present)
![Page 6: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/6.jpg)
Typical Impediments to Continuous Delivery
• Cultural
• Technical practices
• Quality ownership
• Infrastructure
• Architectural
![Page 7: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/7.jpg)
Limiting Factors
• Pipeline serialized at integration
– Errors that occurred in this stage stalled the pipeline
• Stalls in integration induced additional problems
• Increasing frequency of stalls
– As number of development teams grew, frequency of stalls increased
7
Integration
Source Control
Build
Unit Testing
System Testing
Deployment
Team A Team B Team C
error
causes
stall
![Page 8: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/8.jpg)
Everything was coupled!
(Aka, large batch size)
(It became known as the “big blob!”)
8
![Page 9: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/9.jpg)
9
![Page 10: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/10.jpg)
Little’s Law
𝑊𝑎𝑖𝑡 𝑇𝑖𝑚𝑒 =𝑄𝑢𝑒𝑢𝑒 𝑠𝑖𝑧𝑒
𝑃𝑟𝑜𝑐𝑒𝑠𝑠𝑖𝑛𝑔 𝑅𝑎𝑡𝑒
𝑄𝑢𝑒𝑢𝑒 𝑠𝑖𝑧𝑒 ∝ 𝐵𝑎𝑡𝑐ℎ 𝑠𝑖𝑧𝑒
We can reduce wait time (cycle time) by reducing batch size
without changing demand or capacity.
10
![Page 11: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/11.jpg)
Problems with Large Batches
• Increases cycle time
• Increases variability non-linearly as 2n
• Increases risk
• Reduces efficiency
• Limited by its worst element.
from Principles of Product Development Flow, Don Reinertsen
11
![Page 12: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/12.jpg)
Answer?
Utilize small batches.
12
![Page 13: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/13.jpg)
Fluidity Principle
Loose coupling between product systems
enables small batches
“Once a product developer realizes that small batches are desirable, they start adopting product architectures that permit work to flow in small, decoupled batches.”
from Principles of Product Development Flow, Don Reinertsen
13
![Page 14: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/14.jpg)
Fluidity Principle
Loose coupling between product systems
enables small batches
“Once a product developer realizes that small batches are desirable, they start adopting product architectures that permit work to flow in small, decoupled batches.”
from Principles of Product Development Flow, Don Reinertsen
14
![Page 15: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/15.jpg)
Creating an Architecture for Agility
15
![Page 16: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/16.jpg)
Architectural Impediments
• Cross-Component Coupling
– Creates groups of systems that must be deployed together
• Insufficient Rollback Capability
– Causes teams to resort to cascading rollback
• Poor Testing and Monitoring
– Requires a long testing period
– Lengthens feedback cycle
– Allows quality problems to escape to and affect customers
16
![Page 17: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/17.jpg)
Architectural Methods for Removing Impediments
• Partition into small single-responsibility components “There should only be one reason for a [component] to change”
- Robert Martin
• Decouple deployment of components
– Separately deploy components
– Remove order dependent deployment
• Support Independent Rollback
– Enforce strict backward and forward compatibility
17
![Page 18: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/18.jpg)
Backward and Forward Compatibility
Client
Version 1
Service
Version 1
Service
Version 2
Client
Version 2
Deployment
Rollback
Deployment
Rollback
Backward Compatibility
Backward Compatibility
Forward Compatibility
Forward Compatibility
• Server Backward Compatibility
– Newer servers work with clients written to old interface
• Server Forward Compatibility
– Existing servers work with clients written to newer interface
– Supports early client deployment
• Client Backward Compatibility
– Newer clients work with servers that implement old interface
– Supports server rollback
• Client Forward Compatibility
– Old clients work with servers that implement new interface
![Page 19: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/19.jpg)
Enforcing Decoupled Components
• Implementing standards is insufficient
– Independent deployment forces some decoupling
– High rate of deployment issues indicate remaining coupling
• Improve integration testing
– Verify backward and forward compatibility
– Identify breaking changes quickly
– Make writing integration tests easier
19
![Page 20: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/20.jpg)
Improving Interface Verification
20
Server
Client
In Process
Client/Server
Server
Client • Remember when you could run your
entire application in one process?
• How do we get better interface verification with services?
In Process
Thrift
SOAP/WSDL
REST
Veri
fiable
Decoupled
Google Protocol Buffers
![Page 21: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/21.jpg)
Interface Verification using Proxies and Stubs
21
Stub
Proxy
Client
Server
Client/Server with
Proxy/Stub
• Verifies interface at compile time
• Isolates code from versioning issues
• Easier to provide mock implementations
• Can test backward and forward compatibility
![Page 22: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/22.jpg)
Managing a Complex Network of Services
• Ancestry Scale
– About 40 different teams
– Over 300 separate application or service systems
– Stack is 5+ levels deep
• Historical Diagnostics
– Presented a client centric or top down view
– Insufficient for identifying problems in a network of services
• Solution: Deep Status Check
– Components provide dynamic status information for each client and dependency
– Report traverses dependencies up to a given depth
22
![Page 23: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/23.jpg)
Ancestry Deep Status Check
23
Service
1
Service
A
Service
C
Service
B
App 2
Service
2
GetStatus Monitor App 1
Unidentified Client
Unregistered Client
Connection Error
Database File
System
Level 1
Level 2
Level 3
Level 4
![Page 24: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/24.jpg)
< 2 s
95%
< 4 s
fail 99.9%
SLA
Service Level Agreements
• Understand Business Expectations
– Each application and service establishes a contract with each client specifying the expected performance characteristics
24
• Methodology
– Use percentile buckets rather than an average
– Performance is a component of availability
![Page 25: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/25.jpg)
Verify Client Identity Check Circuit Breaker
Track Each Incoming Request Report Compliance
Incoming Requests
Check Circuit Breaker Provide Fallback Mechanism Track Each Outgoing Request
Report Compliance
Outgoing Requests
Service Level Agreement System
• Compare incoming and outgoing SLA compliance
Service
![Page 26: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/26.jpg)
Conclusion
• Architecture affects agility and continuous delivery capability as much or more than other factors.
• Process and tool improvements alone are insufficient.
• Good architecture techniques enable effective continuous delivery at large scale.
– Partition to single-responsibility components.
– Decouple deployment
– Support independent rollback
– Improve testing and monitoring infrastructure
26
![Page 27: Russ Barnett, Chief Architect John Esser, Director of ... · Problems with Large Batches •Increases cycle time •Increases variability non-linearly as 2n •Increases risk •Reduces](https://reader034.vdocument.in/reader034/viewer/2022050105/5f43c6a4a8444a2ed15be788/html5/thumbnails/27.jpg)
• Questions?
Contact info:
Ancestry is hiring in San Francisco and Utah
27