right...ops testing with fabric making easy = right 36 rpm •does rpm have correct metadata linking...
TRANSCRIPT
![Page 1: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/1.jpg)
DevOps @ Wotif
Making Easy = Right
Alexandra Spillane • @ajbwMatt Callanan • @mcallana
RightEasy
![Page 2: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/2.jpg)
Outline
• What is “The Right Thing”?• How do we Make it Easy?• Lessons Learned
Making Easy = Right 3
![Page 3: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/3.jpg)
What is “The Right Thing”?DevOps @ Wotif: Making Easy = Right
Making Easy = Right 4
![Page 4: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/4.jpg)
“The Right Thing”
Making Easy = Right 5
Collaborate!
Automate!
![Page 5: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/5.jpg)
But… The Dev<->Ops Chasm
Making Easy = Right 6
![Page 6: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/6.jpg)
What Were we Doing Wrong?
Making Easy = Right 7
![Page 7: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/7.jpg)
Downward Spiral of Manual Activity
Making Easy = Right 8
![Page 8: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/8.jpg)
Entrenched Silos
Making Easy = Right 9
![Page 9: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/9.jpg)
High Release Overheads
Making Easy = Right 10
![Page 10: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/10.jpg)
Huge Batch Sizes
Making Easy = Right 11
![Page 11: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/11.jpg)
Poor Visibility
Making Easy = Right 12
![Page 12: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/12.jpg)
Calendar of Doom
Making Easy = Right 13
![Page 13: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/13.jpg)
Unpredictable Prioritisation
Making Easy = Right 14
![Page 14: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/14.jpg)
Superstitious Gatekeeping
Making Easy = Right 15
just in case
![Page 15: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/15.jpg)
Stagnating Applications
Making Easy = Right 16
![Page 16: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/16.jpg)
Manual Deployment Anti-Patterns
Making Easy = Right 17
Extensive, detailed release documentation
Reliance on manual testing to confirm app is correct
Explaining why deployment is going wrong on release day
Frequent corrections to release process during release
Environments that differ in their configuration
Releases that take more than a few minutes to perform
Releases that are unpredictable in their outcome
![Page 17: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/17.jpg)
Cycle Time measured in
weeks/months
Application Pipelines Ops
Days
Weeks!
Days
Staging Load Test
Production
Days
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA !?
Prio
ritis
ed R
elea
se Q
ueue
![Page 18: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/18.jpg)
Let’s Migrate to Microservices!
• Explosion of manual effort
Making Easy = Right 19
AB
C
D
E F
A
B1C1 GC2 H I
B2D1 J
D2 KL1 ML2 N
EF1 O P
F2 Q RST
![Page 19: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/19.jpg)
HeavyWeight to LightWeight
Glassfish• Feature overhead• Encouraged manual config• Complex/Slow deployment
DropWizard• Trimmed down to basics• Standard config files• Simple/Fast deployment
Making Easy = Right 20
![Page 20: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/20.jpg)
But… Combinatorial Explosion
Making Easy = Right 21
![Page 21: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/21.jpg)
Unpredictability Slows You Down
Making Easy = Right 22
![Page 22: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/22.jpg)
Moving from wrong to right
Making Easy = Right 23
Wrong
Right
![Page 23: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/23.jpg)
Reduce Cycle Time
Making Easy = Right
![Page 24: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/24.jpg)
Flexibility vs Predictability
• Freedom is Great for Innovation• Consistency is Essential For Speed• How do we promote a culture of freedom
and responsibility but still provide predictability?
Flexible Predictable
Making Easy = Right 25
![Page 25: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/25.jpg)
Standardisation
Making Easy = Right 26
Standardised
Freedom
• Standardise on how services are deployed and how they communicate.
• Be flexible about their contents.
![Page 26: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/26.jpg)
Defining Standards
• Discussions– One-on-one– IM– Emails– Meetings
• Incentivise input• Confluence– Application Deployment
Standards 1.0
Making Easy = Right 27
![Page 27: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/27.jpg)
Example Standards
Making Easy = Right 28
Logging• Log file locations,
filenames
Log rotate Directories/Files• Ownership,
permissions
Puppet
Supervisord config.yaml• Filename, location
Metrics JVM settings
Network• Ports• Firewall rules
Cron Endpoints• Healthchecks, status,
metrics
SSL• Certs, keystores
RPM• Versioning• Packaging
init.d scripts• Sub-commands
Versioning Migration Notes
![Page 28: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/28.jpg)
Example Standards
Making Easy = Right 29
![Page 29: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/29.jpg)
Future Considerations
Making Easy = Right 30
![Page 30: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/30.jpg)
Standards Lifecycle
Making Easy = Right 31
![Page 31: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/31.jpg)
Common Ground
Making Easy = Right 32
![Page 32: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/32.jpg)
How do we Make it Easy?DevOps @ Wotif: Making Easy = Right
Making Easy = Right 33
![Page 33: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/33.jpg)
Automated Verification
Making Easy = Right
Compliance Test SuiteBring Operations into DevelopmentFast Operational FeedbackTest Driven Operational CompatibilityBackwards Compatibility
![Page 34: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/34.jpg)
Ops Testing with Fabric
Making Easy = Right 36
•Does rpm have correct metadata linking to git repo? Check existence/absence of filesrpm•Is correct version actually installed?yum•Check existence/absence of files. Check file permissions/ownershipls• Is correct version actually running?lsof•Check correct ports exposednetstat • Is correct log format in use?tail•Check cron configgrep•Check contents of standalone jar file – e.g. metadata, library versionsunzip•Check standard endpoints for e.g. content, status code, response timecurl•Check JVM optionsjps•Check 1 and only 1 process runningps
![Page 35: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/35.jpg)
Compliance Test Example
Making Easy = Right 37
![Page 36: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/36.jpg)
Reference Implementation
• “helloworld-service”• Deployed to production
Making Easy = Right 38
![Page 37: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/37.jpg)
Rolling Upgrade
• Safely orchestrate cluster upgrades – automated deployment & testing
• Initially interactive
Making Easy = Right 39
• Check Load Balancer
Exit Pool
• Orchestrate Puppet
Upgrade
• Compliance Test• Smoke Test• Feature Test
Test
• Check Load Balancer
Enter Pool
• exitpool• stop• yum remove• yum install• Wait for manual
testing!• enterpool
Manual Release
![Page 38: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/38.jpg)
SLIPway
Making Easy = Right 40
• Big review of old release processes– Focus on reducing cycle
time• Simple Lightweight
Independent Path Way• Simple rules for release
process• Keep changes
independent
![Page 39: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/39.jpg)
Simple Rules for SLIPWay
• Independent• One application per release• Backwards-compatible • No DB/Network/OS changes
• No manual testing• Cannot book specific time
• Feature switch for time• Ops will service queue within 24 hours
• Compliance with latest standards
![Page 40: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/40.jpg)
Simple Rules for SLIPWay
42Making Easy = Right
![Page 41: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/41.jpg)
Simple Rules for SLIPWay
43Making Easy = Right
![Page 42: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/42.jpg)
Application Pipelines Ops
Days
Weeks!
Days
Staging Load Test
Production
Days
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA !?
Prio
ritis
ed R
elea
se Q
ueue
![Page 43: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/43.jpg)
45
Application Pipelines Ops
Prio
ritis
ed R
elea
se Q
ueue
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
HoursHours
Sim
ple
Rule
s
![Page 44: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/44.jpg)
Making Easy = Right 46
![Page 45: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/45.jpg)
SLIPway Chat Room
Making Easy = Right 48
![Page 46: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/46.jpg)
SLIPway Example Release
Making Easy = Right 49
![Page 47: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/47.jpg)
Freeing up 3 full time employees
0
5
10
15
20
25
Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov
Man
-hou
rs p
er re
leas
e*
man
-hou
rs t
o sh
ip a
rele
ase 95%
reduction in man-hours
GRIPway
SLIPway
SLIPway introduction
-18.5hrs
Hours Spent Shipping Release
![Page 48: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/48.jpg)
0
5
10
15
20
Cycl
e ti
me
in b
usin
ess d
ays
busi
ness
day
s-8
-2
86%cycle time reduction
SLIPway
GRIPwayaverage 2013
GRIPway today
Jan JanJulApr Oct Apr
20142013
Jul Oct
Time-to-market Reduction
Making Easy = Right
2014
![Page 49: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/49.jpg)
0
20
40
60
80
100
120
Num
ber o
f rel
ease
s pe
r m
onth
Thro
ughp
ut x
2.9190%
releases
GRIPway average 2013
SLIPway introduction
GRI
Pway
SLIP
way
SLIPway average
GRI
Pway
Jan JanJulApr Oct Apr
20142013
Jul
2.9x Releases Per Month
Making Easy = Right
![Page 50: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/50.jpg)
Release Cycle Time Comparison
Making Easy = Right
7x lower Avg. Cycle Time
0
1
2
3
4
5
6
7
8
9
10
GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days
![Page 51: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/51.jpg)
Commit stage
Acceptance stage
Capacity Testing
Production
Continuous Delivery
Continuous Deployment
Automatic Trigger Manual Trigger
Commit stage
Acceptance stage
Capacity Testing
Production
![Page 52: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/52.jpg)
Release Cycle Time Comparison
Making Easy = Right
7x lower Avg. Cycle Time 30x lower vs SLIPway200x lower vs GRIPway
21mins
0
1
2
3
4
5
6
7
8
9
10
GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days Continuous DEPLOYMENT
![Page 53: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/53.jpg)
Upward Spiral
Making Easy = Right 56
![Page 54: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/54.jpg)
Lessons LearnedDevOps @ Wotif: Making Easy = Right
57
![Page 55: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/55.jpg)
Lessons Learned
Making Easy = Right 58
Get the Balance RightChip Away At The IcebergMigration NotesThe Carrot, Not The StickStrongly Type Your DiscussionsUse Semantic VersioningBe Opinionated – Not ArrogantWarranty PeriodEmbrace Legacy
![Page 56: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/56.jpg)
Get the Balance Right- Optimise for Speed
No Rules Too Many Rules
Making Easy = Right 59
![Page 57: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/57.jpg)
Chip Away at the Iceberg
Making Easy = Right 60
![Page 58: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/58.jpg)
Migration Notes
• Annotate changes since previous standards – E.g. “(since v2.3)”
• Each standard version has migration notes from previous version
• Can upgrade between several versions by following migration notes nav bar
Making Easy = Right 61
1.0 2.0 2.1 2.2 2.3
![Page 59: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/59.jpg)
The Carrot, Not The Stick
• We tried the stick approach for years– You’re doing it wrong!– (But we won’t tell you how
to do it right)
• Blame games, anger• No business impetus to
improve things
• Then we tried the carrot approach– Offer a great new
alternative– It’s optional!– You know you want it…
• Business invested in the process
Making Easy = Right 62
![Page 60: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/60.jpg)
Strongly Type Your Discussions
• Choose catchy names– Helps make decisions– Helps climb out of muck
• Use State Transition tables– State -> Action -> New State– Visualise problems– Decide what to support
Making Easy = Right 63
![Page 61: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/61.jpg)
State Transitions – Current
Making Easy = Right 64
![Page 62: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/62.jpg)
State Transitions – Future
Making Easy = Right 65
![Page 63: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/63.jpg)
Use Semantic Versioning
• Especially for shared libraries
Making Easy = Right 66
1 . 2 . 3Major Minor Patch
semver.org
![Page 64: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/64.jpg)
Be Opinionated – Not Arrogant
• Strong decisions are important• But you’ll never get it 100% up front
Making Easy = Right 67
I’m kind of a big
deal
![Page 65: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/65.jpg)
Warranty Period
• Can’t get standards 100% right
• But need to lock them down
• Two-week warranty period
• Trial in range of different applications
Making Easy = Right 68
![Page 66: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/66.jpg)
Small Batch Size for Standards
• Minor updates every 1-2 months
• Fast feedback• Enable innovation• Easier to update
Making Easy = Right 69
![Page 67: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/67.jpg)
Embrace Legacy
• You’re creating legacy• Make migration easy• Incentivise catch-up
Making Easy = Right 70
![Page 68: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/68.jpg)
Building BlocksSLIPway• Simple Rules, Independence
Rolling Upgrade• Automated Deploy & Test
Compliance Tests• Test-driven Ops Compatibility
Standards• The Agreed “Right Thing”
Making Easy = Right 71
The Right Thing
Making It Easy
![Page 69: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/69.jpg)
Key Takeaways• Make the interface between teams (dev/ops)
consistent and predictable• Cultural choices (agreement) > tool choices• Prioritise operational feedback to developers• Faster releases leads to smaller batch size leads
to lower risk leads to happier customers• Start with interactive automation• Independent releases with no dependencies• Incentivise legacy catch-up through operational
compliance to latest agreed standards
Making Easy = Right 72
Consistent and predictable interface between teams (dev/ops)
Cultural choices (agreement) > tool choices
Prioritise fast operational feedback to developers
Faster releases à smaller batch size à lower risk à happier customersBuild trust with interactive automation
Decoupled releases – independent& dependency-‐freeà speed
Incentivise legacy catch-‐up
![Page 70: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/70.jpg)
Making Easy = Right 73
RightEasy
![Page 71: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/71.jpg)
Thanks for listening!Any questions?
Alexandra Spillane • @ajbwMatt Callanan • @mcallana
slides, etc • mattcallanan.net
Making Easy = Right
![Page 72: Right...Ops Testing with Fabric Making Easy = Right 36 rpm •Does rpm have correct metadata linking to git repo? Check existence/absence of files yum •Is correct version actually](https://reader034.vdocument.in/reader034/viewer/2022043017/5f39cd66daaba2036b65b5ff/html5/thumbnails/72.jpg)
Image AttributionImage
“Downward Spiral” (http://flic.kr/p/67Giiz) by Chad K is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)
“Angels Landing from the Deertrap Mountain Trail” (Chasm -‐ https://flic.kr/p/abuWHt) by Zion National Park is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)
“WRONG WAY” (https://flic.kr/p/cRjc6q) by David Goehring is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)
“Old Gravel Silos, Buffalo Bayou, East of Jensen, Houston, Texas 0906091555BW” (https://flic.kr/p/6WueVm) by Patrick Feller is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)
“Fisher men lifting a boat in Bangladesh. Photo by Finn Thilsted” (https://flic.kr/p/ddUf4S) by WorldFish is licensed under CC BY NC ND 2.0 (https://creativecommons.org/licenses/by-‐nc-‐nd/2.0)
“Poor visibility from Kelud eruption 14 February 2014, Yogyakarta” (http://bit.ly/1N3Lf7A) by Aldnonymous is licensed under BY SA 3.0 (https://creativecommons.org/licenses/by-‐sa/3.0/deed.en)
“Changed priorities ahead” (https://flic.kr/p/debvm) by Peter Reed is licensed under CC BY NC 2.0 (https://creativecommons.org/licenses/by-‐nc/2.0)
“Stop Sign” (https://flic.kr/p/aUEW1D) by DonkeyHotey is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)
“green scum” (https://flic.kr/p/3MvvsH) by M&R Glasgow is licensed under CC BY ND 2.0 (https://creativecommons.org/licenses/by-‐nd/2.0)
“Thinking… please wait “ (https://flic.kr/p/6wdLat) by Karola Riegler is licensed under CC BY ND 2.0 (https://creativecommons.org/licenses/by-‐nd/2.0)
“Elastic“ (https://flic.kr/p/a2XZDB) by Chris Stevenson is licensed under CC BY NC 2.0 (https://creativecommons.org/licenses/by-‐nc/2.0)
“A Garden Of Climbs” (https://flic.kr/p/4U2BB1) by Jasen Miller is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)
“Alexander Vinokourov competing in the London 2012 Men's Olympic Time Trial” (http://bit.ly/1QndiBd) by Diliff is licensed under BY SA 3.0 (https://creativecommons.org/licenses/by-‐sa/3.0/)
“Upward Spiral” (https://flic.kr/p/noFia) by Clint Vigil by is licensed under CC BY NC 2.0 (https://creativecommons.org/licenses/by-‐nc/2.0)
“Rivers of Humanity” (Indian Traffic -‐ https://flic.kr/p/3yMSMW) by pangalactic gargleblaster and the heart of gold is licensed under CC BY NC ND 2.0 (https://creativecommons.org/licenses/by-‐nc-‐nd/2.0)
“Red Tape“ (https://flic.kr/p/b7T3JD) by Free Press/ Free Press Action Fund is licensed under BY NC SA 2.0 (https://creativecommons.org/licenses/by-‐nc-‐sa/2.0)
“To the Right A Bit” (Slipway -‐ https://flic.kr/p/9oQVh8) by tiffany terry is licensed under CC BY NC ND 2.0 (https://creativecommons.org/licenses/by-‐nc-‐nd/2.0)
“Iconic iceberg sail-‐by” (https://flic.kr/p/hV1vwH) by Visit Greenland is licensed under CC BY NC ND 2.0 (https://creativecommons.org/licenses/by-‐nc-‐nd/2.0)
“Luckiamute Falls” (https://flic.kr/p/7EMs4T) by Ian Sane is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)
“Three Keys” (https://openclipart.org/detail/219938/water) by [email protected] is licensed under unlimited-‐commercial-‐use (https://openclipart.org/unlimited-‐commercial-‐use-‐clipart)
Presentation includes stock images used under license from Shutterstock.com and iStockPhoto.com. Authors include: Stepanek Photography/Shutterstock.com, skvoor/Shutterstock.com, Michael D Brown/Shutterstock.com, argus/Shutterstock.com, Sergey Nivens/Shutterstock.com, Anze Mulec/Shutterstock.com, Jaz_czc/Shutterstock.com, DenisNata/Shutterstock.com, Arnaud Weisser/Shutterstock.com, iStock.com/gehringj
Standardisation Theory Diagram inspired by Sam Newman: http://www.slideshare.net/spnewman/practical-‐microservices-‐yow-‐2013/56