velocity conference nyc 2014 - real world devops
DESCRIPTION
In a world where agility has become a requirement, business and engineering demands have decreed the death of the “Department of No”. This talk will cover the journey of an IT Operations department from a single DevOps team to a business-wide cultural shift that has affected the way people interact and work with each other. In order to make sure that our DevOps initiative would be successful, we needed to make changes to the corporate organization, rearrange teams and roles in several areas, and make sure that everyone fully understand where we were being headed to. All these steps will be covered in this talk that will demonstrate some common pitfalls and misconceptions that jeopardize the DevOps adoption, particularly in large enterprises with several compliancy requirements and some outdated bureaucracy.TRANSCRIPT
![Page 1: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/1.jpg)
Real World DevOpsRodrigo Campos (@xinu)
Velocity NY 2014
![Page 2: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/2.jpg)
Hi I’m Rodrigo!
• Proudly crimping cables since 1992
• System Administrator
• Operations Director at walmart.com in Latin America
![Page 3: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/3.jpg)
I’m not going to talk about
• Silver Bullets
• Unicorns
• Tools and Utilities
![Page 4: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/4.jpg)
Instead I’ll focus on
• The challenges we faced
• How DevOps has helped us
• Bumps and bruises along the way
![Page 5: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/5.jpg)
Our challenges
![Page 6: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/6.jpg)
Our challenges
• Deploy a whole new platform before Black Friday
![Page 7: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/7.jpg)
Our challenges
• Deploy a whole new platform before Black Friday
• Expand our private infrastructure to withstand web traffic spikes
![Page 8: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/8.jpg)
Our challenges
• Deploy a whole new platform before Black Friday
• Expand our private infrastructure to withstand web traffic spikes
• Develop a company-wide agile mindset
![Page 9: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/9.jpg)
Our challenges
• Deploy a whole new platform before Black Friday
• Expand our private infrastructure to withstand web traffic spikes
• Develop a company-wide agile mindset
• Rebuild trust between tech and business
![Page 10: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/10.jpg)
Different companies, same problems
![Page 11: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/11.jpg)
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
![Page 12: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/12.jpg)
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
• BOFH attitude started to kick-in
![Page 13: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/13.jpg)
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
• BOFH attitude started to kick-in
• Over the wall production deployments
![Page 14: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/14.jpg)
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
• BOFH attitude started to kick-in
• Over the wall production deployments
• Tug of war between Dev and Ops
![Page 15: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/15.jpg)
Different companies, same problems
• Silos not only between Developers and Operations, but within departments as well
• BOFH attitude started to kick-in
• Over the wall production deployments
• Tug of war between Dev and Ops
• Lots of bureaucracy and red-tape
![Page 16: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/16.jpg)
Some phrases (roughly translated)
![Page 17: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/17.jpg)
Some phrases (roughly translated)
• “I'll go to the 9th floor to help you kick everyone’s a**”
![Page 18: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/18.jpg)
Some phrases (roughly translated)
• “I'll go to the 9th floor to help you kick everyone’s a**”
• “I don’t care if it’s stable as long as it passed QA testing”
![Page 19: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/19.jpg)
Some phrases (roughly translated)
• “I'll go to the 9th floor to help you kick everyone’s a**”
• “I don’t care if it’s stable as long as it passed QA testing”
• “This is operations’ job”
![Page 20: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/20.jpg)
Some phrases (roughly translated)
• “I'll go to the 9th floor to help you kick everyone’s a**”
• “I don’t care if it’s stable as long as it passed QA testing”
• “This is operations’ job”
• “Consider this a favor”
![Page 21: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/21.jpg)
How DevOps was seen
![Page 22: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/22.jpg)
So what is DevOps anyway?
![Page 23: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/23.jpg)
So what is DevOps anyway?
• Being loosely defined can actually be a good thing!
![Page 24: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/24.jpg)
So what is DevOps anyway?
• Being loosely defined can actually be a good thing!
• For us: “DevOps is a culture that fosters communication, transparency and collaboration"
![Page 25: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/25.jpg)
So what is DevOps anyway?
![Page 26: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/26.jpg)
So what is DevOps anyway?
• Communication: I’ll stand up and go to your desk if I need to talk to you, you’re more than welcome to do the same!
![Page 27: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/27.jpg)
So what is DevOps anyway?
• Communication: I’ll stand up and go to your desk if I need to talk to you, you’re more than welcome to do the same!
• Transparency: I want to know your problems, I’ll make sure you know mine!
![Page 28: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/28.jpg)
So what is DevOps anyway?
• Communication: I’ll stand up and go to your desk if I need to talk to you, you’re more than welcome to do the same!
• Transparency: I want to know your problems, I’ll make sure you know mine!
• Collaboration: We’re in the same boat and I really mean it!
![Page 29: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/29.jpg)
Operations Developers
UsThem InfoSec
Works on my Machine
We needed to change people's mindset
![Page 30: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/30.jpg)
Source: http://youtu.be/Fx8OBeNmaWw
We needed to change people's mindset
![Page 31: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/31.jpg)
We needed to change people's mindset
![Page 32: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/32.jpg)
• Presentations to technology, business and human capital teams
We needed to change people's mindset
![Page 33: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/33.jpg)
• Presentations to technology, business and human capital teams
• Reinforcing an inclusive and welcoming culture was essential
We needed to change people's mindset
![Page 34: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/34.jpg)
• Presentations to technology, business and human capital teams
• Reinforcing an inclusive and welcoming culture was essential
• Everyone felt prepared and willing to change
We needed to change people's mindset
![Page 35: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/35.jpg)
Typical Scenario
Infrastructure
Operations
Developers
Business
{ }
• Things go over the wall
• No alignment between tech and business
• One-way communications
• IT as a necessary evil (aka cost center)
![Page 36: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/36.jpg)
We were already a little bit different…
Infrastructure
Operations
Developers
Business
{ }
DevOps Team
![Page 37: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/37.jpg)
1st step
Infrastructure
Operations
Developers
Business
{ }Production Engineering
DevOps
![Page 38: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/38.jpg)
2nd step
Infrastructure
Production ManagementDevelopers
Business
{ }
Production Engineering + { }
DevOps
![Page 39: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/39.jpg)
InfoSec
![Page 41: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/41.jpg)
• First and foremost: end the dossier culture
• Communication + collaboration + transparency = safer systems
InfoSec
![Page 42: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/42.jpg)
• First and foremost: end the dossier culture
• Communication + collaboration + transparency = safer systems
• Integrate security tests into continuous integration
InfoSec
![Page 43: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/43.jpg)
• First and foremost: end the dossier culture
• Communication + collaboration + transparency = safer systems
• Integrate security tests into continuous integration
• Rugged DevOps: http://slidesha.re/1r87Vs7
InfoSec
![Page 44: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/44.jpg)
• No more FUD
• What is riskier to the business?
• Lack of speed is a risk too!
InfoSec
![Page 45: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/45.jpg)
• ITIL: “You keep using that word, I don’t think it means what you think it means”
• People often blame ITIL as an excuse for badly designed processes
• Surprisingly ITIL compliance was not our biggest problem
Compliance
![Page 46: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/46.jpg)
• Audits are still our biggest challenge
• In our case, we have a sponsor (the CTO)
• Someone (who happens to be me) will be held accountable if something goes wrong
Compliance
![Page 47: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/47.jpg)
The bad apples
• We’ve adopted the “No asshole rule"
• There will be obstructive people that won’t embrace change
• If you can't get rid of them, simply let them fade into oblivion
![Page 48: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/48.jpg)
KPIs
![Page 50: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/50.jpg)
KPIs
• Deployment Frequency
• Speed of Deployment
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)
![Page 51: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/51.jpg)
KPIs
• Deployment Frequency
• Speed of Deployment
• Deployment Success Rate
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)
![Page 52: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/52.jpg)
KPIs
• Deployment Frequency
• Speed of Deployment
• Deployment Success Rate
• Service Recovery
Source: Puppet Labs Blog (http://bit.ly/Z3gRpu)
![Page 53: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/53.jpg)
KPIs
• Deployment Frequency
• Speed of Deployment
• Deployment Success Rate
• Service Recovery
• CultureSource: Puppet Labs Blog (http://bit.ly/Z3gRpu)
![Page 54: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/54.jpg)
Bumps & Bruises
![Page 55: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/55.jpg)
Bumps & Bruises• DO:
![Page 56: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/56.jpg)
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
![Page 57: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/57.jpg)
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
• Use measurable KPIs
![Page 58: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/58.jpg)
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
• Use measurable KPIs
• Explain to business areas what you’re doing
![Page 59: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/59.jpg)
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
• Use measurable KPIs
• Explain to business areas what you’re doing
• Align with InfoSec and Compliance ASAP
![Page 60: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/60.jpg)
Bumps & Bruises• DO:
• Understand beforehand what you want to achieve
• Use measurable KPIs
• Explain to business areas what you’re doing
• Align with InfoSec and Compliance ASAP
• Work hard on a sponsor
![Page 61: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/61.jpg)
Bumps & Bruises
![Page 62: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/62.jpg)
Bumps & Bruises• DON'T:
![Page 63: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/63.jpg)
Bumps & Bruises• DON'T:
• Create a “Ninja Superstar team”
![Page 64: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/64.jpg)
Bumps & Bruises• DON'T:
• Create a “Ninja Superstar team”
• Postpone communication
![Page 65: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/65.jpg)
Bumps & Bruises• DON'T:
• Create a “Ninja Superstar team”
• Postpone communication
• Delay role changes
![Page 66: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/66.jpg)
Bumps & Bruises• DON'T:
• Create a “Ninja Superstar team”
• Postpone communication
• Delay role changes
• Underestimate the managers' ability to influence behavior
![Page 67: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/67.jpg)
• In the beginning it will be a lonely fight
• Some people will label you as the “hippie-anarchist”
Keep in mind
![Page 68: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/68.jpg)
Results
![Page 69: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/69.jpg)
Results• New eCommerce platform successfully deployed
![Page 70: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/70.jpg)
Results• New eCommerce platform successfully deployed
• Only major eCommerce website in Brazil to have 100% uptime during Black Friday
![Page 71: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/71.jpg)
Results• New eCommerce platform successfully deployed
• Only major eCommerce website in Brazil to have 100% uptime during Black Friday
• Increase in deployment frequency and success rate
![Page 72: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/72.jpg)
Results• New eCommerce platform successfully deployed
• Only major eCommerce website in Brazil to have 100% uptime during Black Friday
• Increase in deployment frequency and success rate
• Business is incorporating agile methodologies
![Page 73: Velocity Conference NYC 2014 - Real World DevOps](https://reader033.vdocument.in/reader033/viewer/2022052821/553a327c550346bf308b4aaf/html5/thumbnails/73.jpg)
Any questions?
• Special thanks to: https://www.flickr.com/photos/51321237@N07/ https://www.flickr.com/photos/kh-67/ https://www.flickr.com/photos/jrparis/ https://www.flickr.com/photos/andrewdubock/ https://www.flickr.com/photos/rachelyoung/ !https://bitly.com/bundles/o_4p62vc3lid/5