measuring dev ops vital signs
DESCRIPTION
DevOpsDays London 2013 presentation.TRANSCRIPT
Nokia Bristol
John Clapham & Paul Swartout Nokia Entertainment
Checking DevOps’ vital signs - how healthy is your culture?
© 2013 Nokia
• This is the story of our foray into [devops] culture monitoring…
© 2013 Nokia
• Disclaimer:
© 2013 Nokia
We don’t know what we’re talking about…
© 2013 Nokia
…or at least we’re not experts
We don’t know what we’re talking about…
© 2013 Nokia
The order of things…
• Reasonings – why we started• Identifyings – Choosing what to measure• Data things – What we actually measured• Findings – What we found from data• Learnings – What we learnt from data and doing.
© 2013 Nokia
Reasonings…
© 2013 Nokia
Has including everyone worked?
6 releases / 140 components
January to December 2010
246 releases / 246 components
January to October 2011
Voices in our heads whispered…
© 2013 Nokia
Are we getting complacent?
….and would we actually notice?
© 2013 Nokia
•We knew the answers for our systems, but not our culture
• Could we monitor culture?
• If we monitored culture we could learn about health
•We could keep fit…
© 2013 Nokia
© 2013 Nokia
• But monitoring culture is hard.
© 2013 Nokia
There are some measures already:
• Organisational Health Index• Agile & Scrum Checklists• DevOpsDays Boston Open space
© 2013 Nokia
What is culture anyway?
© 2013 Nokia
What does a DevOps culture look like anyway?
Dev Land Ops World
© 2013 Nokia
Indentifyings…
© 2013 Nokia
Why do people get into Dev Ops?
© 2013 Nokia
Its all about improving…
EfficiencyEnjoymentQualitySecurityUp time, response timeDelivery & Deployment time
…some of these will be our vital signs
© 2013 Nokia
Turning those motivations into Vital Signs could be challenging…
© 2013 Nokia
What makes a good vital sign?
• Applicable to different organisation sizes• Applicable to varying levels of maturity• Not prescriptive • Not judgemental• Low to no cost
© 2013 Nokia
Vital Sign 1 – (Dev) Cycle Time
• Average time between start of development and delivery to production.
© 2013 Nokia
Vital Sign 2 – Shared Purpose
• Are all of engineering working towards the same goals?
© 2013 Nokia
Vital Sign 3 – Motivation
• Is everyone motivated to learn and improve ways of working?
© 2013 Nokia
Vital Sign 4 – Collaboration.
• Do people work together willingly?
© 2013 Nokia
Vital Sign 5 – Effectiveness
• How much time spent actually adds value?
© 2013 Nokia
Data-things…
© 2013 Nokia
How do you get data?
• Conversation & Interview • Questionnaires • Raw Data • Observations & spying
© 2013 Nokia
The art of the question
• Don’t lead the witness• Leave yourself room to be surprised• Be exacting
© 2013 Nokia
Where knowledge of production systems and applications is required to solve a live problem, would you say there is effective collaboration between engineers?
© 2013 Nokia
Findings
© 2013 Nokia
Vital Sign 3 – Motivation
Dev Ops Other0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Do you believe there are opportunities to improve ways of working?
Yes No Maybe
© 2013 Nokia
Vital Sign 5 – Effectiveness
Dev Ops Other0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
How is your working time spent?
Improvements Stability Waste
We also asked a referential question…
What is in your way?
© 2013 Nokia
What’s in your way?
Common themes:• No Time • Pressure to focus on features• Nothing
© 2013 Nokia
What’s in your way?
• “Convincing others to agree to any proposed changes!”
• “Engineers who are busy or less cooperative”
• “Technology, tools, understanding the technology from the ops perspective”
© 2013 Nokia
Learnings…
© 2013 Nokia
• People were happy to participate!
© 2013 Nokia
• Our DevOps culture is healthy…
© 2013 Nokia
•We’ve established a baseline
• Talking about, and gathering data has been as valuable as the data itself
• Open (but not measurable questions) were insightful.
© 2013 Nokia
•We asked one more question
© 2013 Nokia
• Do you have any comments, or suggestions, about this questionnaire?
© 2013 Nokia
© 2013 Nokia
“No.”
© 2013 Nokia
“Aim is unclear”
© 2013 Nokia
“Some questions are too similarly worded, turns into a game of spot the difference, increasing amount of
work to understand the question rather than answering it.”
© 2013 Nokia
“It might be interesting to do something similar for the relationship between engineers and other parts of the
business.”
© 2013 Nokia
Concludings…
© 2013 Nokia
•Measuring culture is hard
…but rewarding!
© 2013 Nokia
•Would you recommend it to a friend?
© 2013 Nokia
•We learnt a lot about [DevOps] culture by creating our vital signs.
•We gained insight and workable ideas from data and open questions.
•More people are thinking about DevOps.
© 2013 Nokia
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
Health 2013-2014
Collaboration Motivation Shared purpose Engagement Effectiveness
© 2013 Nokia
•What vital signs would you look for?
