measuring dev ops vital signs

Nokia Bristol

John Clapham & Paul Swartout Nokia Entertainment

Checking DevOps’ vital signs - how healthy is your culture?

© 2013 Nokia

• This is the story of our foray into [devops] culture monitoring…

© 2013 Nokia

• Disclaimer:

© 2013 Nokia

We don’t know what we’re talking about…

© 2013 Nokia

…or at least we’re not experts

We don’t know what we’re talking about…

© 2013 Nokia

The order of things…

• Reasonings – why we started• Identifyings – Choosing what to measure• Data things – What we actually measured• Findings – What we found from data• Learnings – What we learnt from data and doing.

© 2013 Nokia

Reasonings…

© 2013 Nokia

Has including everyone worked?

6 releases / 140 components

January to December 2010

246 releases / 246 components

January to October 2011

Voices in our heads whispered…

© 2013 Nokia

Are we getting complacent?

….and would we actually notice?

© 2013 Nokia

•We knew the answers for our systems, but not our culture

• Could we monitor culture?

• If we monitored culture we could learn about health

•We could keep fit…

© 2013 Nokia

© 2013 Nokia

• But monitoring culture is hard.

© 2013 Nokia

There are some measures already:

• Organisational Health Index• Agile & Scrum Checklists• DevOpsDays Boston Open space

© 2013 Nokia

What is culture anyway?

© 2013 Nokia

What does a DevOps culture look like anyway?

Dev Land Ops World

© 2013 Nokia

Indentifyings…

© 2013 Nokia

Why do people get into Dev Ops?

© 2013 Nokia

Its all about improving…

EfficiencyEnjoymentQualitySecurityUp time, response timeDelivery & Deployment time

…some of these will be our vital signs

© 2013 Nokia

Turning those motivations into Vital Signs could be challenging…

© 2013 Nokia

What makes a good vital sign?

• Applicable to different organisation sizes• Applicable to varying levels of maturity• Not prescriptive • Not judgemental• Low to no cost

© 2013 Nokia

Vital Sign 1 – (Dev) Cycle Time

• Average time between start of development and delivery to production.

© 2013 Nokia

Vital Sign 2 – Shared Purpose

• Are all of engineering working towards the same goals?

© 2013 Nokia

Vital Sign 3 – Motivation

• Is everyone motivated to learn and improve ways of working?

© 2013 Nokia

Vital Sign 4 – Collaboration.

• Do people work together willingly?

© 2013 Nokia

Vital Sign 5 – Effectiveness

• How much time spent actually adds value?

© 2013 Nokia

Data-things…

© 2013 Nokia

How do you get data?

• Conversation & Interview • Questionnaires • Raw Data • Observations & spying

© 2013 Nokia

The art of the question

• Don’t lead the witness• Leave yourself room to be surprised• Be exacting

© 2013 Nokia

Where knowledge of production systems and applications is required to solve a live problem, would you say there is effective collaboration between engineers?

© 2013 Nokia

Findings

© 2013 Nokia

Vital Sign 3 – Motivation

Dev Ops Other0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Do you believe there are opportunities to improve ways of working?

Yes No Maybe

© 2013 Nokia

Vital Sign 5 – Effectiveness

Dev Ops Other0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

How is your working time spent?

Improvements Stability Waste

We also asked a referential question…

What is in your way?

© 2013 Nokia

What’s in your way?

Common themes:• No Time • Pressure to focus on features• Nothing

© 2013 Nokia

What’s in your way?

• “Convincing others to agree to any proposed changes!”

• “Engineers who are busy or less cooperative”

• “Technology, tools, understanding the technology from the ops perspective”

© 2013 Nokia

Learnings…

© 2013 Nokia

• People were happy to participate!

© 2013 Nokia

• Our DevOps culture is healthy…

© 2013 Nokia

•We’ve established a baseline

• Talking about, and gathering data has been as valuable as the data itself

• Open (but not measurable questions) were insightful.

© 2013 Nokia

•We asked one more question

© 2013 Nokia

• Do you have any comments, or suggestions, about this questionnaire?

© 2013 Nokia

© 2013 Nokia

“No.”

© 2013 Nokia

“Aim is unclear”

© 2013 Nokia

“Some questions are too similarly worded, turns into a game of spot the difference, increasing amount of

work to understand the question rather than answering it.”

© 2013 Nokia

“It might be interesting to do something similar for the relationship between engineers and other parts of the

business.”

© 2013 Nokia

Concludings…

© 2013 Nokia

•Measuring culture is hard

…but rewarding!

© 2013 Nokia

•Would you recommend it to a friend?

© 2013 Nokia

•We learnt a lot about [DevOps] culture by creating our vital signs.

•We gained insight and workable ideas from data and open questions.

•More people are thinking about DevOps.

© 2013 Nokia

Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4

Health 2013-2014

Collaboration Motivation Shared purpose Engagement Effectiveness

© 2013 Nokia

•What vital signs would you look for?

Thanks!

john.clapham@nokia.com

@JohnC_Bristol

paul.swartout@nokia.com

@pswartout