DevOps and the Business

Keeping Your Pro{du,je}ct {Manag,Own}er Productive

@clintoncwolfe

Who am I?

•Dev => DevOps

•DevOps Practice Lead at OmniTI

•founded devopsdictionary.com

•We’re contractors/consultants

•We’re hiring remote!Personal photo

I’m Clinton Wolfe. This is a picture of me and my wife, which I made out of Lego for a wedding gift. I was a web developer for many years, in the peak mod_perl days, then moved into doing config management with Chef. Today, I’m the DevOps Practice Lead at OmniTI, a small consulting company based out of the Baltimore suburbs.

And we are definitely hiring, remote workers welcome! Come talk to me :-0

Wait, “DevOps Consulting”?

•Big clients, small clients

•Look for friction among

Dev, Ops, and Business

•Suggest organizational &

practice changes

• Implement if desired

Some of you may think that the true DevOps can only come from a grassroots movement within the organization… and some of you may be right!

As I worked with teams, often coaching them about the kinds of work, and the importance of flow, I noticed more and more that the Dev and Ops sides were easy to convince - but the business side could not connect. This won’t be a presentation full of charts and hard numbers, but rather my own personal experiences and advice about reaching out to people in business.

Why talk about “Product”?

❖ Dev + Ops = DevOps❖ Business + DevOps =

PROFIT

gotcredit via flickr / CC-BY

Why do we care about the business side at all? We’re engineers, we do things - leave the marketing and grand initiatives to the leadership, right?

But what is our highest-level goal? To make money. Business leaders are increasingly onboard with DevOps because it allows the business to innovate more rapidly, get to market faster with compelling features, and provide seamless uptime to huge numbers of users.

So, DevOps and the Product side of the house are going to need to get along!

Product, PO, PM…

❖ Product Owner = PO

❖ Represents customer needs

❖ defines releases

❖ Project Manager = PM

❖ schedules resources

❖ worries about a plan

There are different roles here - a PO represents the customer’s needs, while the PM allocates resources. In some organizations I have worked with, the lines become very blurry, and we end up seeing one person handling both aspects - generally representing the interests of the business.

Some Assumptions❖ Your organization is agile, Agile, or “agile”

❖ There is a history of command-and-control

❖ Friction between dev and ops is getting better

❖ Friction between business and engineering is getting worse

tambako via flickr / CC-BY

Nearly all organizations have adopted Agile to a greater or lesser degree, or have attempted to in the past. You may know it as Scrum, or as a Kanban board; Agile gurus will tell you that those are both approaches to Agile, and then argue at length about the purity of them.

Why talk about Product, Really?

❖ PO’s set priority

❖ PM’s have allocate resources, work with estimates

❖ Both may lack understanding of the work at hand

No one can make you as miserable as a bad PO or PM.

jasohill via flickr / CC-BY

When I work as an individual contributor on a team with a bad PO or PM, it is a nightmare. You end up looking forward to days when they are on PTO.

That bad relationship shuts down communication. Is less communication ever a good thing?

Are PO’s/PM’s Terrible Ogres Sent to Spread Hate and Misery?

I know I have felt this way. Who here has felt this way?

Are PO’s/PM’s Terrible Ogres Sent to Spread Hate and Misery?

NO!

No one thinks they are the bad guy. Let’s try to look at things differently.

Changing the Dynamic

❖ understand a PO

❖ understand a PM

❖ understand the conflicts

❖ find ways to bridge the gapandy via flickr / CC-BY

DevOps is all about reaching out across the aisle and understanding problems from the other side’s perspective. That’s true for development and operations, but even more so between engineering and the business itself.

Meet Your PO

❖ Typically from a “non-technical” background

❖ Often defend the team to the customer

❖ Judged on outcomes that they cannot control

❖ Often a transitional role

A PO is truly between a rock and a hard place.

Being a good PO is a rare thing - if you’re good at it, or lucky, you stick around; if not, you are held more accountable than anyone else. That means there are a lot of inexperienced POs out there.

Meet Your PM

❖ Likely a quantitative / financial background

❖ “Plan the work, work the plan”

❖ PMP certification typically trains people in Waterfall

❖ Many PMs are on their first software project

Project managers are trained in the art of managing scope, cost, and schedule. Often they have some formal training or certification - PMP is most common. But, there are many PMs who come at it with a generic project management background - construction, business initiatives, etc. Moving into the software development and operations domain can be a difficult transition.

Life as a PM/PO

❖ Lonely

❖ Afraid

❖ Stressed

bernard goldbach via flickr / CC-BY

So, it kind of sounds like these jobs might be difficult and unpleasant. Are miserable people fun to work with? Do they make good decisions?

When Ops was mad at Dev, what did we do? We developed empathy. And we learned more about each others’ job. We found conflict points, and worked to resolve them to improve flow. And it’s working. So, let’s do that again.

Conflict Point: Background

The PO/PM doesn’t understand any technical

details!

The engineers speak in technobabble!

“I feel misunderstood”

Dev / Ops: PO / PM:

This is probably the number one source of friction and alienation.

“You are not from my tribe”

Being “Non-Technical”❖ Engineers need an outside perspective

❖ Can you speak in customer language?

❖ Personal investment, workmanship bias

❖ The skills you would need to be a PO:

❖ People skills, patience, listening

❖ Negotiation, compromise

❖ Business expertise

When I refer to “workmanship bias,” I mean that we are proud of our work, and the hours we put into it. When a PO comes along with an arbitrary change, it frustrates and angers us to have to discard or mutilate the prior work. But the customer - via the PO - have a better idea of what they need now, and that is a good thing - to continue building the old version would be wrong.

We need to stop saying non-technical in a sneering voice. We all have skillsets that can come into play in the innovation process. Take a big step back - our real goal is to make something that people want to give money to experience. NO ONE knows the details of that need better than the PO - and a good one know hows to express it.

Here’s one way I’ve learned to help a PO or PM ramp up, and build trust.

Vocabulary Corner

Meet privately with the PO/PM

❖ Opens a back-channel❖ Low expectations about

knowledge transfer❖ No feigned surprise at

ignorance❖ No gossiping afterwards

krista kennedy via flickr / CC-BY

Setup a weekly meeting with the PO or PM, one-on-one. Say it’s because you want to have a chance to reach out and understand the other’s terminology better. It’s important for this one-on-one time to be judgment-free. “During scrum today, you used the word ‘refactoring’ in a way that isn’t typical. Usually, it means ‘modifications that do not result in a change in functionality’” Did I misuse anything, like “velocity”?Away from the team, it’s easier for both of you to forgo appearances and speak from a place of vulnerability, which builds trust.

Conflict Point:Estimation

PMs expect software development to be predictable,

and it simply never will be.

The engineers can’t be trusted to give good estimates, and my schedules keep getting ruined!

“I cannot control the outcomes for which I am accountable.”

Dev / Ops: PO / PM:

Every engineer has been burned by this before. We have pressure to underestimate (often unconscious), while others have made it a habit to always overestimate for safety - either of which makes the estimation worth less.

Estimating Work

❖ Modeling a phenomenon means discarding detail

❖ One estimate input per task (not range)

❖ One estimate output per task

❖ Discards a lot of context and risk data

choices of what to discard has tremendous impactThese decisions are often made for you in the “work modeling software” - Jira, Trac, Rally, etc.The details modeled can vary widely from team to team within the same organization … and practices are often informal, with tribal customs (“we use the hours worked field for tracking defect count”)

Three EstimatesBestCase Typical Worst

Case Expected Risk Index

A 3 4 20 6.5 0.8

B 1 4 20 6.2 0.8

C 3 4 5 4 0.2

Inputs:• Best-Case• Typical• Worst-Case

Outputs:• Expected Effort

• (1xBest + 4xTypical + 1xWorst) / 6• Risk

• 1 - (Typical / Worst)See Software Estimation

by Steve McConnell - p120

There are many ways of estimating software, but one of my favorites is the three-point system.Instead of creating one estimate, put in three as inputs. For the estimator, this absorbs the instinct to over or underestimate - they have they chance to distinguish their realistic est from their initial response.For the PO, you can still have a single number to summarize; you can choose different weights.An additional value is now available - risk. You can use this to flag tasks as being high-risk in the work modeling software.

The ideal time to show this off, of course, is in your 1:1 time with the PM. Keep those conversations up, and let them drift.

Conflict Point:Unexpected Work

It doesn’t matter how much we plan, there will always be

unexpected work.

When we commit to a sprint, that’s exactly the work I as a PO expect to happen - anything else is unauthorized.

“No one understands where work comes from or who commits to it.”

Dev / Ops: PO / PM:

Depending on how authoritarian the organization is, this can result in either newly discovered work added to the current sprint (a PM/PO no-no) or ignoring newly discovered work, which may cause tasks in progress to fail.

If the team is too dictatorial, engineers may go rogue, and implement “dark tasks” if they can’t get them on the official board.

Types of Work❖ Planned Work

❖ features

❖ process improvement

❖ Unplanned Work

❖ incidents

❖ emergent tasks

❖ reworkSee The Goal and The Phoenix Project

Often, an inexperienced PO thinks that the only work to be done is the work that they are requesting. But there is often a large gap between what is planned and what actually happens.

Incidents are emergencies - firefighting, outages, etc. You must drop everything and fix it, switching contexts immediately.Emergent tasks are those that are discovered in the course of doing other work often unmet dependencies.Rework is time spent dealing with defects that were passed down to you – work that was thought to be complete but was in fact incomplete or defective.

Reducing Unplanned Work - Emergent

To reduce emergent work:• Explicitly track dependencies• Spend more engineering time

validating tasks• cross-check with a second

engineer• cost: increased planning overhead• risk: will not protect you from

unfamiliar work (spike)kevin o’mara via flickr / CC-BY

All efforts to reduce unplanned work are simply efforts to convert unplanned work into planned work. That conversion will not eliminate work and may in fact increase the amount of overhead involved. Like all process changes there is a point of diminishing returns. But by showing your PM these conversion techniques, they can start to capture and measure the unplanned work - and then start planning for it more. You can place the PO back in control of what work gets added, and will help them have a better understanding of why the remaining unplanned work happens.

Reducing Unplanned Work - Rework

To reduce rework:• automate to reduce human error• add testing upstream• reach out, build trust• cost: CI upkeep• risk: upstream may not

agree to testing katharine shilcutt via flickr / CC-BY

This type of unplanned work responds well to efforts to build out a build/deploy pipeline. Think of a factory being careful to only pass correct work downstream. There should be a lot of other sessions at this conference covering those ideas - but here, we have a lever to try to get buy-in from you PM and PO. By implementing improved build/test/deploy pipelining, you can help them make their work more predictable.

Reducing Unplanned Work - Incidents

To reduce incidents:• “devop more”

US Navy via flickr / CC-BY

I hope you’re picking up some ideas about this one at this conference :)

Conflict Point:Variation in Skills

We all have different areas of expertise, but those are ignored

when assigning tasks.

I have no idea who can do what, and to me, it’s all details anyway.

“We can’t make the best use of our people.”

Dev / Ops: PO / PM:

This one is especially a problem on new teams, in which the PM is unfamiliar with the work. A lot of breathe in sprint planning meetings is wasted on “who can do what”

T-Shaped People

❖ Broad, shallow skills, e.g.

❖ conversational python, ruby

❖ a CM tool

❖ One area of deep expertise, e.g.

❖ monitoring at scale leo reynolds via flickr / CC-BY

As someone described as a “devops engineer,” I can attest that you need a wide variety of skills, and the ability to reach out for help quickly. On a team made of T-shaped people, it’s possible to have the help you need no farther than a colleague away.But how to find the skills you need on a team? Is that something the PM needs to worry about?

Self-Organizing Teams

❖ People know their skills

❖ Team has freedom to change assignments to suit skill set, adapt to change

❖ PM doesn’t have to plan it

The best architectures, requirements, and designs emerge from self-organizing

teams.11th Agile Principle

agilealliance.org

This is an area in which a history of command-and-control can really show up. Some PMs will want to assign each task themselves, to “dole out the work.” But under Agile, the team makes those decisions as a group. The team is accountable for all tasks getting done, not individual tasks.

One way to approach this is to casually suggest that the PM could reduce workload by leaving this field unset, and having each team member grab things that suit them.

Conflict Point: Late Additions

Ugggh, please don’t hire more people! I’ll have to train them

instead of coding, and we’re already late!

Looks like we’re running late - the team will be glad to hear I got us 3 more engineers!

“We disagree about what adding people will do.”

Dev / Ops: PO / PM:

This is one is counterintuitive, but very well backed up.

Hurrying the Baby

If 1 woman can make 1 baby in 9 months…

surely 9 women can make 1 baby in 1 month!

There are some tasks that can’t be parallelized, often due to ramp-up time. It might take you 4 months working alone to finish, and 3 months for a new engineer to ramp-up; if so, you’re not gaining more than a week or two, at great expense.

The Mythical Person-Month

❖ Communication burden

❖ Training burden❖ Split too-large team?

Fred Brooks

Much worse than ramp-up, though, is the intercommunication problems that occur. As a team grows in size, the number of connections is a combinatorial, which grows quickly. Teams larger than 5-7 individual contributors a generally impractical, and a team split to accommodate larger numbers late in the schedule is likely to be very disruptive as well.

Aside from communication, PMs and POs are often blind to the fact that ramp-up is not a solitary activity. The existing team is called upon to train up the new members - and they are thus less productive.

Alternatives to Team Growth

❖ Reduce Scope❖ Extend Schedule❖ Choose how to fail

alliekf via flickr / CC-BY

If adding people will only postpone an already-late software project, what else can be done?Reducing scope means cutting down on features. That means there is less to do, and if done carefully, could bring the project back on track.Some customers will be more willing to accept a late delivery than one with missing parts.But when there is no other choice, you still can choose how to fail:- unresolved tech debt- unfixed defects- poor docs- manual automation- other contractual violationAll of these choices are represented by the PO - both to the customer and to the business. A successful PO will have kept the customer involved in these discussions, and collaborated on which shortcoming would be best for the customer.

Conflict Point: Ops Needs

Every service must be scalable, monitorable, manageable -

basically, operable.

The customer didn’t ask for monitoring; we’re not going do it.

Later: Why do we keep getting surprised by outages?

“Operations needs are not considered until too late.”

Dev / Ops: PO / PM:

These are some of the “non-functional requirements” that haunt software projects. They often arise from miscommunication and assumptions. For example, the customer may assume that their website will be fast and available 24/7 - but since that is never said aloud, those feature never become user stories, and never get allocated resources.

Operational Needs

❖ Ops on the team early❖ Whole team can add missed

stories❖ Focus on working deployed

software ❖ Make tradeoffs clear

❖ inoperable == missed SLAskevin o’mara via flickr / CC-BY

This is an organizational change, and the people in this room may not be able to make it. If you’re a developer, and a new project is forming, loudly ask, “Where are the Ops people?” When a team forms, get people with operational experience on the team as early as possible. As user stories are written, keep the focus on the Agile value - working software. Until the software is deployed and working, it is without value. So, having the software be operable is a key component of its value - and so must be included in the backlog.

Joy of DevOpsBiz

❖ It’s a tough climb❖ Patience and low

expectations❖ Talk to each other!

tetsumo via flickr / CC-BY

Over time, if you keep up the one-on-ones, and build trust, you can gradually reach a positive relationship with your PO and PM. Your work environment will improve, you will feel listened-to, and your stress level will decrease. And you’ll help deliver those great big sack of money that the business wants :-)

Reach Out @clintoncwolfeomniti.com

devopsdictionary.com

marji beach via flickr / CC-BY

So in conclusion, reach out to your PO. You may be different, but you have common goals.