bringing science to software development

Janelle Kleinopenmastery.org @janellekz

Tour Speaker

Bringing Science to Software Development

How to Measure the PAIN in Software Development

Janelle Klein

http://@openmastery


Janelle Klein

After five years…

Data-Driven Software Mastery

leanpub.com/ideaflow

http://leanpub.com/ideaflow


Janelle Klein

==

2.0

Visibility Changes Everything

Visibility

+

Lean Manufacturing

Then I read this book!

“The Art of the Scientist” Lean Startup Workshop, Ash Maurya

Ash: “The goal isn’t learning, the goal is traction.”

==

2.0

+

Traction (Lean Startup)

We get a design template for how to build a learning organization.

Visibility Changes Everything

News Flash!For the first time in our industry,

we have all the tools necessary

to pull this off.


Five Talks

Stop Getting Crushed By Business Pressure

How to Build a Learning Organization

Top 5 Reasons Why Improvement Efforts Fail

Make a F.O.C.O.L. Point!

This Talk…

Using Science to Fuel Software Mastery(team & organizational level)

Five Disciplines of a Learning Organization

Personal Mastery

Mental Models

Shared Vision

Team Learning

Systems Thinking

These disciplines are emergent with science.

, Developer & Consultant for 15+ yearsSpecialized in Statistical Process Control (SPC) and Supply Chain Optimization from Lean Manufacturing (I <3 cool data)

Continuous Delivery infrastructure, design & automation strategies

Janelle Klein

Who Am I?


Janelle Klein

leanpub.com/ideaflow


I’m also a hobbyist Cognitive Scientist

CTO @

Founder @Industry Peer Mentorship Network

My Job

Summary from Talk #1


Learn & Adapt

Great Team Disciplined with Best Practices Constantly Working on Improvements+

Project FAILURE

About 8 years ago…

Technical Debt Mistakes

I thought the main obstacle was Technical Debt

?Most of our mistakes were in the

most well-written parts of the code.

Mistakes

We made significantly more mistakes in code that we didn’t write ourselves.

Lower Familiarity

More Mistakes=

There had to be more to the story...

Complex(So*ware(

PAIN

This is what I knew...

What made development feel painful?

Unexpected Behavior

Problem Resolved

Measure Painful Interaction with the Code (Friction)

Troubleshooting

Progress

5 hours and 18 minutes of troubleshooting...

PAINFUL

The amount of PAIN was caused by…

Likeliness(of((Unexpected(Behavior(

Cost(to(Troubleshoot(and(Repair(

High(Frequency(Low(Impact(

Low(Frequency(Low(Impact(

Low(Frequency(High(Impact(

PAIN(

What Causes Unexpected Behavior (likeliness)?

What Makes Troubleshooting Time-Consuming (impact)?

Semantic Mistakes

Stale Memory Mistakes

Association Mistakes

Bad Input Assumption

Tedious Change Mistakes

Copy-Edit Mistakes

Transposition Mistakes

Failed Refactor Mistakes

False Alarm

Non-Deterministic Behavior

Ambiguous Clues

Lots of Code Changes

Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

Most of the pain was caused by human factors.

What causes PAIN?




Ambiguous Clues


Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger

What causes PAIN?

Semantic Mistakes


Association Mistakes



Copy-Edit Mistakes



False Alarm


PAIN occurs during the process of understanding and extending the software

Complex(So*ware(

PAIN

Not the Code.

Optimize “Idea Flow”

My team spent tons of time working on improvements that didn’t make much difference.

We had tons of automation, but the automation didn’t catch our bugs.

My team spent tons of time working on improvements that didn’t make much difference.

We had well-modularized code,

but it was still extremely time-consuming to troubleshoot defects.

The hard part isn’t solving the problems it’s identifying the right problems to solve.

“What are the specific problems that are causing the team’s pain?”

measures the time spent on:

Idea Flow

x

Troubleshooting

x

Learning

x

Rework

Quality Risk Familiarity Risk Assumption Risk

Idea Flow Mapping Tools

(Open Source, Supported GA ~June 2016)github.com/ideaflow/tools

http://ideaflow.org/tools

“What caused the pain in this case?”

Categorize the Problems with #HashTags

#ReportingEngine

#Hibernate

#MergeHell

1. Test Data Generation

2. Merging Problems

3. Repairing False Alarms

1000 hours/month

Add up the Pain by Category

0%

100%

Release 1 Release 2 Release 3

Percentage Capacity spent on Troubleshooting (red) and Learning (blue)

Chaos Reigns Unpredictable work stops

fitting in the timebox

Troubleshooting

Progress

Learning

Analyzing Long-Term Trends

Start%Over%

Unmaintainable%So0ware%

Realization: Repeating the Same Mistakes

From the Outside:

This is the pattern of a broken feedback loop.

We Work Under Constant Urgency

Time%Pressure%

Compromise%Safety%for%

Speed%

Increase%Number%&%Severity%of%Hazards%

%

More%Pain%and%Higher%Task%Effort%

Constant'Urgency'

Cycle of Chaos High-Risk Decision Habits

0%

100%

Release 1 Release 2 Release 3

Troubleshooting

Progress

Learning

Percentage Capacity spent on Troubleshooting (red) and Learning (blue)

(extrapolated from samples)

Gambling Habit —> Gradual Loss of CONTROL

Fewer%Problems%to%

Fix%

Stop%%and%Think%

Mi8gate%the%Risk%

Increased%Produc8vity%

and%Innova8on%

Safety'

Cycle of Safety Low-Risk Decision Habits

Optimize for Safety

We don’t improve productivity by trying to go faster, we improve productivity by improving control.

The Safety Principle

Product(Requirements( So1ware(Task(

Possible(Decision(Paths(

So1ware(

Software is a Reflection of our Decisions


High%Risk%Decision%Habits%

So1ware((and(all(our(problems)(

(

RESET



High%Risk%Decision%Habits%

So1ware((and(all(our(problems)(

(

Technical Debt is just a Symptom of the Problem


Learn & Adapt

Just because we repair the car… doesn’t mean we know how to drive.

Quality Target

Lower Control Limit

Upper Control Limit

X"

This is “Out of Control”

Lower Variability = Better Control

Steering Wheel = Process Control

“Pain Control” in Software Development

Optimal Friction

Upper Control Limit X"

“Out of Control”

20min

0m

Average Pain per Incident

This is Better

Target

Control Limit

Categorize the Pain Type

Input: Decision Constraints

Target: Optimize the Rate of Idea Flow

short-term loop long-term loop

1. Visibility

2. Clarity

3. Awareness

F ocus!

Output: “Friction” in Idea Flow

Improve Quality of Decisions

Idea Flow Learning Framework




1. Visibility

2. Clarity

3. Awareness

F ocus!


Target - The direction of “better”






1. Visibility

2. Clarity

3. Awareness

F ocus!


Input - The constraints that limit our short-term choices…





1. Visibility

2. Clarity

3. Awareness

F ocus!


Output - The pain signal we’re trying to improve





1. Visibility

2. Clarity

3. Awareness

F ocus!


Focus on the biggest pain…

F ocus!





1. Visibility

2. Clarity

3. Awareness

F ocus!


1. Visibility - Identify the specific patterns

1. Visibility





1. Visibility

2. Clarity

3. Awareness

F ocus!


2. Clarity - Understand cause and effect

2. Clarity





1. Visibility

2. Clarity

3. Awareness

F ocus!


3. Awareness

3. Awareness - Stop and think to adjust habits





1. Visibility

2. Clarity

3. Awareness

F ocus!


4. Run Experiments to Learn What Works


Optimal Friction



20min

0m


This is Mastery

Target

Control Limit


This is the Process of Mastery

This Talk…

Using Science to Fuel Software Mastery(team & organizational level)

“The Art of the Scientist” Workshop, Ash Maurya

Ash: “What’s the most important thing that scientists do?”

“Run Experiments!”

“Analyze Data!”

Getting Involved with Lean Startup


NO!

??

?

?

?

??



Ash: “The most important things scientists do is build models.”


Ceiling of Achievement, Ash Maurya

Lots of learning, but not learning the “right things.”

Floor of Improvement?

Lots of improving, but not improving the “right things.”

Test the Model (Experiments)

Refine the Model (Patterns)

False&Predic,on&

Scien,fic&Model&

Modeling (Inductive)

Experimentation (Deductive)

Empirical Data

The Scientific Method

We can’t ignore this part.



False&Predic,on&

Scien,fic&Model&

Modeling (Inductive)

Experimentation (Deductive)

Empirical Data

Science is a Feedback Loop that Fuels Discovery

Creating an explicit model makes our beliefs testable and our discoveries additive.

We can’t ignore this part.

“The Art of the Scientist” Lean Startup Workshop, Ash Maurya

Ash: “The goal isn’t learning, the goal is traction.”

Optimize the Rate of Flow in a Supply Chain

Focus on Limiting

Constraint

(Goldratt’s Theory of Constraints, “The Goal”)

“The Customer Factory”, Ash Maurya

Optimize the Rate of Happy Customer Flow

Constraint

Optimal Friction



20min

0m

Average Friction per Customer

This is Traction

This is the Process of Lean Startup

High Friction

Min Friction

“No” Threshold

Customer Happiness

Categorize the “No” Type

“The Idea Flow Factory” (supply chain model)

Optimize the Rate of Idea Flow

Across the Software Supply Chain

100 hours

50 hours

Troubleshooting

Learning

Rework

Focus on the Biggest Pain Type

Focused Effort = Visible Improvement

Optimal Friction



20min

0m


This is Traction

Target Rate

Control Limit


This is the Process of Idea Flow Mastery

We can apply the entire suite of cost & risk reduction tools from Lean Manufacturing!!

Supply Chain Optimization

Statistical Process Control (SPC)

Optimal Friction



20min

0m


Target

Control Limit

Throughput Accounting



False&Predic,on&

Scien,fic&Model&

Clarity (Why)

Awareness (How)

Visibility (What)

Use Science to Fuel “Explicit Mastery”

Model “Decision Principles”

What’s a Decision Principle?

1. How do I evaluate my situation?

2. What should I optimize for?

Answers Two Questions

The Haystack Principle

Lots of unvalidated changes

Easier to find the needle.

Decision Principle: The Haystack Effect

Optimize for small manageable haystacks.




1. Visibility

2. Clarity

3. Awareness

F ocus!



Systematically codify Decision Principles from lessons learned

Three Stages of Mastery

Visibility

Clarity

Awareness

The Visibility Principle

Our mind filters what we see and experience

in terms of the “shapes” we know.

George Lakoff

Metaphors We Live ByJeff Hawkins

On Intelligence

+

How do we “see” (and not see)?

Observe Pattern

“Do I know this pattern?”

Metaphors are patterns of conceptual “Shapes”

Three Different Types of Metaphors

Object Patterns (Thing)

Context/Relationship Patterns (Space)

Process Patterns (Time)

We “see” the world through a fabric of metaphor.

Technical Debt

Debt is a “Thing”

“This looks like a tech debt thing.”

1. Categorize patterns by concept type

“This looks like a painful tech debt thing.”

2. Categorize variations by intensity.

Pain level = 8

= Little Debt

= Medium Debt

= Big Debt

Our Beliefs about Developer Pain

PAIN occurs during the process of understanding and extending the software

Complex(So*ware(

PAIN

Pain occurs over TIME.

What do you think happens when we make this transition?

Process Patterns (Time)

Object Patterns (Thing)



Take a Closer Look at the Patterns

Familiarity Mistakes


Semantic Mistakes



Copy-Edit Mistakes



False Alarm


Ambiguous Clues


Noisy Output

Cryptic Output

Long Execution Time

Environment Cleanup

Test Data Creation

Using Debugger


“What caused the pain in this case?”

We Start to Experience the World Differently

#ReportingEngine

#Hibernate

#MergeHell

2.#Record#an#Idea#Flow#Map#

So#ware(Task(Strategy((Design(

Experience(Review(

Iden:fy(the(Pa>erns(

1.#Clarify#the#Strategy#

So#ware(Task(

3.#Reflect#on#Decisions#

Focus(Mee:ng(

4.#Share#Discoveries#

Visibility Framework


Visibility

Clarity

Awareness

The Clarity Principle

Our understanding deepens when we explain

the similarities and differences of experiences

and the relationships of cause and effect

JourneyBeginning End

Think about the Developer’s Experience as a Problem-Solving Journey

Journey

Constraints

What were the biggest challenges on the journey?

Friction

Journey


Journey Strategy


Journey Uncertainty


???

Situation Strategy Consequence

Developer Journeys

Breakdown the Patterns in Developer’s Experience

Archetype Journeys Archetype Journeys

Journeys that Repeat


Decision Principles What are we trying to optimize for?


Pain TypeFrom the “Ten Pains” of Software Development

Circle Leader

Circle Member

Focus: What’s the problem to solve?

Observe: Ask questions about the facts

Conclude: What’s causing the pain?

Optimize: How do we reduce pain?

Learn: What questions does this raise?

Observation Questions

Make a F.O.C.O.L Point!

Summarize lessons learned in a #HashTagged Blog.

Deep Reflection in Open Mastery Circles

Clarity Framework (Why)Repeat&Weekly&

Focus&Mee1ng&

1.#Explain#Decisions# 3.#Search#for#Archetypes#

4.#Model#Decision#Principles#

Zoom In Zoom Out

2.#Model#the#Journey#

Focus#

Reset Password Login

Username:

Password:

To continue to our site, please login:

Task: Implement THIS

George’s Experience Review

1 hour 10 min to find this bug.

Me: "So first, George, can you describe what you were trying to do before you encountered the conflict?”

George: "This was back when I was getting the token to work in the email. I wrote the code to generate the token, edited the email template to include the token in the email link, but the token wasn't showing up in the actual email."

Me: "So what did you do next?"

George: At first, I thought there was something wrong with the token variable code, but the problem turned out to be a misspelled variable name in the email template. Doh!"

Focus: What Major Factors Caused the Pain?

Observe: Ask Questions About the Facts.

Conclusions: What Caused the Pain?

Why did it take 1 hour 10 min to find this bug?

Painful Consequences: Consequence

Conclusions: What Caused the Pain?

Ran experiments by sending emails over and over again (23 emails with a missing tokenId)

Redeployed application when uncertain about the cause

Ambiguous Clue

Manual Execution (slow)

Optimize: How could we have avoided the pain?

What’s an alternative strategy that could have mitigated these risks?

George: My main strategy was to do a small #VerticalSlice by getting the “email send” part working, before adding the token logic for the security part. I understand how the #ManualExecution and an #AmbiguousClue caused the friction, but I'm not really sure what I would do differently if I could do it again.

Me: "George, can you describe your original strategy to mitigate risk?”


What’s the strategy pattern we’re trying to implement?

Alternative Strategy:

APIs for each email workflow

Wire up email last

Strategy


Alternative Strategy:

APIs for each email workflow

Wire up email last

Situation

What’s the situation we need to recognize in order to decide to use the strategy?

Situation Strategy Consequence

Developer Journeys

#HiddenOutputs Create #ObservableInterfaces

Avoid #AmbiguousClues & #ManualExecution


Learn: What questions does this raise?

What questions should we ask ourselves in the future?

In what situations should we ask the question?

Journey

Field Guide

What questions should we add to the field guide?

Situational Questions

Look for Patterns Across Experiences


Journeys that Repeat

Describe the Generalized Causes of Pain

Experiment Time

Our Perception of Time is WAY OFF

Setup Experiment Analyze Results & Decide Next Experiment

Execute

waiting - time goes slow

doing - time zooms by

“Experiment Pain”

16:10

Optimizing for Human Cycles

The Code Sandwich Principle

The thickness of the sandwich increases

troubleshooting difficulty

Behavior Complexity

Observability

Ease of Manipulation

Optimize for experimentation.

Optimize for additive clues.

The Narrowing Cloud Principle

The bigger the cloud of possibilities, the more we need a strategy.

The Clarity Principle

Our understanding deepens when we explain

the similarities and differences of experiences

and the relationships of cause and effect


Visibility

Clarity

Awareness

The Awareness Principle

We can only avoid a mistake

if we predict the consequences

of the decision in the moment.

18:120:00

George’s Painful Experience

I know I made a big haystack, I’ll do better next time.

This is hindsight bias, it’s not that easy!

The Experience Review

The ChallengeWe tend to make auto-pilot decisions and do what’s familiar

How do we break decision habits?

Imagine your brain is a decision-making engine

written in code.

BreakpointStop and Think!

Changing Decision Habits

BreakpointStop and Think!

Changing Decision Habits

We have to predict the big haystack during the decision.

The Checklist Manifesto

Atul Gawande

Scaling Lean (Lean Startup)

Ash Maurya

+

Change Decision Habits with Strategy Experiments

14:230:00

I want to avoid this…

Thinking Checklist

Is my current approach likely to cause a big haystack?

Situation: start of subtask

Let’s Make a Checklist!

“What question could I ask my future self to recognize similar risks in the future?”

“In what situation would I ask the question?”

0:00

Stop and Think:


Predict: Small haystack

Strategy Experiments

18:120:00

Stop and Think:


Predict: Small haystack

False Prediction


18:120:00

Stop and Think:


False Prediction


High-Risk Situations

1. Unraveling sweater 2. Integration-heavy change 3. High state variation 4. Minimum scope is big

Q: Is my current approach likely to cause a big haystack?

Start of Subtask

Strategy Types (“do”)

1. DependencyAnalysis 2. IncrementalIntegrationTest 3. DataDrivenTest 4. IsolateHardToTestCode

High-Risk Situation Types (“see”)

1. UnravelingSweater 2. HeavyIntegrationLogic 3. HighStateVariation 4. CoupledExternalDependencies

Haystack Decisions

Codify What Works

DoSee

So#ware(Task(

Software Task Strategy((Design(

Thinking(checklists(in((key(situa6onal(contexts(

!(!(!( !(!(

!( !(!(!(

Experience(Review(

!(!(3.#Refine#the#

Checklists#

!(!(!(

Failed(Experiments(

Successful(Experiments(

2.#Run#Checklist#Experiments#

1.#Design#Thinking#Checklists#

+(

Awareness Framework (How)

The Awareness Principle

We can only avoid a mistake

if we predict the consequences

of the decision in the moment.


Visibility

Clarity

Awareness

See

Explain

Predict




1. Visibility

2. Clarity

3. Awareness

F ocus!



Systematically codify Decision Principles from lessons learned

Five Disciplines of a Learning Organization

Personal Mastery

Mental Models

Shared Vision

Team Learning

Systems Thinking

These disciplines are emergent with science.


Janelle Klein

These are the Blueprints!

++

How Do We Scale Up?

Collaboration to Perform a Function (Team)

Social Network of Brains

Component




1. Visibility

2. Clarity

3. Awareness

F ocus!


How do my decisions affect your experience?


Management

Dev Team

Fire x10

What if our organization was a robot?

If the feedback loop is broken, we burn.

Pain Sensor

Bigger Social Network of BrainsCollaboration to Perform a Bigger Function (Organization)

Component

ComponentHub

Hub

Hub

The Challenge: Knowledge & Decision-Making are Distributed

Learn & Adapt

Role

Decision(Type(

Required(Knowledge(

Visibility and Decisions Coupled

Visibility and Decisions De-coupled

Role A

Required(Knowledge(

Role B

Decision(Type(

ROOT CAUSE of the Broken Feedback Loop

Manager

Alloca&on(Decisions(

Knowledge(of(Risks(

Risk(Mgmt(Decisions(

Developer

Communication Breakdown

Broken Feedback Loop (Manager Role)

So#ware(Task(

Mi.gate(the(Risks(

Product(Development(

Product(Work(Queue(

Risk(Mgmt(Work(Queue(

Product(Owner(

Product(Decisions(

Knowledge(of(Customers(

Technical(Risk(Manager(

Risk(Mgmt(Decisions(

Knowledge(of(Risks(

Dev$Team$Capacity$

Manager

Alloca.on(Decisions(

Turn Risk Management into an Investment Decision

Now we can steer!!

Three Dimensions of Organizational Knowledge

Discovery (Expansion)

Peer Mentorship (Diffusion)

Optimize the Whole (Acceleration)

“Decision Principles” (models)

Discovery (Expansion)

Peer Mentorship (Diffusion)

Optimize the Whole (Acceleration)

Priority 1

Priority 2

Priority 3

Priority 1

Priority 2

Priority 3

Priority 1

Priority 2

Priority 3

Biggest Pains

$$Biggest Skill Gaps

$Limiting Constraints

$$$$

Turn Risk Management into an Investment Decision

Decisions are managed by a distributed Network of Open Mastery Circles

4. Optimize the Whole

2. Design the Strategy

1. Identify Obstacles (Focus)

3. Learn What Works (Traction)

Steering Wheel

Create a Steering Wheel to Drive the Organization!

$$ $$

Focus: Quality of Improvement Decisions

Focus Traction

$$$ $$

Focus: Quality of Investment Decisions

Risk Translator

(interface)

Management(Coordination)

Engineering(Execution)

Open Mastery Learning Framework

Idea Flow Pain Sensor

If your Interested in doing this:

Free Support Network for Data-Driven Software Mastery

Open Mastery Learning Platform

Idea Flow Mapping Tools

Team Mastery Platform

TeamJoe Sally Mark Eric

Community Collaboration Platform

Anonymized Data

ProjectTiger

ProjectBear

(REST)

Shared #HashTag Glossary of Patterns & Principles

with Examples

The Conversation is on Slack

We’re debating the patterns and principles of Idea Flow in Wikipedia

It’s a Developer Party!

We’re sharing our ideas over Blogs

(and managers are invited)

Industry Peer Mentorship Network

Companies

Community Groups

HQ in Austin

Open Mastery Austin

meetup.com/Open-Mastery-Austin

http://meetup.com

Janelle Kleinopenmastery.org @janellekz

Check out openmastery.org for details.

Read my Book.

Think About It.

FREE with Reading GroupBuy It


Janelle Klein

http://@openmastery

http://openmastery.org

openmastery.org

@janellekz