“Your systems are currently optimized to deliver the results you are now getting.

If you want different results you have to change the system. To expect anything

else is by definition, insanity.” ~ Joseph Flahiff

June 16, 2015

Michael Cockrill, CIO

Washington State Office of the CIO

Dear Michael,

Attached you will find the WA Business Hub Project’s QA report for May. This is only the second

report sent. From now on they will be coming on a monthly basis. Since this approach to QA is a

work in process you may see some categories changing, being added, or being removed.

It will appear that the overall numbers have gone down but it really is an effect of the number of

categories changing.

Thank you again for this exciting and innovative project, I am proud to be a part of it.

Sincerely,

Joseph Flahiff,

CEO

Whitewater Projects, Inc.

206.276.1386

joseph@whitewaterprojects.com

WA Business Hub Project

Agile Project QA

Prepared for: Michael Cockrill, CIO

Prepared by: Joseph Flahiff, CEO

Description

An independent assessment of the agile practices on the WA Business Hub project. This

report includes an executive summary and detailed analysis of the project’s agile project

delivery practices.

QA for WA Business Hub Project: WA OCIO We want you to be successful

Executive Summary

Project Assessment

This project is a follow on to the research project held in 2014 to address Senate Bill 5718 which found

that, “regulatory agencies were directed through an executive order in 2006 to develop a one stop

business portal, but that a one stop business portal has not been developed.”

The project is early in the startup and is being run with the Scrum framework. The work began in

2014. The project budget is $737,114

Findings Summary

As stated in the last report in January, they know what product they are developing but they do not

have a well-defined and prioritized/ordered backlog. This results in some confusion and a lot of

mixed priorities.

The culture of the team is its greatest strength.

I am optimistic about the chances for success of this project based upon the culture, teamwork and

their receptivity to suggestions.

A few items will be critical to this success. These items include:

Reducing the interruptions from other work

Obtaining feedback from actual customers

Establishing a good backlog of stories that can be used to forecast the completion of the

project

How to read the assessment The agile coach/independent QA observed the team and provided interactive coaching to help the

team improve their agile practice. The assessment includes an evaluation of the team in 4 areas.

The 4 Aspects of Agility

Every organization is a complex system that can be described with 4 key aspects:

Business processes

Technical practices

Organizational Culture

Leadership Approach

Business Processes - Business Processes are the set of continuously evolving and improving; processes,

tools, beliefs and policies that support the business. Agile/nimble business systems enable managers and

leadership to make quick decisions and they support the prioritization of work that enables the delivery of

customer and business value.

Agile/nimble business processes support the technical practices and are enabled by the leadership. They are the

systems that enable teams, business leaders, analysts and managers to easily work together in tight knit groups

to deliver customer value.

Technical Practices - Technical Practices are the practical tools and techniques used by teams,

management, and leadership to deliver customer value quickly and incrementally.

Flexible Technical Practices do not merely make exceptions for the possibility of change, they are based upon

the certainty of change and are designed to leverage it for the benefit of the customer and the business.

Organizational Culture – Organizational Culture is a set of organizational beliefs and behaviors. In

organizations that supports the ability to adapt to changes as they come. A high value is placed on being

transparent and realistic, even about the ‘bad news’ these cultures focus on delivery of one thing, innovation

and customer value above all. These organizations have fully engaged teams that self-organize with motivated

team members who collaborate in real-time. A mindset of continuous improvement is key to the long-term

success.

Leadership Approach- The leaders’ job is twofold. The first job of the leader is to create psychological

safety for the teams, empowering them to innovate. Second, leaders need to provide clear strategic intent,

guidance on WHAT is to be done helping the team focus on the value that they are delivering to the customer.

Measurement The following scale was used to evaluate the team. These results were shared with the team and they

agreed with the assessment.

Scale

0 indicates does not exist

1: indicates exists but not understood

2: indicates exists and understood but not improving

3: indicates is in a kaizen state

N/A Not Applicable to this context

Risk Ranges:

Low Risk = Average between 3 and 2.1

Moderate Risk = Average between 2 and 1.1

High Risk = Average between 1 and 0

Project Summary

Jan-2015 May 2015 June-2015

Technical

Practices High Risk

Medium

1.25

Business

Processes High Risk

Medium

1.4

Organizational

Culture Low Risk

Medium

1.81

Leadership

Approach Low Risk

Low

2.42

Observations

Technical Practice

Technical Practice Observation Recommendations Response

Pairing 2 – Some swarming and some

pairing.

Team will continue to push to

reduce work streams.

Continuous Integration/

Continuous Delivery

2 – Vendors are using GIT Hub. Pairing is occurring within

every recent sprint and

swarming was exceptional

recently.

Loosely coupling systems 2 – They do it but are not really

improving it.

New QA plan and definition of

done will require continuous

integration. This process was

just documented and reviewed

with team.

Refactoring 1 – some is being done but not

consistently

I believe this is more of a

symptom at this time than a

cause. It is difficult to refactor

when the team is not focused

on one project.

Plan for Business Licensing

Guide is to decouple it into its

own app, separate from the

site.

Emergent Design 1 – It is happening but not at a

conscious level

Agreed. Quite a bit of

refactoring happens because

of the separate work streams

and it is not currently

purposeful.

Test First Development 0 – Not really doing this yet.

Technically don’t have the skills

in PHP

More attention could be

placed upon developing

acceptance criteria for stories.

Agreed. Team unfortunately

sacrificed user engagement the

past few months to gain dev

cycles. It wasn’t an easy

choice.

Collective Code Ownership 3 – Still improving. Ryan has

integrated well with this

culture.

+1 Goal is to include user

community after v1.

Coding Standards 2 – developed standard on

Accessibility, compliance, API,

etc… given to the vendors and

following internally,

There is not clear

understanding across the team

about this issue. Having a brief

discussion would help.

Somewhat disagree. Current

state is we faltered in this due

to terminating work with

primary vendor, but it was in

place and awareness of it

exists.

Business Processes

Business Process Observation Recommendations Response

Backlog 1 – Partial backlog exists. But it is

not sufficient to plan the work

Improvements have been

made but the team is so busy

delivering that it has not

backed off to plan.

Agreed. We have developed a

complete backlog for features

like search and BLG—an

improvement--but not for the

entire site when stories are

created typically prior to the

sprint.

Will continue to press on this

important issue.

Velocity/Throughput tracking 1 – They are tracking it but Do

not appear to be doing

anything with it. Commitment

is consistently higher than

delivery

Agreed. To further use velocity

as a predictive measure, we

need a backlog to predict.

Currently, It is used solely used

to measure yesterday’s

weather to estimate sprint

commitment.

Stakeholder Demo 3 – Their demo from what I

saw was good.

No additional emphasis need

be put on this topic until the

team can focus and is not

randomized.

Thanks!

Customer Demo 0 – There appeared to be one

planned but it did not come

through.

No customer demos yet. Agreed. Unfortunately, we cut

the planned customer demo

work due to capacity issues.

Vertical slicing of customer

value for stories

2 – they are working on it.

Stories still do not appear small

enough.

No additional emphasis need

be put on this topic until the

team can focus and is not

randomized.

Noted.

Stories meet INVEST criteria 2 – Do not really understand

Value for the stories at this

point and stories are not very

negotiable

Ben needs to better articulate

value to the customers and

stories should be focused on

the why and some of the what

but not the how.

Agreed. Value is the piece

missing from the cards. Plan is

to introduce hypothesis driven

development after V1 launch to

supply foundation for

measuring value.

Daily Standup 3 – they are improving on the

practice

Thanks!

Story to express intent 3 – Kudos. (Told but not

validated)

Thanks!

Small Release Cycles of

product

2 – Released search,

autocomplete is released,

contact suggestion, others to

Business.wa.gov every sprint

Agreed. Team really wants to

do more of this.

Incremental development and

release of product

1 – Product hasn’t been

released yet. But they are

doing incremental

development

Identify the MVP, the stories to

complete it and estimate the

date.

Somewhat disagree with “1”

score. We have released

incrementally several features

for search, especially.

Iterative improvement of

product

1-Getting input from the search

vendor and curating content

based on the feedback.

Ibid Somewhat disagree with a “1”

score. We did a lot of curation

and tweaking of content to

improve search results for

users and tracked those

significantly improved results

via user behavior.

Will publish these for broader

awareness.

Do team members know that

they are working on the most

important thing every day

1 – Stories are almost all the

same size. (1-2-3 points) limit

of 3 in motion. Stack ranked

with an interrupt buffer.

Create an experiment of your

own to define what work is top

priority and work on it.

Agreed. Will emphasize during

sprint planning.

Clearly articulated and

understood vision for the

overall effort

2 - Agreed.

Vendor Management 1 – Out of control. Agreed.

We came up with an as yet

unimplemented idea for a

Vendor Playbook, as well as a

yet unfilled role of Vendor

Manager to clearly identify the

responsibility. These two

actions are on our backlog.

Is value measured? 0- Agreed. It is not.

Is the project delivering more

value than it is costing to run?

0-

Does the team use a team

developed definition of done?

0 - Don’t a DOD but not

everyone knows what it is.

Take 20 minutes and define a

DOD.

Completed. We now have a

definition of done.

Does the team, on a regular

basis, reflect on how they are

working together and make

improvements

3- Consistently holding them

and identify 1-2 actions to

improve.

Clearly identify the experiments

and make them visible in the

room.

Use the 5 part sprint retro

format.

Agreed.

Will try the recommended

format in upcoming sprints.

Organizational Culture

Culture Trait Observed Recommendations Response

Commitment to the

team

3 – Really committed and passionate.

They are not excited about the

Vendors though.

Keep it up but do not over commit.

Your health and your family are

more important than the work. See:

Sustainable Pace below

Thanks, they are super committed.

Agreed that sustainable pace needs to

happen.

Interdependence 1 – It happens but not consistently and

not by design. “random happy

accidents”

I suggest that the team try to work

in pairs more but this really isn’t a

high priority. The team is

randomized too much to make

interdependence actually pay off.

Agreed. Team is progressing to more and

more pair coding work. Additionally, we

have built into our definition of done the

requirement of paired code review prior to a

card being marked “done.”

Sustainable Pace 1 – Several people related that they

were working on the weekend or

evenings, it was said in the context

that they are committed to the team.

Encourage NOT working on the

weekends but pushing back on

management to create realistic

working expectations.

Agreed. Will continue to try and give air

cover in this regard. This team needs to

focus.

Focus (One thing at a

time)

1 – This is a constant struggle. People

are randomized and pulled off for

other work. Plus the time spent

managing vendors.

This is an issue mostly for leadership

(Michael D. and Ben) to address.

This is the single most important

issue for the organization. If they

can reduce the amount of

randomization from leadership, they

will improve their productivity and

can focus on some of the other

areas.

Agreed. We will work harder on focus.

Common Goal 2 – They do this well. They all know

the common goal.

No additional emphasis need be put

on this topic until the team can

focus and is not randomized.

Agreed.

Positive Failure 2 – In their team improvements they

are safe to fail. Produce piple-line

work (e.g. can we deliver BLG) it is not

ok to fail on.

Vendors are a 0 we don’t fail and

cannot fail with them because of the

vendor management process.

No additional emphasis need be put

on this topic until the team can

focus and is not randomized.

Agreed. Team has progressed on

embracing an experimental, curious attitude

towards failure and leveraging it to improve.

Restating vendor management issue is that

contract(s) have constrained vendor choice

too much, procurement reduced time to

deliver, ownership of delivery was unclear,

and requirements missed.

New role implemented to oversee this.

Collaboration 1 - No additional emphasis need be put

on this topic until the team can

focus and is not randomized.

Agreed. We will work on increasing focus.

Flow of Value or

Utilization Focus

1 – The focus on vendors is

detrimental to focusing on value, it is

mostly focusing on utilization.

Additionally the randomization by

other people and departments is a

utilization not value focus.

No additional emphasis need be put

on this topic until the team can

focus and is not randomized.

Agreed. We will work on increasing focus.

Psychological Safety 3 Thanks!

Team spirit - esprit de

corps (Happiness)

3 Thanks!

Is there clarity about

responsibilities/

accountabilities and

corollary authority

(Roles/whole

team/etc)

2- No additional emphasis need be put

on this topic until the team can

focus and is not randomized.

Agreed.

Leadership Approach

Leadership Trait Observed Recommendations Response

Servant Leadership 2 1:1 coaching for the leaders of

the organization is

encouraged. They could even

coach each other.

Agreed. This is a great idea.

We’re exploring the notion of

coaching as a service.

Continuous involvement 2 This is impacted by the leaders

(PO) being randomized, as

much if not more than others

on the team.

Agreed.

Individual leadership is

encouraged

3 Holacracy has helped in this Thanks!

Encouraging Decision Making

at the level of the information

2 They are working on this. Agreed.

Does the leadership foster

psychological safety

3 The team seems to support the

holacracy approach.

Agreed. Team seems

confident and trusting of each

other, and have made great

gains in this area.

Does the leadership provide

lean/agile coaching for the

team

3 Though external training,

consultants and coaches.

Thanks! We brought in Joe

Justice and look for other

opportunities in this area as

well.

Does the leadership focus on

communication of strategic

intent rather than details and

micromanagement

2 No additional emphasis need

be put on this topic until the

team can focus and is not

randomized.

Agreed.