it operations in agile israel april 11

Breaking the Vicious Cycle in IT Operations

using an Agile/Kanban RecipeTal Aviv

[email protected] [email protected]

mailto:[email protected]


FORCES THAT SHAPE IT SERVICES

A little bit of history: Where are we today and how did we get here?

Growing Pains – How did we get here?

• Organizations started with a single Person IT department.

• Growth led to increased IT needs.• Growth led to Complexity.• Complexity led to splitting IT into

specialized groups.• Pressure for Cost cutting led to off shoring

and out sourcing.

Organization growth

Additional rules and procedures are

created

Complexity in IT grows

Additional IT Resources needed

IT Is Expensive

IT is not a profit center

Market forces demand cut in IT

expenditure

People are the most expensive resource

Cut IT expenses

Use less expensive resources

IT groups take longer to respond

Use specialized teams to gain

economy of scaleOutsource IT groups

Each team is specialized in it own

area

Communication with remote team is more

complicated

Quality of Service is reduced

Delivery time is missed

Service Delivery Managers are

brought on board to coordinate Cust/IT

Proj mgr added for coordination

between groups

Customer satisfaction goes

down while costs go up

The organization is less flexible

IT Dept growth

Pressure to increase efficiency

Matrices created for group performance

Local Optima is encouraged

Coordination between team is

more complicated

IT’s Current Reality Tree

Current State - Impact on Users

Confusion

Delays

Management intervention

Current State - Impact on Users

Users feel that Quality of service goes down due to:Delays.Missed due dates.System complexity.Total System unpredictability

Current State - Impact on IT Groups

Work in FIFO mode (unless work is expedited).

Constant expediting led to conflicting demands.

Conflicting demands led to heavy multi tasking.


Specialized groups caused “silos” and local measurements.

Specialized teams led to complex coordination.

Complex Coordination led to more management oversight.


Longer SLA in order to ensure work is completed in the time promised.

short “touch time” VS.

Long “wait time”.


Complexity leads to increase in

operating costs.

ExampleHow can we break the cycle?

RM Number Opened DateRequested Due Date

Projected End Date Status Brief Description

RM10818090 1/10/11 2/7/11 2/21/11work in progress Add port 443 to efs.disney.com

RM10839292 1/31/11 2/8/11 3/14/11work in progress corp-pc-reverse Proxy request- for Rostr application--RM10863474 2/22/11 3/4/11 4/5/11work in progress Set up external reverse proxy for AprimoRM10823279 1/14/11 1/31/11 1/31/11work in progress New GlobalScape FTP SiteRM10780366 11/23/10 11/30/10 12/7/10work in progress create globalscape ssh private key

RM10867039 2/25/11 3/11/11 3/11/11work in progress New GlobalScape ftp site for sftp with external vendor.

RM10854853 2/14/11 2/15/11 2/18/11work in progressAdd Theresa Gallo GALLT001 to Business Objects Group Members in Identity Manager

RM10856644 2/15/11 2/17/11 2/17/11work in progress DCL West Coast Office needs abiltiy to reset passwordsRM10864590 2/23/11 3/2/11 3/2/11work in progress IDM Access for STORM appRM10867477 2/25/11 3/2/11 3/10/11work in progress Create new OU in EC for MetaData/Storm

RM10828286 1/20/11 2/22/11 2/22/11work in progressRequesting siteminder authentication for Casting Factory website

RM10830017 1/21/11 1/24/11 2/11/11work in progress SETUP SITEMIDER TO DI WEB SERVER

RM10842863 2/3/11 2/11/11 2/24/11work in progressNeed access to database to authenticate EISS users through their HUB ID and PWD

RM10846983 2/7/11 2/11/11 2/28/11work in progressdis/pc/siteminder - requesting integration with playtest metrics

RM10804573 12/22/10 1/5/11 1/19/11work in progress Gate New Siebel Development and Test Servers

RM10814291 1/6/11 1/14/11 1/27/11pendingSiteminder installation for new Windows 2008 server (.net)

RM10819539 1/11/11 1/14/11 2/1/11pendingInstall Siteminder agent for new Cognos 10 Tm1 for deployment on iis webservers.

RM10842520 2/2/11 2/18/11 2/23/11pendingReverse Proxy Siteminder setup request for mobilebi.disney.com

RM10864079 2/23/11 2/25/11 3/16/11work in progressSiteminder QA installation for new Windows 2008 server (.net)

RM10866148 2/24/11 3/2/11 3/17/11work in progress Setup new Site Minder for new application for AprimoRM10866656 2/25/11 3/11/11 3/18/11work in progress DCP - add siteminder to PDL Dev

RM10866934 2/25/11 2/28/11 3/18/11work in progressRequest to change siteminder protected directory to / from /transfer

Req. Date < SLA

Due Date missed

Req. Stuck in Queue

Due Date missed

Req. Date < SLA

Work in Progress even though we have lots of time

Real life problemsMarch 1st



















no correlation between requested date to the projected date

Requested Date before SLA Date

Where do we start?

Define Services and SLA

Service Is a user story that define the action from the point of view of the user and has a definition of done.

Define Services and SLA

• No request will be allowed to enter the queue with requested date less then SLA date.

• Special emergencies will be allowed only by approval from queue manager

• Separate Requests from Incidents

Define Services and SLA – Why?

Predictability Leads to Trust

Users will know exactly what they get and when they are going to get it.



















Many requests still miss their projected end date

Missed Due Date

Establish flow based on Prioritization and coordination

scheme

Is everyone

on the same page ?

What is urgent?

Did we miss anything? Is anythi

ng stuck ?

WHAT IS NEXT?

What do we want?• Self managed teams• Reduced Multitasking• Focus• Increase flow

Rules of the gameSimple and to the point

• Limit the WIP – No work can become work in progress if there are X number of requests in the queue.

• SLA is published• Requests should be processed by

SLA time.

Rules of the gameEstablish Priority System

Due Date is King !Subordinate priority decisions to the Due

Date

Rules of the gameEstablish Priority Visual

System• Requests will be color coded (for

example):–Green – More then 66% of time is

available–Yellow – Between 10% - 66% of time

is available–Red – Less then 10% of time is

available – We are at risk of missing due date

Rules of the gameEstablish Visual Priority

SystemRequests that missed their due

date will be color coded in BLACK– Each Black request will be prioritized

case by case.– Usually Black requests will either be

prioritized or due date will be renegotiated.

RM NumberOpened Date Req. Date

Projected Date SLA 3/1/2011Status Brief Description

Green Level

Yellow Level

Red Level

RM10818090 1/10/11 2/7/11 2/21/11 41 -8.00 work in progress Add port 443 to efs.disney.com 27 14 4

RM10839292 1/31/11 2/8/11 3/14/11 41 13.00work in progresscorp-pc-reverse Proxy request- for Rostr

application-- 27 14 4

RM10863474 2/22/11 3/4/11 4/5/11 41 35.00work in progress Set up external reverse proxy for Aprimo 27 14 4

RM10823279 1/14/11 1/31/11 1/31/11 16 -29.00 work in progress New GlobalScape FTP Site 11 5 2

RM10780366 11/23/10 11/30/10 12/7/10 13 -84.00 work in progress create globalscape ssh private key 9 4 1

RM10867039 2/25/11 3/11/11 3/11/11 13 10.00work in progressNew GlobalScape ftp site for sftp with external vendor. 9 4 1

RM10854853 2/14/11 2/15/11 2/18/11 3 -10.88 work in progressAdd Theresa Gallo GALLT001 to Business Objects Group Members in Identity Manager 2 1 0

RM10856644 2/15/11 2/17/11 2/17/11 1 -12.00 work in progressDCL West Coast Office needs abiltiy to reset passwords 1 0 0

RM10864590 2/23/11 3/2/11 3/2/11 6 1.00work in progress IDM Access for STORM app 4 2 1

RM10867477 2/25/11 3/2/11 3/10/11 12 9.00work in progress Create new OU in EC for MetaData/Storm 8 4 1

RM10828286 1/20/11 2/22/11 2/22/11 32 -7.00 work in progressRequesting siteminder authentication for Casting Factory website 21 11 3

RM10830017 1/21/11 1/24/11 2/11/11 20 -18.00 work in progress SETUP SITEMIDER TO DI WEB SERVER 13 7 2

RM10842863 2/3/11 2/11/11 2/24/11 21 -5.00 work in progressNeed access to database to authenticate EISS users through their HUB ID and PWD 14 7 2

RM10846983 2/7/11 2/11/11 2/28/11 20 -1.00 work in progressdis/pc/siteminder - requesting integration with playtest metrics 13 7 2

RM10804573 12/22/10 1/5/11 1/19/11 27 -41.00 work in progress Gate New Siebel Development and Test Servers 18 9 3

RM10814291 1/6/11 1/14/11 1/27/11 20 -33.00 pendingSiteminder installation for new Windows 2008 server (.net) 14 7 2

RM10819539 1/11/11 1/14/11 2/1/11 20 -28.00 pendingInstall Siteminder agent for new Cognos 10 Tm1 for deployment on iis webservers. 13 7 2

RM10842520 2/2/11 2/18/11 2/23/11 20 -6.00 pendingReverse Proxy Siteminder setup request for mobilebi.disney.com 13 7 2

RM10864079 2/23/11 2/25/11 3/16/11 20 15.00work in progressSiteminder QA installation for new Windows 2008 server (.net) 14 7 2

RM10866148 2/24/11 3/2/11 3/17/11 20 16.00work in progressSetup new Site Minder for new application for Aprimo 13 7 2

RM10866656 2/25/11 3/11/11 3/18/11 20 17.00work in progress DCP - add siteminder to PDL Dev 14 7 2

RM10866934 2/25/11 2/28/11 3/18/11 20 17.00work in progressRequest to change siteminder protected directory to / from /transfer 13 7 2

BLACK Request will be dealt with case by case basis

Pending Request might need to get back into queue

Red Request, Deal with it NOW!

Yellow Req. are next

Plenty of time for Green Request

March 1st 2011

Color Coded Priority System

Due date is King, all priority decisions are

subordinated to the due date!

Visual Management - Cleaning up

• Visual Simplicity is the key for self management.

• You cant improve if you cant see.

• Start by cleaning up!



Green Level

Yellow Level

Red Level

RM10818090 1/10/11 2/7/11 2/21/11 41 -8.00 work in progress Add port 443 to efs.disney.com 27 14 4


application-- 27 14 4RM10863474 2/22/11 3/4/11 4/5/11 41 35.00work in progress Set up external reverse proxy for Aprimo 27 14 4


RM10856644 2/15/11 2/17/11 3/17/11 29 16.00work in progressDCL West Coast Office needs abiltiy to reset passwords 19 10 3

RM10864590 2/23/11 3/2/11 3/2/11 6 1.00work in progress IDM Access for STORM app 4 2 1RM10867477 2/25/11 3/2/11 3/10/11 12 9.00work in progress Create new OU in EC for MetaData/Storm 8 4 1

RM10828286 1/20/11 2/22/11 3/22/11 60 21.00work in progressRequesting siteminder authentication for Casting Factory website 40 20 6

RM10842863 2/3/11 2/11/11 3/24/11 49 23.00work in progressNeed access to database to authenticate EISS users through their HUB ID and PWD 32 16 5

RM10846983 2/7/11 2/11/11 2/28/11 20 -1.00 work in progressdis/pc/siteminder - requesting integration with playtest metrics 13 7 2





First, Clean Up the Queue• Some of the Old work was removed• New Due Date negotiated for some of the Black

requests• 2 Black requests are approved to be priority

Cleaning up the mess makes it easier for the team

member to understand what he/she needs to work

on

Visualize the work , establish flow

Kanban

Visualize the work - Kanban

• Queue manager transfer request from Service Center to Kanban board based on color priority

• Team members pull work based on order

• WIP limit is strictly enforced

Backlog Ready In Progress Customer Acceptance

RM10818090

RM10846983

RM10864590

RM10839292

RM10867039

RM10864079

RM10866148

RM10856644

RM10866148

RM10866656

RM10866934Definitions

WIP Limit 6 WIP Limit 3

All open requests waiting to be processed

Queue Manager to

ensure that no information is missing from

the request before entering

ready state

Item pulled by a team

member and is being

worked on.

Item that was work completed and

moved to customer for approval.

Use Kanban Board for Visual

Self Management


RM10818090

RM10846983

RM10864590

RM10839292

RM10867039

RM10867477

RM10864079

RM10866148

RM10856644

RM10866148

RM10866656

RM10866934

March 1st 2011


Top 3 requests from the ready queue are

pulled in to work in progress

Tech Team Pull work while keeping WIP Limits



Green Level

Yellow Level

Red Level














RM10869023 3/1/11 3/5/11 3/5/11 3 3.00work in progress Urgent class request 2 1 0

The Next Day – March 2nd

• The Red request and 2 black are completed• 3 New requests came in


RM10818090

RM10846983

RM10864590

RM10839292

RM10867039

RM10867477

RM10864079

RM10866148

RM10856644

RM10866148

RM10866656

RM10866934

March 2nd 2011


RM10869023RM10869012RM10869001


RM10818090

RM10846983

RM10864590

RM10839292

RM10867039

RM10867477

RM10864079

RM10866148

RM10856644

RM10866148

RM10866656

RM10866934

March 2nd 2011


RM10869023

RM10869012

RM10869001

As work is being completed, new work can

flow into the queues, based on color priority


RM10818090

RM10846983

RM10864590

RM10839292

RM10867039

RM10867477

RM10864079

RM10866148

RM10856644

RM10866148

RM10866656

RM10866934

March 2nd 2011


RM10869023

RM10869012

RM10869001

March 3rd

• 2 urgent requests came inRM Number

Opened Date Req. Date


Green Level

Yellow Level

Red Level














RM10869023 3/1/11 3/5/11 3/5/11 3 2.00work in progress Urgent class request 2 1 0


RM10869077 3/2/11 3/6/11 3/6/11 3 3.00work in progress Urgent class request 2 1 0RM10869097 3/2/11 3/6/11 3/6/11 3 3.00work in progress Urgent class request 2 1 0


RM10818090

RM10846983

RM10864590

RM10839292

RM10867039

RM10867477

RM10864079

RM10866148

RM10856644

RM10866148

RM10866656

RM10866934

March 3rd 2011


RM10869023

RM10869012RM10869001

RM10869077

RM10869097RM10869101

Color system keeps it simple


RM10818090

RM10846983

RM10864590

RM10839292

RM10867039

RM10867477

RM10864079

RM10866148

RM10856644

RM10866148

RM10866656

RM10866934

March 3rd 2011


RM10869023

RM10869012

RM10869001

RM10869077

RM10869097

RM10869101


RM10818090

RM10846983

RM10864590RM10839292

RM10867039

RM10867477RM10864079

RM10866148

RM10856644

RM10866148

RM10866656

RM10866934

March 3rd 2011


RM10869023

RM10869012

RM10869001

RM10869077

RM10869097

RM10869101

RESULTS

• Standard flow• Self Managed teams• Clear priority system• Reduced time to market• Predictability – Trust – Customer Satisfaction

How do we coordinate several groups?

• Each Service will have tasks for one or multiple groups.

• Each task will have an SLA. • Each group will use the color

coding system to prioritize work towards a common Service due date (see example).


RM10864079-3

RM10864079-2

RM10867039

RM10864079-1

RM10864088

RM10864099

RM108640192


Request RM10864079 has 3 tasks that go to 3 separate

groups .Based on the Service SLA, each group will have a sub

SLA for its task .

If there are dependencies, the Queue Manager will not designate his task as ready

until work on the previous task is in Customer

Acceptance

Use the same principles to coordinate between teams


RM10864079-3

RM10864079-2

RM10867039

RM10864079-1

RM10864088

RM10864099

RM108640192



RM10864079-3

RM10864079-2

RM10867039

RM10864079-1

RM10864088

RM10864099

RM108640192


Color of task might have changed while in the works

RM10864079-1

We gained even more

• Management oversight for coordination was reduced.

• Increase cross team collaboration.

• Increased transparency.• Reduced Complexity and Operating costs.

RESULTS• Standard flow• Self Managed teams• Clear priority system• Reduced time to market• Management oversight for coordination was

reduced.• Increase cross team collaboration.• Reduced Complexity and Operating costs.

PredictabilityTrust

Customer Satisfaction

Questions?

Frequently Asked Questions

• What if the customer does not approve? – Requests not approved by the customer

will go back to the queue with priority Red and flow through the process again.

• Why do we need Ready queue? – In order to ensure that work flows through

the process, we need to ensure that all information is available to the technical staff. The ready queue is for work ready to be processed and all information is available


• How do we ensure that Green tickets will be done on time?– The color of the ticket will change with

time. If we penetrate the buffer and the ticket becomes yellow or red, it will automatically move up on the priority list

• What about super urgent work such as incidents?– Incidents might need to be a separate

class of service. Prioritization of incidents might be based on Severity and not by due date


• How do we find the right SLA?– Too many green – SLA too big– Too many Red – SLA too short

For Additional Information

[email protected]@hotmail.com



Thank you

About the Speaker Tal Aviv/Ovadia

http://il.linkedin.com/pub/tal-ovadia/2/76/586

Blog: Duckmanagement.blogspot.com

http://il.linkedin.com/pub/tal-ovadia/2/76/586

Appendix

Cleaning up

• Phases of LEAN 5S – 1.1 Sorting (Seiri)– 1.2 Straightening or setting in order /

stabilize (Seiton)– 1.3 Sweeping or shining or cleanliness /

systematic cleaning (Seiso)– 1.4 Standardizing (Seiketsu)– 1.5 Sustaining the discipline or self-

discipline (Shitsuke)

it operations in agile israel april 11

Business

new application

new windows

external reverse proxy

userscurrent state impact

new cognos

reverse proxy request

progresscreate new ou

progresssetup new site