minnebar 2013 agile methods

Bryan J Welch, Minnebar 2013 1

Agile methods for infrastructure groups

• How well do Agile methods work for a group of system administrators and DBAs?

• One company's story• What worked?• What didn't?


Agile methods for infrastructure groups

Agenda● Backstory● Agile for developers● Growing pains● Agile infrastructure● Tweaking the process● What worked and what didn't


Backstory

Web based product● Custom server, code and database per client● Home grown ticket system● Paper change management (sometimes)


Backstory

Silos and infrastructure issues● Infrastructure and developers very separate● Little interaction or communication● Requests only through managers● Reactionary resource planning● New client install was difficult● DBAs in the middle


Proof of concept in production

V5 to the rescue● Enter the mad Russian

– Mathematician consultant

– Friend of friend of senior manager

● Created proof of concept ● Immediately placed in production

– Little input from Inf group– No developer knowledge of v5


Proof of concept in production

Stability issues for a year● Hard to diagnose issues● Code, app server, or database?● Chaos ensued...


A purge and a new CIO

A purge● New CIO, layoffs, and many new developers● Agile, pairing, CI, and TDD● Scrum teams, sprints, and standups● Infrastructure left completely out


Agile for developers

Infrastructure watches from outside● Daily standup of scrum teams● Sprint planning ● Sprint review ● Backlog meetings


Agile for developers

The first two months● Initial confusion ● That's a lot of meetings● Those stories look like projects● What's a stakeholder?

Infrastructure: We'll just keep doing our work...


Growing pains

Six months update● Infrastructure still observing● One big standup and sprint review for all● Infrastructure, sales, and product managers

attend● Stakeholders attend to clarify stories● Scrum task boards with stories


Growing pains

DBA perspective● Issue tickets: 40% to 70% of time● DBA spikes impact dev stories● Solution: DBAs don't 'own' stories

– Still blamed for dependency delays


Growing pains

Same issues for infrastructure● Not involved enough in dev projects● Not aware of sales promises ● Not prepared for large new clients● Priorities for other groups unclear● DBAs still in the middle● Stress and blame at all time highs


Agile infrastructure

Goals for adopting Agile● Improve communication● Improve visibility of infrastructure projects● Understand other group's projects● Stakeholders submit stories for requests● Priorities negotiated between stakeholders● No paper; Jira tickets instead



Workflow changes● Added Jira ticket type: story● All requests must be stories

– Resistance at first

– Issue tickets versus story tickets

● Pairing system admins and/or DBAs– Justification to management needed



Meetings● One scrum team daily standup● Infrastructure manager is scrum master● One week sprints● Sprint planning and review weekly ● Attend all development Agile meetings

– One DBA and system admin for each



Agile resource allocation● System admin ticket box rotation● Week 1: work on issue tickets (no stories)

– Only help with stories if ticket box was empty

● Week 2 to 4: story work● Pair with developers as needed



Three week DBA rotation ● Week 1 Pair with developers● Week 2 Ticket box● Week 3 Pair with system admins



Triage rules needed● Issue requests interrupting story work

– “Can you help me...”

● All issue tickets go to ticket box first● Ticket box hat● Issues assigned to others as needed



Jira tickets instead of story cards● Already used Jira issue tickets● Web access of story status by stakeholders● Clear ownership● Three remote admins, worked well● Change management easier and more clear



Story ticket flow ● Submitted● Suspended*● Accepted and pointed or rejected● Pending● In progress● Completed● Work approved



Sprint planning meetings● Story acceptance was hard

– Many dependencies

– Unclear requests

– Too big

– Conflicting stories

● Voting story point values– Points as approximate days of labor

– 2, 3, 5, 8, 13 (unknown)



Story handling● May split story into smaller ones● May convert story to a project● Review in-progress and pending story tickets● Choose which to pick up● Must have pair to pick up any story● Don't pick up too many stories



Story Issues● Too many story requests● No Prioritization, FIFO only● Which group gets priority?● Stories missing info (suspended)● Spike story requests● Why wasn't my story picked up?


Tweaking the process

Story handling updates needed● Reasonable points per sprint● Prioritized FIFO for stories

– Same as issue tickets

– No SLA for stories

● Let groups negotiate priorities



Spikes● “This must be done now!”● To pick something up during sprint, something

else must be dropped● Solution: let stakeholders negotiate spikes● Stakeholders pick story to drop● Moved emphasis to story priority



Ticket box rotation● At times, incident tickets blocked stories● Two system admins sometimes needed● Ticket backlog very visible



Dealing with very large stories● Some stories could span several sprints● They need to be tracked, but aren't really

stories



Epic tickets created● Jira ticket with project category● Child story tickets noted in epic● Not pointed or counted for velocity● Infrastructure vote needed to pick up epic● Owned by one system admin or DBA



Six month updates● Sprint planning meeting every Tuesday● Project review / planning meeting every

Thursday● No standups● No more sprints● No more points


What didn't work?

● Sprints– Spikes and maintenance window conflicts

– Stories that were much larger than estimated

– Incidents with failed storage, rollback of release patches, etc.

● Standups– With pairing, meetings, IM, e-mail, and tickets, not

needed


What didn't work?

Points and velocity measurement● Very useful at first● Dependencies outside group slowed story

progress● Points difficult for stories from outside group● Stories picked up mostly based on priority

– Every story is a P1

● Spikes


What worked well?

Stories for work requests● Better control of work load in infrastructure● Story priorities determined by stakeholders● Spike priority negotiation done outside

infrastructure● Emphasis on story, not stakeholder● Less stress and blame


What worked well?

Epic tickets● Better visibility of infrastructure projects● Easier tracking of ownership and parallel work● Better historical tracking of changes


What worked well?

Attending sprint meetings of other groups● Better collaboration between groups● Kept infrastructure aware of company projects

and directions


What worked well?

Pairing within infrastructure group– Cross training

– Extra eyes for critical work

– Vacation became somewhat possible

– Productivity was better

– Camaraderie

minnebar 2013 agile methods

Data & Analytics

bryan j welch

agile infrastructure

pains agile infrastructure

agile infrastructure

agile methods

standups infrastructure

new developers agile

story dont