swimming against the waterfall@grnet

12 Nov 2014

Swimming against the waterfall @GRNET

Running Scrum in a conservative, multi-constrained setting Challenges & Risks from the PO perspective

Byron Georgantopoulos, GRNET, e-Infrastructures byron@grnet.gr, linkedin.com/in/ibyron, @digibyron

11th Agile Meetup, Athens

12 Nov 2014

Outline• Introduction • Constraints at the starting line • Embed Scrum into RFP and subsequent contract

• Running Scrum project • Evaluation • Challenges and Risks

2

12 Nov 2014

The CompanyGRNET SA (Greek Research and

Technology Network)

Founded in 1998

Provision of network and computing services for Greek Academic & Research Community National and European R&D Projects GRNET Network Management

3

~okeanos Public IaaS cloud

Started in 2011 Virtualized Compute, Network, Storage resources

Build on top of proven OSS (e.g. Google Ganeti) OpenStack API-compatible

currently >7000 active VMs okeanos.grnet.gr

12 Nov 2014

The "eScience" Project:

IaaS => PaaS => AaaS The first project in public sector to be implemented using Scrum

Main pillars:

(1) Hadoop operations over ~okeanos cloud (2) Virtual Research Environment (3) Reproducible Research

cloud-enabled data-intensive science for A&R community

4

12 Nov 2014

Why are we here today?Discuss Scrum application and lessons learnt, under the following constraints:

• The contracting authority is a public body • The contractor is a private sector supplier • The original RFP and the ΕΣΠΑ framework are waterfall-oriented • It is the first Scrum experience for everyone involved (almost)

5

12 Nov 2014

The Fixed Data

6

12 Nov 2014

Why Scrum?• Avoid common public project pitfalls:

• Transparency and control - ensure early & sustained visibility • Business risk minimization - able to modify the PB

• Test the Scrum waters

How:

• Short iterations

• Potentially shippable increment at the end of every sprint

• Prioritized PB items

• Avoid upfront design, flexibiity to change scope

• Frequent interaction and feedback7

12 Nov 2014

RFP preparation

Very detailed initial specifications (“The system shall…”) Flat structure of specifications - not hierarchally organized

Scrum explicitly stated as the implementation framework Scrum as a factor for scoring candidates (10%)

8

12 Nov 2014

Contract• Scrum explicitly stated on contract

• Mid-to-Large duration (14m), although reduced from original

• Reporting on a monthly basis • Payments based on reports (checkpoints) & features tested • Fully estimated Product Backlog plus Definition-of-Done

defined at the end of Pre-Game sprint • Not finished SBIs re-inserted into next sprint • Grooming to update and refine Product Backlog

9

12 Nov 2014

Initial Challenges

• Winning proposal close to original RFP

• The team has unknown technological and agile skills

• Expected defensive attitude towards the unknown new framework

• Open source everything enforces full transparency, may conflict with isolated dev environments

regarding the Team

10

12 Nov 2014

The Dev Team• Private sector and a university research lab

• Partially co-located

• Limited familiarity with technology, agile, co-development

• 3 persons at the beginning, additional developers in following sprints

11

12 Nov 2014

Scrum Master• Also the contractor’s PM

• Previously participated in Scrum-flavored projects

• Unclear boundaries when wearing both (PM & SM) hats

12

12 Nov 2014

Product Owner• Learned his lessons from past waterfall projects • Received feedback from the company's partnership in

other Scrum projects • Surprised when realised level of engagement (how close &

how often needed to work with the team)

13

12 Nov 2014

Scrum adoption challenges• Under-estimation • Expect the managers to give orders • 99% "Done" (activity vs. result-based) • Silos of code • Back-door waterfall attempts (e.g. request fully-fledged

design)

14

12 Nov 2014

Glad :)• Homogeneous team that 'gels' and works well together • Minimal interpersonal issues • Close collaboration with PO (full collab 1d/week, 40-50%

time devoted to the project so far) • All ceremonies conducted and timeboxed • Acceptance of Scrum, resistance less than expected

15

12 Nov 2014

Glad wrt. “Why Scrum” choices

What the Project has gained from Scrum: • Demonstratable software • Deployable software • External stakeholders involvement • Continuous feedback • Avoid wasted work and upfront design • Scope changes allowed

16

12 Nov 2014

Sad :(• Long-lasting tasks (frequently exceeding 2 days) • 1st story always finishes late >> not smooth burndown • Unfinished sprint stories >> unpredictable velocity • Building technology skills vs. business value: 1-0 • “Working software” mentality lagging • Increasing technical debt (coding standards, test coverage) • Context switch & not full-time dedication

17

12 Nov 2014

Mad ~:(• Delays and impediments surface towards the end of sprint,

ignoring the 'elephant in the room' • ‘Hero’ attitude: work overtime and finish everything at the end • Status meetings masked into daily scrums • Definition-of-Done not followed • More transparency needed (frequent commits, meaningful

comments)

18

12 Nov 2014

Sprint retrospectives• Have emerged as a major tool for inspecting & adapting

• Gradually people express their views more openly

• So far focused on processes and tools (not people or relationships)

• Scrum Master and Product Owner still in 'protected' zone

• Need fresh ideas on how are executed and how to fully engage all team members

19

12 Nov 2014

Corrective measures taken• Scrum training >> better understanding of process and estimation • PO close to the team >> direct feedback, tech guidance, team spirit • Split stories >> better estimates • Retrospective outcomes embedded into next sprint • Improvise process when needed, outside the Scrum textbook: (e.g.

live daily scrum at the end of collab day instead of teleconf) • Pull back, empower dev members >> transfer decision-making to

the team in order to promote self-management

20

12 Nov 2014

Risks and hard questions• Burn-out:

Estimation & ceremonies Continuous effort, without breaks

• Self-organizing, managing and owning • What if the team will react to failures by overestimating

required effort (generally: abusing the rules) • Engagement level of PO may discourage Scrum adoption • Will it finish without compromises?

21

12 Nov 2014

Towards Scrum-friendlier RFPs

• Stricter requirements for Scrum team composition: • Scrum experience / certification • Seniority • Sustained effort

• Higher-level, more business-value oriented specifications • More types of delivery checkpoints

22

12 Nov 2014

The Scrum Coach• Agile Meetup community >> knowledge exchange forum • Met coach within Meetups • A great boost for PO and team: Scrum experience and

guidance, 'external observer', servant leadership • A public ‘thank you’

23

12 Nov 2014

Conclusion: Scrum increases business value in public sector projects, certainly worthy to promote and expand

Scrum will pass the Turing test: • If the Dev team / SM choose to run their next project using Scrum • If critical Scrum adoption know-how is created and transferred to new

teams inside GRNET and the broader public sector

The Turing Test

24

12 Nov 2014

Thank You

25