Lessons learned while re-platforming a complex system

Mik Quinlan

March 2014

AGILE EXECUTION

Background – The Inventory System

THE CHALLENGE: NEED TO MIGRATE OFF THE CURRENT INVENTORY SYSTEM…

…TO A FUTURE ARCHITECTURE:

The Inventory Management Platform (IMP)

r

CREATE A PLATFORM: IMP VERSION 1.0

Agile Execution

KEY PRINCIPLES…

• Iterative and incremental• Introspect as much as is possible without affecting progress

• How would you approach executing the following types of software projects?

• Green field development• New feature add / maintenance of a mature product• Complex legacy migration

STRUCTURE: PHASED APPROACH• Huge uncertainty initially

• Sprint 0 doesn’t cut it

Borrowed from DSDM –divide the work up into smaller pieces:• Foundations

• Exploration

• Engineering

• Incremental Deployment

STRUCTURE: CEREMONY AND SYSTEM COMPLEXITY

Be comfortable with uncertainty

Source: Information Age

DE-RISKING: GO TO PRODUCTION EARLY…

• Early understanding of:– Application / technology in OUR

organisation• Prove it works throughout all

environments:– Validate features ASAP– Decreases number features in ‘ready for

release but not yet proven’ state• Allows focus on features• Easier if have good automated tests

Source: Growing Object Oriented Software Guided by Tests (Freeman et al)

WALKING SKELETONS

“A Walking Skeleton is the implementation of the thinnest possible slice of real functionality that we can automatically build, deploy and test end-to-end”

- Alistair Cockburn in Crystal Clear: A Human Powered Methodology for Small Teams (2004)

THE KEY IS IN THE DAY TO DAY EXECUTION…(1)• Psychology of an Engineer

• Team setup and iteration structure

• US – EU distributed team

• 1 week iterations

• Small stories (IMP alpha launch: mode size = 1, median = 3)

• Splitting stories – Research vs Implement

• You CAN make the stories smaller!

• Product and Technical decisions:

• ALWAYS know your Minimum Viable Product (MVP) / Minimum Marketable Product (MMP)

• Keep things simple

• Defer all decisions to the last responsible moment

• Take a lean approach wherever possible

• Product and Technical MUST work closely, each with an appreciation and understanding of the other

THE KEY IS IN THE DAY TO DAY EXECUTION…(2)• Attention to our own Productivity:

• Sharing code – re-usable libraries when it makes sense

• Simplifying project setup via archetypes

• Building specialised tools, e.g.:

• Git branch/merge scripts

• Legacy -> new platform migration tools

• Specific category of story

• Roles vs doing:

• Everybody does QA!

• Mid iteration execution:

• What to do if a story is

• not going to be completed?

• is bigger than anticipated?

THE KEY IS IN THE DAY TO DAY EXECUTION…(3)• Fix bugs immediately!

Source: http://pettermahlen.com/2011/04/

THE KEY IS IN THE DAY TO DAY EXECUTION…(4)• Continuous “Inspect and Adapt”:

• Version 2.0 feed spec in current system

• Feed Quality Service component

• Shore up DR solution for publishing as interim primary publishing mechanism

• Support:

• Ownership of code all the way to production

WHAT DOES IT MEAN TO HAVE SMALL STORIES?• Less to carry over

• PO and Lead must THINK about what is being delivered – less nebulous

• Acceptance Criteria MUST be detailed as well as succinct. E.g. 1 point story:

[DP] – Snapshot Service meets organisational standardsAs mycompany.com, I want to be able to deploy, configure and troubleshoot the snapshot service so that the it can integrate into the organisation’s operational environment.

Acceptance criteria• Logging configuration follows the organisation’s standard wrt access/error/console/syslog• Configurable parameters follows the organisation’s standard (properties file outside of WAR)• Packaging follows the organsiation’s standard (deployable WAR, probably nothing to do here)• Build/archive functionality follows company standard (build and archive uses CM scripts on Jenkins)• There is a health-check that checks that the service and its dependencies are healthy. • There is an index page with the following features:

• a link to the health check• a link to the configuration information• a link to the build version information• a form that allows users to submit JSON ingestion requests

• Relevant statistics are exposed via JMX

THE LARGEST STORY ACCEPTED – 5 POINTS[Quality] Reliable VoltDB connectionsAs mycompany.com, I want services accessing Volt to have reliable connections so that they may continue to process on the event of a cluster node or network failure.

Acceptance criteria:

• Services that lose the connection to VoltDB will make reconnection attempts.

• A successful reconnection attempt means that the service resumes functioning correctly.

• Services can start up without a running VoltDB instance.

• Every lost VoltDB connection and each failed reconnection attempt is logged at ERROR level.

• Lack of VoltDB connectivity should lead to failing health checks.

• The reconnecting algorithm should be configurable using CM properties - so, for instance, if the chosen algorithm means that the service will make a certain number of reconnection attempts and then give up, the number of attempts to make and time interval between attempts should be CM parameters.

• The blackbox tests are updated so Volt does not need to be restarted on re-deployment of a component using a Volt client

Notes

It is possible that we will want to contribute this code back to VoltDB, as it may benefit us if the maintenance burden for the code is shifted to VoltDB. So the code should contain as few dependencies as possible.

Affected services: DI, OIDS, RDEP

TECHNICAL EXECUTION• Break the complex problem down incrementally – v1 is:

• Walking skeletons

• Primary Scenario (aka Happy Path)

• Alternative scenarios

• Sharing solutions to problems across components:

• Dependency POM projects

• Restricted technology stack

• Solve once and share

• Having a cohesive view to the solution of a problem:

• Systems Architecture!

• “Considered haste”

• Think more, work less

CONTINUOUS DELIVERY• Story not BA’d til it’s in Production

• Small stories mean:

• small changes to each release

• rollbacks are minor

• Helps determine what we should monitor when we move to automated Continuous Delivery:

• High risk vs low risk points

• Engineers own the application and this process so we better get it right!

CONTINUOUS DELIVERY AND QUALITY• Two paths:

• Low risk features

• High risk features

• An extensive, trusted test suite means:

• Manual verification for high risk stories only

• Send everything else automatically…

WHAT DID NOT WORK…• Prototype stories. Instead:

• Research -> stops design in planning

• Primary Scenario

• Alternative Scenarios

• Planning was too long – sometimes 2+ hours! Instead:

• Pre-planning sessions with Tech Lead, PO, QA rep

• Team divide up stories to task out

• External Support stories as part of the overall commitment and burndown. Instead:

• Have separate [External Support] type stories with 0 points

• Too many engineers on a story. Instead:

• Have story owners

FUTURE GOALS: AGILE FLUENCY

Source: http://martinfowler.com/articles/agileFluency.html

QUESTIONS?

Advanced Agile Techniques

WHEN THE ENGINEERING ORG IS READY…• What are we trying to do?

Go faster in a controlled manner

• With an eye towards…

Value Learning

• Supported by…

Continuous Delivery

• Delivered by multiple teams…

Scaling execution while minimising bottlenecks

and

Supporting each other

VALUE LEARNING• Know what we’re going to measure – indicators of success

• Isolation of variables for a successful test

• Iterate quickly

CONTINUOUS DELIVERY• Is a cultural change

• Do manually first

• Supported by Architecture:

• Minimum: Service based architecture

• Optional: Micro Services

• Supported by execution strategy:

• Everything is easily scriptable

• E.g. run embedded Tomcat / Jetty through command line

FEATURE TOGGLES• Allows:

• Switch on/off of experimental features

• Delivery of unfinished features

• APIs must be backwards compatible for clients

INTERNAL OPEN SOURCE MODEL

• Google model?

• Amazon model?

• What are YOUR pre-requisites?

Engineering Excellence

UNNECESSARY COMPLEXITY

• Akin to Lean, but approached at a different angle

• Five types:

• Product

• Execution

• Architecture / Design

• Technology

• Code

“It’s just not necessary to do extraordinary things to get extraordinary results”

Warren Buffett

THE PRAGMATIC MANIFESTO

Iterative and incremental dark launch over expensive up front performance testing

Features that are immediately required over features that may be required

Just enough design up front over diving straight in

Fact based product enhancement over theory

SOME PRINCIPLES TO CONSIDER…

Lean is better than Perfect

Simple is better than Complex

Think More, Work Less

THANK YOU

p. 34

Blog: www.mikquinlan.com

Follow me: @MikQuinlan

Connect: www.linkedin.com/in/mikquinlan