take the red pill: how criteo revamped its software development process
DESCRIPTION
These are the slides of the talk I gave at the Global Scrum Gathering Berlin in 2014. It describes how Criteo managed to change the software engineering culture of its R&D department from 'Testing is cheating' to 22K+ tests being run every half hour.TRANSCRIPT
TA K E T H E R E D P I L LD R . A D R I A N P E R R E A U D E P I N N I N C K
T R E M E N D O U S G R O W T H . C R I T E O : G O G O G O ! ! !
• From 20 to 160 engineers in less than 5 years
• 160 code repositories
• 7 million lines of code
• Took months to release
W E M U S T S E E R E A L I T Y T O P U T T H E M A C H I N E S U N D E R C O N T R O L
W H AT W A S R E A L I T Y ?
• Average lead time for new features = 60 days
• Average time to merge commits = 20 days
• The release pipeline was blocked for months
• It was getting worse each year
S O M E T H I N G H A D T O C H A N G EI T W A S G O I N G T O TA K E T O P M A N A G E M E N T T O A G R E E O N S O M E T H I N G L I K E T H I S
H O W W O U L D Y O U S E L L T H I S I N I T I AT I V E ?
PA I R D I S C U S S I O N :
O B J E C T I V E S
• Reduce commit merge time to hours instead of days
• Reduce time from commit to Prod to < 4 days
• Reduce number of cancelled release candidates
T E C H N I C A L S O L U T I O N
• Tailor-made Build System
• Mandatory Code Reviews
• Continuous Integration Pipeline
• Sandbox to execute tests
P R O C E S S F O L L O W E D
E S TA B L I S H S E N S E O F U R G E N C Y
• Code freeze in 2012 crippled the team during months
• The release pipeline had been blocked for months
• Things weren’t getting better for 2013’s code freeze
E N E R G Y W I L L D I S S I PAT EW I T H O U T A S T R O N G G U I D I N G C O A L I T I O N
S T R O N G G U I D I N G C O A L I T I O N
• Executive Vice President of Engineering
• Vice President of QA
• 10 Software Engineers, high level of seniority
• 2 Senior Engineering Program Managers
D E V E L O P I N G A C H A N G E V I S I O N
• Brainstormed for the key elements
• Prepared a document explaining the vision
• Shared the document with Key players
• Modified the document to take into account feedback
• Shared the document with Dev Leads
• Modified the document to take into account feedback
• Gave presentation explaining the project
• Sent document to all
C H A N G E V I S I O N
• 26 page document
• Took over a month to get alignment in version 0.1
• Sent to 20 senior engineers for feedback
• Last version took into account all feedback
C O M M U N I C AT E T H E V I S I O N
• Monthly updates at all hands meetings
• Weekly emails with progress reports
• Events with food
E M P O W E R B R O A D - B A S E D A C T I O N
E M P O W E R A C T I O N
• The improvement backlog was shared with everyone
• Some items prioritised by voting
• Voluntary-based work encouraged
• Test-a-thons and other wide range initiatives
S H O R T- T E R M W I N S
• Used Scrum to guide the change process
• Developed usable tools from the beginning
• Developed one tool at a time
• Pushed adoption slowly until tipping point
• Forced rest to adopt once tipping point passed
I N T E G R AT I O N T I M E S
0.00%$
10.00%$
20.00%$
30.00%$
40.00%$
50.00%$
60.00%$
70.00%$
80.00%$
90.00%$
100.00%$
12/11/13$
12/18/13$
12/25/13$
1/1/14$
1/8/14$
1/15/14$
1/22/14$
1/29/14$
2/5/14$
2/12/14$
2/19/14$
2/26/14$
3/5/14$
3/12/14$
3/19/14$
T E S T P L A N
P R O J E C T B U R N D O W N
0"
10"
20"
30"
40"
50"
60"
70"
80"
90"
100"
5/13/13" 6/13/13" 7/13/13" 8/13/13" 9/13/13" 10/13/13" 11/13/13" 12/13/13" 1/13/14" 2/13/14" 3/13/14" 4/13/14" 5/13/14"
Commit&to&Prod&Lag&C O M M I T A G EF R O M 3 5 D A Y S T O 1 5 D A Y S
D O N ’ T L E T G OI T ’ S A LW A Y S H A R D E R T H A N Y O U T H O U G H T
L A R G E S T I M P E D I M E N T S
• Technical
• Eat your own dog food
• Replicating the Sandbox
• Human
• People added to the team not sharing vision
• Getting teams to use the tools
• Managing expectations and communication
D O N ’ T L E T G O
• Fully dedicated team for a whole year.
• After the project ended the team remained as a new unit.
• Even through the hardest times management gave full support.
I N C O R P O R AT E C H A N G E I N T O T H E C U LT U R E
I N C O R P O R AT E C H A N G E I N T O T H E C U LT U R E
• Training program for new recruits
• Engineering partners are expected to conform to new way of working
• Training program for engineering partners
• Tools in place become the cultural gates
T H E R E I S M O R E T O C H A N G E T H A N K O T T E R
O T H E R T E C H N I Q U E S W E U S E D
B R I D G E B U I L D E RW E A S K E D T H O S E A L R E A D Y O N B O A R D T O H E L P O T H E R S D O T H E S A M E
C H A M P I O N S K E P T I C
Find someone senior in the organisation:
• That wants the goal to be reached
• That doesn’t fully support your technical solution
• Talk with him regularly
D E D I C AT E D C H A M P I O N
Y O U W O N ’ T G O FA R W I T H O U T O N E
D O F O O DM A K E S U R E T O B R I N G S N A C K S T O G E T T O G E T H E R S
E A R LY A D O P T E R SW I L L I N G T O T R Y T H I N G S B E F O R E T H E Y A R E R E A D Y
E X T E R N A L VA L I D AT I O N
L I N K E D I N W E N T T H R O U G H T H E S A M E P R O C E S S
G R O U P I D E N T I T Y
M A K E P E O P L E P R O U D O F B E I N G PA R T O F T H E C H A N G E
I N Y O U R S PA C EW E U S E D L A R G E S C R E E N S T O S H O W P R O G R E S S
I N V O LV E E V E R Y O N EI F Y O U W A N T L A S T I N G C H A N G E …
U S I N G S C R U MA N D W E D I D I T A L L
- A G E N T S M I T H
“Never send a human to do a machine’s job”
R E F E R E N C E S
• Leading Change - John P. Kotter
• Fearless Change - Linda Rising, PhD
• Continous Delivery - Jez Humble, David Farley