how build and deploy affects architecture at thetrainline.com - akqa anoraks
DESCRIPTION
By ensuing our software builds and deployments are Reliable, Repeatable, Rapid and Recurring, we can produce software which is lightweight, testable, monitorable, configurable and recoverable. These properties enable features such as load-balancing, scaling, elasticity and rapid recovery, all key for high-performing web systems. R-R-R-R builds and deployments also enable Continuous Delivery of features into Production. Talk given at AKQA Anoraks on 29 November 2012TRANSCRIPT
#akqaanoraks @matthewpskelton
How build and deployment shapes software architecture
at thetrainline.com
Matthew Skelton | thetrainline.com
AKQA Anoraks, London | #akqaanoraks
29 November 2012
#akqaanoraks @matthewpskelton
architecture
= f (build & deploy)
(for some systems)
#akqaanoraks @matthewpskelton
Systems engineering (robotics, control theory, sensors, neuroscience)
Software development
(finance, insurance, travel, pharma, media, medical imaging)
now
Build & Deployment at thetrainline.com
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
9m visits per month to thetrainline.com
2nd most visited UK travel booking website
(behind Easyjet)*
Over 100,000 daily visits via our mobile apps
*source: IMRG hotshops list Nov 2011
#akqaanoraks @matthewpskelton
Booking engines for many UK train operators
Ticketing systems for Corporates and TMCs
Several hundred individual websites
> £1bn ($1.5bn USD) per year in train ticket sales
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
• Early “distributed agile” success story
• http://www.thoughtworks.com/trainline
#akqaanoraks @matthewpskelton
10+ dev teams in 2 locations
#akqaanoraks @matthewpskelton
Server Farm (Build, Deployment, Replication)
100+ build agents 20 dev environments x 20 VMs per
environment = 400 dev VMs Git replication (Gitolite), …
MORE SERVERS THAN PRODUCTION
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
architecture
= f (build & deploy)
(for some systems)
#akqaanoraks @matthewpskelton
“HERESY!”
#webperfdays @matthewpskelton
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
#webperfdays @matthewpskelton
#akqaanoraks @matthewpskelton
#webperfdays @matthewpskelton
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
#webperfdays @matthewpskelton
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
Web-based
Frequently- changing
Public-facing
High-volume
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
RELIABLE
REPEATABLE
RAPID
RECURRING #webperfdays
@matthewpskelton #akqaanoraks
@matthewpskelton
#akqaanoraks @matthewpskelton
‘R-R-R-R’ BUILD AND DEPLOYMENT
Helps to avoid the Ball of Mud
Enables high-performance architectures
#webperfdays @matthewpskelton
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
#webperfdays @matthewpskelton
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
LOAD BALANCING HIGH AVAILABILITY
SCALING ELASTIC
RAPID RECOVERY
#webperfdays @matthewpskelton
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
BUILDABLE
Small pipelined builds on generic build machines
Seconds, not minutes or hours
Short feedback cycles (Dan Worthington-Bodart, @danielbodart - http://bit.ly/M85wsX)
#webperfdays @matthewpskelton
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
Build dependencies may mirror runtime deps
msbuild /m for multi-threaded builds (forces component isolation)
Gallio (MbUnit) – replacement engine for NUnit
NuGet + Squid for super-fast build artefacts
RAM disk for some working copy operations
#akqaanoraks @matthewpskelton
TESTABLE
Test (separation, harnesses, points)
IDENTIFIABLE
Meaningful versions, packages, defined dependencies, artefact
management
(think component boundaries)
#webperfdays @matthewpskelton
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
Enforced “onion” testing layers (test separation, build machine firewalls, build agent types, …)
Semantic Versioning (A.B.C.D + contract to ‘clients’)
30+ external services (test harnesses)
#akqaanoraks @matthewpskelton
DEPLOYABLE Rapid, scriptable, simple failure modes
MONITORABLE Logging, metrics, transaction tracing
CONFIGURABLE
Inject settings – no ‘black boxes’
LIGHTWEIGHT Keep things small and easily comprehendible
#webperfdays @matthewpskelton
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
GO, MSDeploy, hand-rolled, Octopus, PS remoting (“as simple as xcopy”)
Graphite + SqlToGraphite (https://github.com/perryofpeek/SqlToGraphite)
LogStash + log4net (tune logging levels with config, transaction tracing)
SCOM – operations manager (early monitoring in Dev)
#akqaanoraks @matthewpskelton
INSTANTIABLE
No snowflakes or singletons
RECOVERABLE
No nasty zombies after failures
MTTR more important than MTBF* * for most kinds of F
#webperfdays @matthewpskelton
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
Rebuild Build Agents with Chef
Chef for environment rebuilds
Call Start() and Stop() on each component
#akqaanoraks @matthewpskelton
#webperfdays @matthewpskelton
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
Well-known monolithic business rules engine
Costly to build, deploy, configure, test
Being replaced
#akqaanoraks @matthewpskelton
RELIABLE
REPEATABLE
RAPID
RECURRING #webperfdays
@matthewpskelton #akqaanoraks
@matthewpskelton
#akqaanoraks @matthewpskelton
LOAD BALANCING HIGH AVAILABILITY
SCALING ELASTIC
RAPID RECOVERY
#webperfdays @matthewpskelton
LIGHTWEIGHT TESTABLE MONITORABLE CONFIGURABLE RECOVERABLE IDENTIFIABLE
#akqaanoraks @matthewpskelton
#akqaanoraks @matthewpskelton
London Continuous Delivery
Next meetup: 17 January 2013, with
Opscode (Chef)
#londoncd
#akqaanoraks @matthewpskelton
architecture
= f (build & deploy)
(for some systems)
thank you
@AKQAAnoraks
Blog: http://engineering.thetrainline.com/
matthewskelton.net | @matthewpskelton
Thanks to: Attila S, Jack R and Owain P for feedback.
Picture credits:
Label: seanlabel.com; Sheep: gallery.hd.org; Train: worldontrains.blogspot.co.uk; Gurner: mog.com; Petra: Wikimedia/Berthold Werner; army engineers: US DoD; ball of mud: pwern.blogspot.co.uk; sports car: xarj.net; zombie: bjj.org; feather: Wikipedia; punch:
thelegalblitz.com; passport: coverpalace.com; dogs: reluctantmemsahib.wordpress.com; Meccano: dalefield.com