This document discusses how build and deployment practices shape software architecture at TheTrainline.com. It notes that TheTrainline.com handles over 9 million visits per month and processes over £1 billion in annual train ticket sales. It argues that for some systems, particularly web-based systems that are frequently changing, public-facing, and high-volume, architecture is a function of how the system is built and deployed. It outlines TheTrainline.com's build and deployment practices which focus on reliability, repeatability, rapidness and recurrence to help achieve high performance architectures and avoid issues like "balls of mud".

1. How build and deployment
shapes software architecture
at thetrainline.com
Matthew Skelton | thetrainline.com
AKQA Anoraks, London | #akqaanoraks
29 November 2012
#akqaanoraks
@matthewpskelton

2. architecture
= f (build & deploy)
(for some systems)
#akqaanoraks
@matthewpskelton

3. Systems engineering
(robotics, control theory, sensors, neuroscience)
Software
development
(finance, insurance, travel, pharma, media, medical imaging)
now
Build & Deployment
at thetrainline.com
#akqaanoraks
@matthewpskelton

4. #akqaanoraks
@matthewpskelton

5. 9m visits per month to thetrainline.com
2nd most visited UK travel booking website
(behind Easyjet)*
Over 100,000 daily visits via our mobile apps
*source: IMRG hotshops list Nov 2011
#akqaanoraks
@matthewpskelton

6. Booking engines for many UK train operators
Ticketing systems for Corporates and TMCs
Several hundred individual websites
> £1bn ($1.5bn USD) per year in train ticket sales
#akqaanoraks
@matthewpskelton

7. #akqaanoraks
@matthewpskelton

8. • Early “distributed agile” success story
• http://www.thoughtworks.com/trainline
#akqaanoraks
@matthewpskelton

9. 10+ dev teams in 2 locations
#akqaanoraks
@matthewpskelton

10. Server Farm
(Build, Deployment, Replication)
100+ build agents
20 dev environments x 20 VMs per
environment = 400 dev VMs
Git replication (Gitolite), …
MORE SERVERS THAN PRODUCTION
#akqaanoraks
@matthewpskelton

11. #akqaanoraks
@matthewpskelton

12. #akqaanoraks
@matthewpskelton

13. architecture
= f (build & deploy)
(for some systems)
#akqaanoraks
@matthewpskelton

14. “HERESY!”
#webperfdays
#akqaanoraks
@matthewpskelton

15. #webperfdays
#akqaanoraks
@matthewpskelton

16. #webperfdays
#akqaanoraks
@matthewpskelton

17. #webperfdays
#akqaanoraks
@matthewpskelton

18. Web-based
Frequently-
changing
Public-facing
High-volume
#akqaanoraks
@matthewpskelton

19. #akqaanoraks
@matthewpskelton

20. RELIABLE
REPEATABLE
RAPID
RECURRING #webperfdays
#akqaanoraks
@matthewpskelton

21. ‘R-R-R-R’
BUILD AND DEPLOYMENT
Helps to avoid the Ball of Mud
Enables high-performance
architectures
#webperfdays
#akqaanoraks
@matthewpskelton

22. #webperfdays
#akqaanoraks
@matthewpskelton

23. LOAD BALANCING
HIGH AVAILABILITY
SCALING
ELASTIC
RAPID RECOVERY
#webperfdays
#akqaanoraks
@matthewpskelton

24. BUILDABLE
Small pipelined builds on generic build machines
Seconds, not minutes or hours
Short feedback cycles #webperfdays
#akqaanoraks
#akqaanoraks
(Dan Worthington-Bodart, @danielbodart - http://bit.ly/M85wsX) @matthewpskelton
@matthewpskelton

25. Build dependencies may mirror runtime deps
msbuild /m for multi-threaded builds
(forces component isolation)
Gallio (MbUnit) – replacement engine for NUnit
NuGet + Squid for super-fast build artefacts
RAM disk for some working copy operations
#akqaanoraks
@matthewpskelton

26. TESTABLE
Test (separation, harnesses, points)
IDENTIFIABLE
Meaningful versions, packages,
defined dependencies, artefact
management
(think component boundaries)
#webperfdays
#akqaanoraks
@matthewpskelton

27. Enforced “onion” testing layers
(test separation, build machine firewalls, build agent types, …)
Semantic Versioning
(A.B.C.D + contract to ‘clients’)
30+ external services
(test harnesses)
#akqaanoraks
@matthewpskelton

28. DEPLOYABLE
Rapid, scriptable, simple failure modes
MONITORABLE
Logging, metrics, transaction tracing
CONFIGURABLE
Inject settings – no ‘black boxes’
LIGHTWEIGHT
Keep things small and easily comprehendible
#webperfdays
#akqaanoraks
@matthewpskelton

29. GO, MSDeploy, hand-rolled, Octopus, PS remoting
(“as simple as xcopy”)
Graphite + SqlToGraphite
(https://github.com/perryofpeek/SqlToGraphite)
LogStash + log4net
(tune logging levels with config, transaction tracing)
SCOM – operations manager
(early monitoring in Dev) #akqaanoraks
@matthewpskelton

30. INSTANTIABLE
No snowflakes or singletons
RECOVERABLE
No nasty zombies after failures
MTTR more important than MTBF*
* for most kinds of F
#webperfdays
#akqaanoraks
@matthewpskelton

31. Rebuild Build Agents with Chef
Chef for environment rebuilds
Call Start() and Stop() on each
component
#akqaanoraks
@matthewpskelton

32. #webperfdays
#akqaanoraks
#akqaanoraks
@matthewpskelton
@matthewpskelton

33. Well-known monolithic business rules engine
Costly to build, deploy, configure, test
Being replaced
#akqaanoraks
@matthewpskelton

34. RELIABLE
REPEATABLE
RAPID
RECURRING #webperfdays
#akqaanoraks
@matthewpskelton

35. LIGHTWEIGHT
LOAD BALANCING
TESTABLE
HIGH AVAILABILITY
MONITORABLE
SCALING
CONFIGURABLE
ELASTIC
RECOVERABLE
RAPID RECOVERY
IDENTIFIABLE
#webperfdays
#akqaanoraks
@matthewpskelton

36. London Continuous Delivery
Next meetup:
17 January 2013, with
Opscode (Chef)
#londoncd
#akqaanoraks
@matthewpskelton

37. architecture
= f (build & deploy)
(for some systems)
thank you
@AKQAAnoraks
Blog: http://engineering.thetrainline.com/
matthewskelton.net | @matthewpskelton
Thanks to: Attila S, Jack R and Owain P for feedback.
Picture credits:
Label: seanlabel.com; Sheep: gallery.hd.org; Train: worldontrains.blogspot.co.uk; Gurner: mog.com; Petra: Wikimedia/Berthold Werner; army
engineers: US DoD; ball of mud: pwern.blogspot.co.uk; sports car: xarj.net; zombie: bjj.org; feather: Wikipedia; punch:
thelegalblitz.com; passport: coverpalace.com; dogs: reluctantmemsahib.wordpress.com; Meccano: dalefield.com #akqaanoraks
@matthewpskelton