1. Living System
or
Build Factory?
!
Chris Maxwell
!
@WrathOfChris
Cloud guy @ Ubiquiti Networks

2. Who is this talk for?
• We build systems
• We turn design into implementation
• We operate systems forever**not forever

3. We keep systems from failing

4. Holding up the system?

5. We do it wrong

6. Not helping

7. What is a Living System?

8. The business has a need

9. Instructions are unclear

10. What about THIS space?

11. Instructions are not followed

12. ...except a VCR, CD’s, and a remote?

13. Production loads change
unexpectedly

14. Additional capacity added

15. Wrong tools at hand

16. 12ga crimp on 24ga wire doesn’t work

17. Patches are put into production

18. Patches become production

19. Production is unexpected

20. Unexpected Scotsman

21. Parts not available

22. No ventilation ducting available

23. Inconsistent Origins

24. One of these
things
is not like the
other
http://www.flickr.com/photos/msittig/4801606785/

25. Living System
• Has a long life
• Mutates over time
• Each system slightly
different
• The system is the
state
http://www.flickr.com/photos/anijdam/2362660336/

26. Problems
!
Can you rebuild it
… fast?
… accurately?
http://www.flickr.com/photos/adactio/2856955322/

27. Problems
!
Do you keep:
• HA failover?
• Standby host?

28. Problems
!
Have you tested?
!
When both fail?

29. What is a Build Factory?

30. For business…:

31. http://www.flickr.com/photos/pasukaru76/6893926948/
Run the line

32. For devs…:

33. make clean && make

34. For ops…:

35. http://www.flickr.com/photos/comedynose/3549464622/
Kill it with fire

36. Looking back on Build Factory

37. Copying systems

39. Cloning systems

40. Early build factories took time

41. ...hours on 10baseT

42. ...tumbleweeds...

43. It was a manual process

44. How to restore a VM image...
...in 7 easy pages of screenshots
Sorry, couldn’t release these pics :(
It was just lots of screenshots
+ do this, then that instructions

45. It was a manual process

46. http://spalinux.com/wp-content/uploads/2008/10/
windows_join_domain.png
...and still manual work after

47. Scripting is a start

48. I will replace you with a short script

49. Makefiles track dependencies

50. or a Makefile
it builds your kernel, and probably your OS

51. Today’s tools

52. Not a complete list

53. Describe

54. http://www.flickr.com/photos/spree2010/4960430323/
Write down all of the steps

55. Automate

56. http://www.flickr.com/photos/sepblog/3679949333/
“too complex for any one man’s memory”

57. Replicate

58. Build 3 the same, tie them together

59. Repeat

60. Training makes elements replaceable

61. Build Factory
• Has a short life
• Replaced for each
change
• Each system the same
• The procedure is the
state
http://upload.wikimedia.org/wikipedia/commons/5/50/Krispy_Kreme_Doughnuts.jpg

62. When something jams
!
deep in the system

63. a scrap of paper caught in the drive gear

64. Replace It!
•fix it later
•send it for
repair
•get production
working!

65. When production patches take over

66. Production maintenance changes the system

67. Rebuild it!

68. Sometimes you get to rebuild them

69. In the
cloud?
!
live replace might
cost up to 1 hour
of billing

70. In the
datacentre?
!
live replace can
be done with
spare capacity

71. Takeaway
• Describe it
• Automate it
• Produce it
• Repeat it

72. Questions?
!
@WrathOfChris
!
chris@wrathofchris.com
!
Like the Loch Ness Monster,
Answers are over there...

73. • Forgotten attributions:
•
ghost-1: http://www.all-internet-security.com/images/pc-copy.jpg
•
ghost-2: http://www2.expta.com/uploaded_images/Ghost-741732.png
•
ghost-3: http://internship.wesmilliron.com/wp-content/uploads/2012/08/vmware-norton-ghost.jpg
•
laplink: http://thumbs1.ebaystatic.com/d/l225/m/mkhsfLBMjkjjMw9skfQUNag.jpg
•
lap link: http://www.old-games.com/screenshot/5736-1-lap-link-3-for-dos.jpg