"DevOps @ Wotif: Making Easy = Right" Videos: https://vimeo.com/102211796 / https://www.youtube.com/watch?v=yPy44B9h820 Related IEEE Article: http://online.qmags.com/ISW0516#pg56&mode2 Abstract: We know the “right thing” to do is to work together, to remove bottlenecks, to automate, automate, automate. But when the status quo is finger-pointing, mistrust, blame, and over-exhaustion from increasingly complex manual releases, it seems impossible to find a way to climb out of the downward spiral. How did an IT department turn this around to become a harmonious department with common goals and over 10x faster cycle times? Winning the hearts and minds needed for lasting DevOps change requires something more than just great automation. This is the story of how local Brisbane success story, Wotif Group, found a way to incentivise a DevOps transformation across an entire IT department, resulting in cycle times measured in hours instead of weeks/months. While this journey involved tools such as Puppet, Hiera, Fabric & ZooKeeper, Matt (dev) and Alexandra (ops) extract principles that they hope can be applied to any organisation at grassroots and leadership levels using existing toolchains to support not only the best ideas of the present but also the legacy of the past and the unknown innovation of the future. Speakers: Alexandra Spillane (@ajbw) is a systems administrator with more than ten years experience administering Linux and Unix systems. She has been a member of Wotif Group's Continuous Delivery team since its inception in 2013. She loves using tools like Puppet to solve thorny problems in creative ways and will be speaking at PuppetConf 2014. Matt Callanan (@mcallana) is a senior software developer in the Continuous Delivery team at Wotif Group. He has over 14 years development experience including DevOps implementation and agile mentoring in small teams and large enterprises. He is most passionate about helping companies realise their potential through automation and reducing feedback cycles to quickly deliver value to customers. Matt has spoken internationally on DevOps, is a well-received presenter to IT departments and executive teams, and co-organises the DevOps Brisbane Meetup. http://devopsdays.org/events/2014-brisbane/proposals/devops_at_wotif_making_easy_equals_right/

1. DevOps @ Wotif
Making Easy = Right
Alexandra Spillane • @ajbw
Matt Callanan • @mcallana
Right
Easy

2. Outline
• What is “The Right Thing”?
• How do we Make it Easy?
• Lessons Learned
Making Easy = Right 3

3. What is “The Right Thing”?
DevOps @ Wotif: Making Easy = Right
Making Easy = Right 4

4. “The Right Thing”
Making Easy = Right 5
Collaborate!
Automate!

5. But… The Dev<->Ops Chasm
Making Easy = Right 6

6. What Were we Doing Wrong?
Making Easy = Right 7

7. Downward Spiral of Manual Activity
Making Easy = Right 8

8. Entrenched Silos
Making Easy = Right 9

9. High Release Overheads
Making Easy = Right 10

10. Huge Batch Sizes
Making Easy = Right 11

11. Poor Visibility
Making Easy = Right 12

12. Calendar of Doom
Making Easy = Right 13

13. Unpredictable Prioritisation
Making Easy = Right 14

14. Superstitious Gatekeeping
Making Easy = Right 15
just in case

15. Stagnating Applications
Making Easy = Right 16

16. Manual Deployment Anti-Patterns
Making Easy = Right 17
Extensive, detailed release documentation
Reliance on manual testing to confirm app is correct
Explaining why deployment is going wrong on release day
Frequent corrections to release process during release
Environments that differ in their configuration
Releases that take more than a few minutes to perform
Releases that are unpredictable in their outcome

17. Cycle Time
measured in
weeks/months
Application Pipelines Ops
Days
Weeks!
Days
Staging
Load
Test
Production
Days
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA !?
PrioritisedReleaseQueue

18. Let’s Migrate to Microservices!
• Explosion of manual effort
Making Easy = Right 19
A
B
C
D
E F
A
B1
C1 G
C2 H I
B2
D1 J
D2 K
L1 M
L2 N
E
F1 O P
F2 Q R
S
T

19. HeavyWeight to LightWeight
Glassfish
• Feature overhead
• Encouraged manual config
• Complex/Slowdeployment
DropWizard
• Trimmed down to basics
• Standard config files
• Simple/Fast deployment
Making Easy = Right 20

20. But… Combinatorial Explosion
Making Easy = Right 21

21. Unpredictability Slows You Down
Making Easy = Right 22

22. Moving from wrong to right
Making Easy = Right 23
Wrong
Right

23. Reduce Cycle Time
Making Easy = Right

24. Flexibility vs Predictability
• Freedom is Great for Innovation
• Consistency is Essential For Speed
• How do we promote a culture of freedom
and responsibility but still provide
predictability?
Flexible Predictable
Making Easy = Right 25

25. Standardisation
Making Easy = Right 26
Standardised
Freedom
• Standardise on how services are
deployed and how they
communicate.
• Be flexible about their contents.

26. Defining Standards
• Discussions
– One-on-one
– IM
– Emails
– Meetings
• Incentivise input
• Confluence
– Application Deployment
Standards 1.0
Making Easy = Right 27

27. Example Standards
Making Easy = Right 28
Logging
• Log file locations,
filenames
Log rotate Directories/Files
• Ownership,
permissions
Puppet
Supervisord config.yaml
• Filename, location
Metrics JVM settings
Network
• Ports
• Firewall rules
Cron Endpoints
• Healthchecks, status,
metrics
SSL
• Certs, keystores
RPM
• Versioning
• Packaging
init.d scripts
• Sub-commands
Versioning Migration Notes

28. Example Standards
Making Easy = Right 29

29. Future Considerations
Making Easy = Right 30

30. Standards Lifecycle
Making Easy = Right 31

31. Common Ground
Making Easy = Right 32

32. How do we Make it Easy?
DevOps @ Wotif: Making Easy = Right
Making Easy = Right 33

33. Automated Verification
Making Easy = Right
Compliance Test Suite
Bring Operations into Development
Fast Operational Feedback
Test Driven Operational Compatibility
Backwards Compatibility

34. Ops Testing with Fabric
Making Easy = Right 36
•Does rpm have correct metadata linking to git repo? Check existence/absence of filesrpm
•Is correct version actually installed?yum
•Check existence/absence of files. Check file permissions/ownershipls
•Is correct version actually running?lsof
•Check correct ports exposednetstat
•Is correct log format in use?tail
•Check cron configgrep
•Check contents of standalone jar file – e.g. metadata, library versionsunzip
•Check standard endpoints for e.g. content, status code, response timecurl
•Check JVM optionsjps
•Check 1 and only 1 process runningps

35. Compliance Test Example
Making Easy = Right 37

36. Reference Implementation
• “helloworld-service”
• Deployed to production
Making Easy = Right 38

37. Rolling Upgrade
• Safely orchestrate cluster upgrades
– automated deployment & testing
• Initially interactive
Making Easy = Right 39
• Check Load Balancer
Exit Pool
• Orchestrate Puppet
Upgrade
• Compliance Test
• Smoke Test
• Feature Test
Test
• Check Load Balancer
Enter Pool
• exitpool
• stop
• yum remove
• yum install
• Wait for manual
testing!
• enterpool
Manual Release

38. SLIPway
Making Easy = Right 40
• Big review of old release
processes
– Focus on reducing cycle
time
• Simple Lightweight
Independent Path Way
• Simple rules for release
process
• Keep changes
independent

39. Simple Rules for SLIPWay
• Independent
• One application per release
• Backwards-compatible
• No DB/Network/OS changes
• No manual testing
• Cannot book specific time
• Feature switch for time
• Ops will service queue within 24 hours
• Compliance with latest standards

40. Simple Rules for SLIPWay
42Making Easy = Right

41. Simple Rules for SLIPWay
43Making Easy = Right

42. Application Pipelines Ops
Days
Weeks!
Days
Staging
Load
Test
Production
Days
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA !?
PrioritisedReleaseQueue

43. 45
Application Pipelines Ops
PrioritisedReleaseQueue
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Dev QA
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Staging Load Test Production
Hours
Hours
SimpleRules

44. Making Easy = Right 46

45. Making Easy = Right 47

46. SLIPway Chat Room
Making Easy = Right 48

47. SLIPway Example Release
Making Easy = Right 49

48. Freeing up 3 full time employees
0
5
10
15
20
25
Dec Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov
Man-hoursperrelease*
man-hourstoshiparelease
95%
reduction in
man-hours
GRIPway
SLIPway
SLIPway
introduction
-18.5hrs
Hours Spent Shipping Release

49. 0
5
10
15
20
Cycletimeinbusinessdays
businessdays-8
-2
86%cycle time reduction
SLIPway
GRIPway
average 2013
GRIPway today
Jan JanJulApr Oct Apr
20142013
Jul Oct
Time-to-market Reduction
Making Easy = Right
2014

50. 0
20
40
60
80
100
120
Numberofreleasespermonth
Throughputx2.9
190%
releases
GRIPway average 2013
SLIPway
introduction
GRIPwaySLIPway
SLIPway average
GRIPway
Jan JanJulApr Oct Apr
20142013
Jul
2.9x Releases Per Month
Making Easy = Right

51. Release Cycle Time Comparison
Making Easy = Right
7x lower Avg. Cycle Time
0
1
2
3
4
5
6
7
8
9
10
GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days

52. Commit
stage
Acceptance
stage
Capacity
Testing
Production
Continuous
Delivery
Continuous
Deployment
Automatic
Trigger Manual
Trigger
Commit
stage
Acceptance
stage
Capacity
Testing
Production

53. Release Cycle Time Comparison
Making Easy = Right
7x lower Avg. Cycle Time
30x lower vs SLIPway
200x lower vs GRIPway
21mins
0
1
2
3
4
5
6
7
8
9
10
GRIPway Avg Cycle Time Days SLIPway Avg Cycle Time Days Continuous DEPLOYMENT

54. Upward Spiral
Making Easy = Right 56

55. Lessons Learned
DevOps @ Wotif: Making Easy = Right
57

56. Lessons Learned
Making Easy = Right 58
Get the Balance Right
Chip Away At The Iceberg
Migration Notes
The Carrot, Not The Stick
Strongly Type Your Discussions
Use Semantic Versioning
Be Opinionated – Not Arrogant
Warranty Period
Embrace Legacy

57. Get the Balance Right
- Optimise for Speed
No Rules Too Many Rules
Making Easy = Right 59

58. Chip Away at the Iceberg
Making Easy = Right 60

59. Migration Notes
• Annotate changes since previous
standards
– E.g. “(since v2.3)”
• Each standard version has migration
notes from previous version
• Can upgrade between several versions by
following migration notes nav bar
Making Easy = Right 61
1.0 2.0 2.1 2.2 2.3

60. The Carrot, Not The Stick
• We tried the stick
approach for years
– You’re doing it wrong!
– (But we won’t tell you how
to do it right)
• Blame games, anger
• No business impetus to
improve things
• Then we tried the carrot
approach
– Offer a great new
alternative
– It’s optional!
– You know you want it…
• Business invested in the
process
Making Easy = Right 62

61. Strongly Type Your Discussions
• Choose catchy names
– Helps make decisions
– Helps climb out of muck
• Use State Transition tables
– State -> Action -> New State
– Visualise problems
– Decide what to support
Making Easy = Right 63

62. State Transitions – Current
Making Easy = Right 64

63. State Transitions – Future
Making Easy = Right 65

64. Use Semantic Versioning
• Especially for shared libraries
Making Easy = Right 66
1 . 2 . 3
Major Minor Patch
semver.org

65. Be Opinionated – Not Arrogant
• Strong decisions are important
• But you’ll never get it 100% up front
Making Easy = Right 67
I’m kind
of a big
deal

66. Warranty Period
• Can’t get standards
100% right
• But need to lock
them down
• Two-week warranty
period
• Trial in range of
different applications
Making Easy = Right 68

67. Small Batch Size for Standards
• Minor updates every
1-2 months
• Fast feedback
• Enable innovation
• Easier to update
Making Easy = Right 69

68. Embrace Legacy
• You’re creating legacy
• Make migration easy
• Incentivise catch-up
Making Easy = Right 70

69. Building Blocks
SLIPway
• Simple Rules, Independence
Rolling Upgrade
• Automated Deploy & Test
Compliance Tests
• Test-driven Ops Compatibility
Standards
• The Agreed “Right Thing”
Making Easy = Right 71
The Right Thing
Making It Easy

70. Key Takeaways
• Make the interface between teams (dev/ops)
consistent and predictable
• Cultural choices (agreement) > tool choices
• Prioritise operational feedback to developers
• Faster releases leads to smaller batch size leads
to lower risk leads to happier customers
• Start with interactive automation
• Independent releases with no dependencies
• Incentivise legacy catch-up through operational
compliance to latest agreed standards
Making Easy = Right 72
Consistent
and
predictable
interface between
teams
(dev/ops)
Cultural
choices
(agreement)
>
tool
choices
Prioritise fast
operational
feedback
to
developers
Faster
releases
à smaller
batch
size
à lower
risk
à happier
customers
Build
trust
with
interactive
automation
Decoupled
releases
– independent&
dependency-­‐free à speed
Incentivise legacy
catch-­‐up

71. Making Easy = Right 73
Right
Easy

72. Making Easy = Right 74
Right
Easy

73. Thanks for listening!
Any questions?
Alexandra Spillane • @ajbw
Matt Callanan • @mcallana
slides, etc • mattcallanan.net
IEEE Article • http://bit.ly/29htOn8
Making Easy = Right

74. Image Attribution
Image
“Downward
Spiral”
(http://flic.kr/p/67Giiz)
by
Chad
K is
licensed
under
CC
BY
2.0
(https://creativecommons.org/licenses/by/2.0/)
“Angels
Landing
from
the
Deertrap
Mountain
Trail”
(Chasm
-­‐ https://flic.kr/p/abuWHt)
by
Zion
National
Park is
licensed
under
CC
BY
2.0
(https://creativecommons.org/licenses/by/2.0/)
“WRONG
WAY”
(https://flic.kr/p/cRjc6q)
by
David
Goehring is
licensed
under
CC
BY
2.0
(https://creativecommons.org/licenses/by/2.0/)
“Old
Gravel
Silos,
Buffalo
Bayou,
East
of
Jensen,
Houston,
Texas
0906091555BW” (https://flic.kr/p/6WueVm)
by
Patrick
Feller is
licensed
under
CC
BY
2.0
(https://creativecommons.org/licenses/by/2.0/)
“Fisher
men
lifting
a
boat
in
Bangladesh.
Photo
by
Finn
Thilsted”
(https://flic.kr/p/ddUf4S)
by
WorldFish is
licensed
under
CC
BY
NC ND
2.0 (https://creativecommons.org/licenses/by-­‐nc-­‐nd/2.0)
“Poor
visibility
from
Kelud
eruption
14
February
2014,
Yogyakarta”
(http://bit.ly/1N3Lf7A)
by
Aldnonymous is
licensed
under
BY
SA
3.0
(https://creativecommons.org/licenses/by-­‐sa/3.0/deed.en)
“Changed
priorities ahead”
(https://flic.kr/p/debvm)
by
Peter
Reed is
licensed
under
CC
BY NC
2.0 (https://creativecommons.org/licenses/by-­‐nc/2.0)
“Stop Sign” (https://flic.kr/p/aUEW1D) by DonkeyHotey is licensed under CC BY 2.0 (https://creativecommons.org/licenses/by/2.0/)
“green
scum”
(https://flic.kr/p/3MvvsH)
by
M&R
Glasgow is
licensed
under
CC
BY
ND
2.0 (https://creativecommons.org/licenses/by-­‐nd/2.0)
“Thinking…
please
wait “
(https://flic.kr/p/6wdLat)
by
Karola
Riegler is
licensed
under
CC
BY
ND
2.0
(https://creativecommons.org/licenses/by-­‐nd/2.0)
“Elastic“ (https://flic.kr/p/a2XZDB)
by
Chris
Stevenson is
licensed
under
CC
BY
NC
2.0 (https://creativecommons.org/licenses/by-­‐nc/2.0)
“A
Garden
Of
Climbs”
(https://flic.kr/p/4U2BB1)
by
Jasen
Miller is
licensed
under
CC
BY
2.0
(https://creativecommons.org/licenses/by/2.0/)
“Alexander
Vinokourov
competing
in
the
London
2012
Men's
Olympic
Time
Trial”
(http://bit.ly/1QndiBd)
by
Diliff is
licensed
under
BY
SA
3.0 (https://creativecommons.org/licenses/by-­‐sa/3.0/)
“Upward
Spiral”
(https://flic.kr/p/noFia)
by
Clint
Vigil by
is
licensed
under
CC
BY
NC
2.0 (https://creativecommons.org/licenses/by-­‐nc/2.0)
“Rivers
of
Humanity”
(Indian
Traffic
-­‐ https://flic.kr/p/3yMSMW)
by
pangalactic
gargleblaster
and
the
heart
of
gold is
licensed
under
CC
BY
NC ND
2.0
(https://creativecommons.org/licenses/by-­‐nc-­‐
nd/2.0)
“Red
Tape“ (https://flic.kr/p/b7T3JD) by
Free
Press/
Free
Press
Action
Fund is
licensed
under
BY
NC
SA
2.0 (https://creativecommons.org/licenses/by-­‐nc-­‐sa/2.0)
“To
the
Right
A
Bit”
(Slipway
-­‐ https://flic.kr/p/9oQVh8) by
tiffany
terry is
licensed
under
CC
BY
NC ND
2.0
(https://creativecommons.org/licenses/by-­‐nc-­‐nd/2.0)
“Iconic
iceberg
sail-­‐by” (https://flic.kr/p/hV1vwH) by
Visit
Greenland is
licensed
under
CC
BY
NC ND
2.0 (https://creativecommons.org/licenses/by-­‐nc-­‐nd/2.0)
“Luckiamute
Falls”
(https://flic.kr/p/7EMs4T) by
Ian
Sane is
licensed
under
CC
BY
2.0
(https://creativecommons.org/licenses/by/2.0/)
“Three Keys” (https://openclipart.org/detail/219938/water)
by
maryjean.jones@fennwoode.com is
licensed
under
unlimited-­‐commercial-­‐use (https://openclipart.org/unlimited-­‐commercial-­‐use-­‐
clipart)
Presentation
includes
stock
images
used
under license
from
Shutterstock.com and
iStockPhoto.com.
Authors
include:
Stepanek Photography/Shutterstock.com,
skvoor/Shutterstock.com,
Michael
D
Brown/Shutterstock.com,
argus/Shutterstock.com,
Sergey
Nivens/Shutterstock.com,
Anze Mulec/Shutterstock.com,
Jaz_czc/Shutterstock.com,
DenisNata/Shutterstock.com,
Arnaud
Weisser/Shutterstock.com,
iStock.com/gehringj
Standardisation Theory
Diagram
inspired
by
Sam
Newman:
http://www.slideshare.net/spnewman/practical-­‐microservices-­‐yow-­‐2013/56