Jody Mulkey, Chief Technology Officer, Ticketmaster As the largest live event ticketing company in the world, Ticketmaster is in the rare position of having to balance the needs of a multi-billion dollar global business with the need to defend against competitors that are hungry and very agile. Powered by legacy systems, it’s clear that Ticketmaster can’t rest on it’s laurels and must innovate to maintain it’s market leadership. But how do you take such a large, mission critical enterprise service and make it nimble enough to compete with competitors and delight our customers? When Live Nation and Ticketmaster merged 5-years-ago, the executive leadership team had the vision to commit to re-architecting Ticketmaster’s 39-year old ticketing platform. While this is an incredibly exciting prospect and will lead to industry changing innovation, it does create huge challenges for Ticketmaster’s Global Technical Operations team. As with any long lived enterprise, the “muscle memory” around managing, developing and delivering our service is very well formed. Changing habits is hard and in the high stakes game of selling tickets it is fraught with risk, both perceived and real.

1. Ticketmaster’s Transformation Journey
DevOps In the Enterprise
@jodymulkey

2. Powering Life’s Experiences
Amazon Confidential

3. Amazon Confidential

4. Video
 https://www.youtube.com/watch?v=TIA3N
52vm70

5. Even the longest journey must begin where you stand.
-Lao-tzu

6. Background - Live Nation
Largest Live Entertainment company in the world.
 >450 Million fans in 40 countries
 >120 Venues Owned and Operated
 >24K Concerts

7. Background - Ticketmaster
1976- Founded at Arizona State University
1996- Ticketmaster.com launched
2010- Live Nation and Ticketmaster join forces
2011- Transformation journey begins…

8. 2011 2014
Dev
Ops
230% increase
12% increase

9. Uh oh!

10. What now?
DevOps
to the
Rescue

11. Just like Football
Ops = Defense
Dev = Offense

12. Just like Football
Ops = Offensive Line
Dev = “Skill” positions

13. DevOps Strategies
• Empathy
• Empowerment
• Metrics

14. Empathy
 75 in 75
 Breathing Customer Oxygen
 Everyone is a fan
 Breaking bread together

15. Empowerment
 Teams crave responsibility
 One team, one mission, one goal
 Development Teams Oncall
 Self Service.*

16. Metrics
 Business metrics > System Metrics
 Outcomes > Outputs
 Instrument everything
 Democratize the data

17. Lessons Learned
Change is hard

18. Lessons Learned
Empowered Expertise = Lower MTTR

19. Support at the Edge
 Formalized tiered support model
 Move support as close to customers as possible
 Focuses teams on projects that prevent problems

20. Clients and Fans

21. Support at the Edge - TOC
Technical Operation
Center
(TOC)
=
EMT

22. Support at the Edge - SRE
Site Reliability Engineers
(SRE)
=
Emergency Room Doctors

23. Support at the Edge – Production Engineering
Production Engineers
=
Surgeons

24. Before Support at the Edge
0
45
90
135
180
225
0 10 20 30 40 50 60
M
T
T
R
Incidents
TMOL WebOps Incidents
Average MTTR
47 Minutes

25. After Support at the Edge
0
50
100
150
200
0 10 20 30 40 50 60
M
T
T
R
Alerts
TMOL WebOps Incidents
Average MTTR 3.8
Minutes

26. Lessons Learned
DevOps has nothing to do
with technology

27. Ticketing Engine (aka The Host)
 Powers $25B in commerce
 Performance measured in microseconds
 Code first committed 1976
 VMS running on emulated VAX

28. Bedrock Team
• Deploys their own
code
• Integrated PagerDuty
for Oncall Alerts
• Leverages
Cucumber/BDD for
automated functional
tests
Ticketing Engine (aka The Host)

29. Lessons Learned
There are only two states:
You own the system
or
The system owns you

30. Ticketmaster.com (AKA TMOL)
 Drives 40% of revenue
 Majority of system built in 2000 ModPerl
 Tribal knowledge no longer at the
company

32. Metal to Money – No Hands
 Development team taking over svc delivery
 Push Button Deployment of the TMOL stack
 Tribal knowledge encapsulated in code

33. Closing Thoughts
Ego is a force field for learning

34. To know that you do not know is the best
To pretend you know when you do not know is a disease
-Lao-tzu