A brief history of Ticketmaster's journey and some of the bumps in the road that affected our collaboration between Engineering and Operations . . . and what we're doing about it next. International evolved as a franchise of a US company into a large group in it's own right, expanded by the merger in 2010 for Ticketmaster to become part of LiveNation Entertainment. Over the years as teams and products expanded and contracted we have been faced with different barriers - timezones, culture, compliance, politics &apm; technology. Our latest 'DevOps' changes as we have started to migrate to AWS started to highlight some gaps in our thinking so I'd like to share what we are doing next to help us prepare better for the future.

1. EVOLUTION OF
TICKETMASTER’S
JOURNEY TO DEVOPS
WINOPS LDN SEPTEMBER 2017
#WinOps
@connonm
@tmTech

2. WHO AM I & WHY AM I HERE?
Connon MacRae
VP, Technical Operations
(International)
#WinOps
@connonm
@tmTech
HISTORY IS USEFUL
CONTEXT IS EVERYTHING
DON’T BE AFRAID

3. MICROSOFT VS LINUX
MICROSOFT LINUX
NO MORE CYGWIN ON WINDOWS#WinOps
@connonm
@tmTech

4. #WinOps
@connonm
@tmTech
WHO IS TICKETMASTER?

5. LIVE NATION ENTERTAINMENT – 42 MARKETS
Live Nation & Ticketmaster
Live Nation ONLY
Ticketmaster ONLY
#WinOps
@connonm
@tmTech

6. #WinOps
@connonm
@tmTech
THE HOST
8 HOURS
5-9 HOURS
1975 – TM started with Albert Leffler, Peter Gadwaw, Gordon Gunn . . . . . .
Also Bill Gates and Paul Allen develop a BASIC program for the Altair 8800,
Microsoft becomes a registered trademark.
BetaMax is introduced.
Motorola obtains patent for mobile phone!
1976 – TM Starts . . . . IBM Launch the first laser printer.
1977 – ELO is first ticketed event at Johnson Gymnasium, University of New Mexico
1978 – Norway is the first international client with Oslo Nye Theatre
1981 – Ticketmaster UK Opens above the Albery Theatre
1983 – Ticketmaster UK starts to work with Arsenal
1988 – Ticketmaster Australia operations begin

7. THE THING YOU REALLY NEED TO KNOW
#WinOps
@connonm
@tmTech
I WANT THE
BEST SEAT(S)
ONSALE . . . 9AM . . .

8. #WinOps
@connonm
@tmTech
GROWING FAST
8-9 HOURS
5-9 HOURS
5-6 HOURS
10 HOURS
1-4 HOURS
5-9 HOURS
1991 – Ticket Tron in North America acquired
1996 – UEFA Euros with Ticketmaster United Kingdom
1996 – First Ticketmaster internet sale!
1997 – Archtics (Sports & Arts) integrated with and later ‘DSA’ acquired.
1997 – Synchro Systems (later Ticketmaster Sport)
1997 – Ticketmaster Ireland as a joint venture
1998 – Mexcio starts operations
1998 – ticketmaster.co.uk goes online, Windows NT 4
1999 – Joint venture in Australia
1999 - Rugby World Cup in Wales
2000 – TicketWeb, Vista and Admission Canada
2001 – First deal with Clear Channel – later to become Live Nation
2001 – Norway Billetservice acquired
2002 – Manchester Commonwealth Games
2002 – Netherlands & Denmark acquired
2004 – Athens Olympics
2004 – Sweden and Finland acquired
2005 – New Zealand established and Kartenhaus in Germany acquired
2006 – Acquires TickTackTickets in Spain and Biletix in Turkey
2007 – TM China joint venture
2008 – Beijing Olympics
2008 – TM acquires TicketsNow & GetMeIn,
2008 - Paperless tickets debut in North America
2008 – Paperless debuts in UK with 18k people attending at the O2 Arena
2008 – Ticketmaster Spun off from IAC
2008 - FrontLine Entertainment management acquired
2009 – TM UK breaks records with Michael Jackson, 50 dates at the O2.
2010 – LN and TM Merge to become Live Nation Entertainment
2010 - Ticnet in France acquired
2011 – Expansion in Spain with ServiCaixa
2012 – Ticketing for London2012, over 10.9 million tickets sold and processed.
Unprecedented at the time.
2013 - Launches in the middle east with UAE
2014 – Glasgow Commonwealth Games.
2014 - TM acquires Seatwave across Europe.
2014 - Middle east expands to Qatar
2014 – Relaunch DE, ES, acquires PL and Sherpa in Belgium.
2015 – Universe and Frontgate join the company
2015 – England Rugby World Cup
2016 – TM acquires Tickethour
2017 – TM acquires TicketPro in Czech Republic and launches Ticketmaster Israel
14 CORE
TICKETING
PRODUCTS

9. #WinOps
@connonm
@tmTech
THE INTERNET - 1998
WWW
NT4
SQL
THE ‘HOSTS’ - VAXEN 1999
8%?
20%
GULP

10. #WinOps
@connonm
@tmTech
REBUILD v1 – 2000
WWW
LINUX
ORACLE
THE ‘HOSTS’ - VAXEN
MySQL

11. #WinOps
@connonm
@tmTech
THEN THESE . . . . .
SOX -
2002
PCI v1 -
2004

12. #WinOps
@connonm
@tmTech
REMEMBER THESE?

13. #WinOps
@connonm
@tmTech
WE BUILT THIS . . .
DEV QA
CAP
STAGE
5-8 HOURS
PRODUCTION
. . . WE WERE PROUD OF IT!

14. LESS THAN . . .
OPTIMAL?
#WinOps
@connonm
@tmTech

15. Group A
WINDOW
OPS
Service
Delivery
WINDOWS
SYSTEMS
DBA TEAM
LINUX
OPS
LINUX
SYSTEMS
Group B Group C
PLATFORM
Product Support
CLOUD
SYSTEMS
TOOLING
NETWORKINGDATACENTRE
INTERNATIONAL TECHNOLOGY TEAMS IN 2016
#WinOps
@connonm
@tmTech
SECURITY
CORP IT
ARCHITECTURE
NA TEAMS
Product
Group 1
Product
Group 2
Product
Group 1
Product
Group 2
Product
Group 1
Product
Group 2
Product
Group 3
Product
Group 3

16. THE MORE YOU TIGHTEN YOUR GRIP,
THE MORE STAR SYSTEMS WILL SLIP
THROUGH YOUR FINGERS
#WinOps
@connonm
@tmTech

17. WHERE HAVE WE BEEN?
Engineering Ops
Syseng
Database
Engineering Ops
Syseng
Database
Engineering Ops
Syseng
Database
#WinOps
@connonm
@tmTech

18. WHERE HAVE WE BEEN?
#WinOps
@connonm
@tmTech
Engineering
Ops
Syseng Database
Platform

19. PUSH ME, PULL YOU
#WinOps
@connonm
@tmTech

20. PUSH ME, PULL YOU
#WinOps
@connonm
@tmTech

21. Ops & Reliablity
Service
Management
Ops & Reliablity Ops & Reliablity
PLATFORM
ENGINEERING
Product Support
Runtime
Services
& Data
Tooling
Infrastructure
(including DC)
Network
Engineering
Technical Service Delivery
SCALING THE SILOS, LOWERING THE WALL
PLATFORM
INFRASTRUCTURE
#WinOps
@connonm
@tmTech
Group A Group B Group C
Product
Group 1
Product
Group 2
Product
Group 1
Product
Group 2
Product
Group 1
Product
Group 2
Product
Group 3
Product
Group 3
Group B
Product
Group 1
Product
Group 2
Ops & Reliablity

22. HOW CAN WE EVOLVE?
#WinOps
@connonm
@tmTech

23. CAT = LOVE = PRODUCT!
#WinOps
@connonm
@tmTech

24. TOO LONG, EXPENSIVE
#WinOps
@connonm
@tmTech

25. DOES NOT FIT ANYMORE
#WinOps
@connonm
@tmTech

26. PLAY (RE)NICE? KILL -9
#WinOps
@connonm
@tmTech
UNACCEPTABLE!!!!!!

27. TICKETMASTER CLOUD
#WinOps
@connonm
@tmTech

28. SOMEBODY ELSES BOX
TRANSPARENT COSTS
#WinOps
@connonm
@tmTech

29. CONTAINERS – PART I
#WinOps
@connonm
@tmTech

30. CONTAINERS – PART II
#WinOps
@connonm
@tmTech

31. 31
(MORE) MODERN TECH
CHANCE
TO CLEAN UP
WHAT ARE WE USING?
#WinOps
@connonm
@tmTech

32. TECH MATURITY
Open Source @
https://github.com/Ticketmaster/techmaturity

33. 1. CONTAINERISE your application with Docker or AUTOMATE your Operating Systems with CHEF
2. IMPLEMENTATE BETTER INSTRUMENTATION - no SSH or RDP in production!
Log aggregation and rotation, metrics
3. Use SYNTHETIC MONITORING to understand the health of your product
Define SLAs, track & alert on availability and performance
4. AUTOMATE YOUR BUILD PIPELINE by reusing common tools
Terraform & GitLab CI are the products of choice
5. GET SECURE
Pass pen/static testing
No passwords/keys in configs
6. DESIGN A SHARED NOTHING ARCHITECTURE
No shared filesystems (e.g. NFS, CIFS)
7. BUILD FOR AVAILABLITY- NO SINGLE POINTS OF FAILURE
Use auto-scaling groups and without hard-coded IPs
OUR PRINCIPLES FOR MODERNISATION
#WinOps
@connonm
@tmTech

34. SUMMARY
#WinOps
@connonm
@tmTech
HISTORY IS USEFUL
CONTEXT IS EVERYTHING
DON’T BE AFRAID

35. Q & A
#WinOps
@connonm
@tmTech