KnowledgeHut - Switching On DevOps

3. ABOUT ME My journey at Tatts Developer DevOps Engineer DevOps Strategist Engineering Manager https://linkedin.com/in/shawinneshttps://twitter.com/shawinnes Our Achievements Introduced CI (TeamCity) Introduced CD (Octopus) Introduced IAC (Chef, Automation)

4. ABOUT TATTS

5. MY INTERPRETATION OF DEVOPS • Culture, Automation, Lean, Metrics, Sharing • Tools help, but they’re not that important, it’s the mindset • It’s not a team, job title, or a role, it’s the practices* • It’s ALL about the culture • Don’t hire brilliant jerks & heroes, hire collaborators • Watch this YouTube video * sometimes you compromise on things to achieve bigger goals in the long run…

6. BACKGROUND • SCM Team • Environment Management Team (EMT) • CI Automation • Build Automation • Git trunk-based development • Deployment Automation • Big Brick Wall

7. THE PROBLEM • Theory of Constraints • We knew what to tackle next • Lack of knowledge & skills

8. PREVIOUS / EXISTING ATTEMPTS • Ops manually building and maintaining • Microsoft SCCM • Simple scripting

9. HEAD WINDS • No record of configuration • Maybe word documents • Regulated environment • An insane number of new tools • No public cloud available for use • Negative mindsets • Middle managers

10. TAIL WINDS • World-class on-premises data centres • Executive support • Positive mindset (a small group of dedicated followers) • Minimal budgetary constraints

11. APPROACHES • Proof of Concepts • TattsCloud • Patterns & Abstractions • Maximising Re-use

12. PROOF OF CONCEPT - PACKER • To solve these problems • VMWare template sprawl • Building OS Images manually / using MS tool • No version control • Demo on laptop – Packer, Vagrant, VMWare, Windows VM, Chef

13. TATTSCLOUD • Used existing VM infrastructure • VMWare V-Realize Automation • VMWare V-Realize Orchestrator • Adopted cloud-like patterns to maximise re-use • Used an integration approach to minimise product lock-in • Adopted CI/CD processes to manage all configuration • Test-driven infrastructure

14. TATTSCLOUD – CI/CD ChefGitHub Octopus TeamCity VMWare VRA/VROContributor Cookbook Role Environment DNS Template Network VM Host SolarWinds Packer Packages Web Buildspec Packerfile Deployed Instance Amazon EC2 AMI VPC

15. EXAMPLE OF PATTERNS - LAYERING (DIAGRAM) Hypervisor-specific Configuration * box / vmdk / ami Base Operating System Windows / Linux (patched), vm tools Tier 1 Components AV, SCCM, SOE, Octopus Customisation Dependencies .net runtimes, java, folder structures, services * VMWare (Fusion, Desktop, ESXi), VirtualBox, AWS

16. AND NOW... ? • Started September 2015 • Current State (Post IAC Pod) • Transformed way of working • TheLott API story

17. KEY MESSAGES • Start Small • Develop Patterns • Cloud patterns even if you’re not using cloud • Container patterns even if you’re not using containers • Treat it like a product • Maximise Re-use • Don’t underestimate complexity • Don’t underestimate organisational change management • Remember to take a break every so often to reflect and marvel at what’s been achieved

Notes de l'éditeur

Today I’m going to share some lessons about how we switched on infrastructure automation at Tatts Group…
Hi I’m Shaw and I worked at Tatts Group for around seven years. During that time I held a number of roles. I started as a developer in the Lotteries terminal team where we built the software used in lotteries agencies in every state except for Western Australia. It was during this time that we introduced a bit of a skunkworks automated CI build process. After a couple of years I moved into the UBET web team as a DevOps Engineer so I could help get some repeatable processes for their builds and deployment. UBET web was to be one of the first continuously delivered products at Tatts. I then moved into the enterprise agility team where I became a DevOps Strategist – a title we made up to ensure we were looking at the bigger picture, not just the day-to-day. During this time our team worked to transform the entire organization’s build and deployment processes to use fully automated build, test and deployment (where appropriate). It was during this time that we introduced the concept of infrastructure as code (IAC). Over the next two years we transformed the way the company built and managed infrastructure, and that’s what I’m going to talk about today.
But first, a bit about Tatts. There are a lot of people who haven’t heard of Tatts, but you’ll know their Lotteries brands such as powerball, oz lotto and instant scratch-its. They also run a wagering brand, UBET which has retail and online products for gambling on sports and horse racing, as well as a radio station. There is a gaming services division which monitors poker machines in multiple states and also has equipment support contracts for retail outlets and national telecommunications providers. Finally there is George Squared, a small charitable division providing technology solutions to help charities raise funds. Tatts Group is a 175+ year-old company which has gone through a long history of growth, mergers and acquisitions. As a result of this growth the company had a variety of systems and processes, and the challenges that come with this. During my time at Tatts I had the privilege of working in, or consulting with, all four of these business units, and in the wider business. This gave me a broad understanding of the cross-cutting challenges facing our 400 person strong IT department and allowed us to work on solutions which could be applied across the board.
Before I get into what we did at Tatts, I’d like to share my thoughts on what DevOps is, and isn’t. Despite the fact I had two job titles with DevOps in there, I think this is an anti-pattern and in reality everyone should be “thinking” about DevOps Culture, Automation, Lean, Metrics, Sharing -or CALMS. I found that as we were rolling out our agility and DevOps transformation, we spent much more time on culture and people than on tools and technologies. There’s a great talk by Adam Jacob (pictured) from Chef where he introduces the concept of DevOps Kung Fu where it’s more of a way of thinking and mindset than a specific tool. Tools help, but they’re not that important, it’s so much more about the mindset, and this is why if you get the right attitude and culture – your people will be able to achieve anything!
Prior to me moving into the UBET web team to set up some continuous delivery processes, most of our software builds and source code management work was undertaken by a separate team (who incidentally sat in our Ops space). This team eventually evolved into the Environment Management Team (EMT) with a wider remit to also maintain our various large test environments. These poor guys were so under the pump to be doing software builds and releases, branching, environment management and who knows what else. The UBET team had already determined that EMT was a bottleneck and so they were trying to remove their reliance upon EMT. The problem was that they hadn’t automated anything and so they were basically losing a developer's time for a couple of days every 2 weeks to manually build the software and deploy it to a test environment. So when I joined that team the first thing we did was to automate the build process by introducing TeamCity. I guess the tools specifically don't matter, this is just what we used and again, it's more about the mindset than the tools. This would at least give more timely feedback of merge or compilation issues. To make this easier we decided to adopt Git version control at this time. The ultimate goal would be to deploy the UBET website to production via fully automated pipelines, but the first step was to just get it deployable to a test environment without human intervention. For this we chose to use Octopus Deploy. So now we had pretty much got to the point where any test branch or master branch would be automatically deployed to a virtual host in the test environment. There were probably 25 developers working on this project at the time and so that meant that at least once a day when they were committing changes, a 250mb build would be deployed to their test environment. And then things blew up… in the funniest way possible. We filled up all the test environments with all these test builds. Oops.
As we had been working through the various changes up to this point we had been applying the theory of constraints to each problem as we saw it. We’d improved the flow behind the infrastructure by automating the build, deployment and improving version control practices. Infrastructure provisioning was our next constraint. Up until this point we had been using a single test environment, with a couple of long-lived servers. Now we wanted to be able to scale that up, and ultimately use any automation to eventually provision our production environments. What we had to tackle next was infrastructure automation, and it was going to be a challenge because it meant stepping out of the development teams where we had built up some great positive support. The developers and testers were loving their new way of working, that they could push a change and have it available in an isolated environment for a tester to work with. The infrastructure teams, including EMT, had very little exposure to automation, version control or DevOps concepts. A few forward-thinking people in those teams did hear the pain-points of the development teams though.
Those forward-thinkers who were focused on trying to solve the problems of their “customers”, the developers, tried a few approaches. The first attempts were to just do things faster, which we all know wasn’t going to work long term – this just resulted in frustrated people. Another group started to build automation based on the SCCM task-runner, this was a noble attempt, but the SCCM task runner was designed for running a couple of tasks on a managed machine – it wasn’t meant to automate the provisioning of an 80 VM test environment. If any part of the process failed, the whole process failed and you would end up with a mess to clean up. Unfortunately it was very common for these task sequences to fail part-way through. There were also pockets of ad-hoc automation occurring, but again these were generally flaky and not managed or consistently applied. These were also dependent upon who was doing the work.
There were a few things working against us moving to a new way of delivering infrastructure. Due to the fact that much of the existing deployed infrastructure was manually configured, we had minimal record of what had actually been built, and more importantly what had since changed. There were confluence articles and word documents for some things, but most were outdated. Although I hated hearing this excuse, many of the systems managed by Tatts are under some level of regulatory scrutiny. This sometimes meant we weren’t able to take the easiest option, though it usually just meant having conversations with the regulators who were generally excited about progress – they just wanted to understand any impacts changes would have on their ability to maintain integrity of the systems. In reality, the automation of these processes would greatly improve the integrity of everything. We were about to introduce a huge number of new tools, some of which were familiar to those of us from a development background, but definitely not to the infrastructure and ops teams. I remember one meeting where we were talking to some of the infrastructure teams about version control and checkins and checkouts and thankfully someone stopped me and pointed out that they didn’t know what we were going on about. This was a good reminder that DevOps is about sharing, and having understanding and empathy for your colleagues. Now a couple of years after that conversation and almost everyone in the organisation is comfortable with version control and we have ops teams managing DNS via git and automated deployment. Automation, Continuous Delivery, IAC are all really easy with public cloud. It was kind of built for these concepts, but as I mentioned earlier about the regulatory impacts, and the use of public cloud was one area in which our hands were tied to some extent. It wasn’t that we were unable to use public cloud, but for various reasons we just hadn’t got to the stage of seeking approval. Regardless, there were operational reasons why we wouldn’t move our entire workload to the cloud and so any investment in automation of legacy VM infrastructure was going to pay off anyway. There were a few negative people in various teams across the company, mostly due to feeling threatened by automation or changing the way they work. Many people associate their identity with what they do – and so if someone is an infrastructure engineer and you question the way they do their job, suddenly they have an existential crisis. Understanding what makes different people tick goes a long way to minimising these fears, but it’s an imperfect art and an area where we tripped up a couple of times.
On the upside, we also had some great things going for us. Due to the scale and importance of system integrity and availability, Tatts had invested a huge amount of capital into their data centres and they were world class. Some of the nicest data centres I’ve ever seen and with great enterprise software and support. This meant we had access to work with some of the engineers at places like VMWare and Chef when we were trying to bend their products to our puzzling will. We had support for these initiatives from the top of the organisation. Mandy Ross, the CIO was 100% behind these efforts because she understood the benefits. Executive support is one of the things people always ask about at conferences and meetups. We were really lucky to have someone who we didn’t need to convince. Once we started introducing infrastructure as code and some other things, the supporters started to come out of the woodwork and we built a really good team of advocates and guilds of people who were keen to participate in a community to make this stuff a success. In fact, the infrastructure as code guild was one of the most vibrant (and passionate) communities of practice with regular meetings to discuss any problems and to come up with solutions. Finally, one of the benefits of being a company with $4.7B market cap is that budget isn’t as tight as at some other places. The main benefit of this wasn’t so much in our ability to purchase things, but that we were given the freedom of being able to stand up a dedicated R&D project team for almost two years to work on piloting these ideas and to work with other teams to distil what would work best for the whole company.
So, we used a few general approaches for our incremental build of these practices. Proof of concepts, we built our own internal self-service cloud offering ”TattsCloud”, we tried to establish patterns and abstractions so that people could apply them to other similar use cases, and leading on from this we attempted to maximise re-use of anything we built.
We had some “DevOps Jam” sessions with some of our new friends in the infrastructure teams about common problems we were all sharing. One of these was around Windows VM template creation and management. The natural tools for managing these images were not version controllable and as such there were manually copied versions of configurations and templates all over the place. I had been running a couple of side projects and doing research into solutions for scalable infrastructure automation. Although our infrastructure was all on-premise, I looked at tools and patterns in use by public cloud providers like AWS and Azure and identified a few tools which might be worth looking into. The Tatts tech stack was very Windows-heavy at the time so it was crucial that any solutions would work on Windows, but we also anticipated that we would eventually want to adopt more open source products running on Linux, so we didn’t want to select a Windows-only solution. Another factor in the selection of tools and processes for infrastructure automation was that we wanted to create a set of practices where the development experience would match production as closely as possible. It had been an ongoing problem where development, test, and production were inconsistent and we would have configuration drift resulting in protracted release processes. A major keystone in the whole process was Chef. For those who haven’t come across this tool before, it’s a configuration management tool that lets you define your server infrastructure as configuration files. There are other similar tools such as Puppet and Ansible. So I put together a pilot using the tools shown on the slide, using vagrant and packer to do development of base images so that we would have a good developer/contributor experience. Because I knew there would be some resistance to changing things here because “that won’t work here”, I built a replica of our VMWare vSphere setup, with management servers, VM hosts etc. Then on my laptop I did a full end-to-end demonstration of how we could use a configuration file to built a Windows VM template and have it automatically added to my vSphere setup and then stand up a new VM based on this template. There was a lot going on at once, the fan on my laptop was going crazy! But in the end the demo was a success and people were willing to give it a go, and then we started talking about learning git, and packer, and vagrant… and all the other tools we needed to adopt.
Once we’d proven some value in this automation thing, we somehow managed to convince our CTO and CIO to lend us a few more people to start up an agile team to start building automation around our existing VMWare infrastructure and build what we dreamed would be an internal self-service cloud. Because we wanted to maximise portability and re-use though, we didn’t make use of the VMWare products exactly as intended. We had some strong opinions about how we wanted to manage configuration, though code, and the platform was lacking in this respect. So we ended up building glue between a few of our existing tools rather than building heaps of logic into the VMWare platform. One of the other aspects which drove our decision to integrate systems rather than heavily configure the existing products was that we wanted to use a CI/CD process for all configuration changes and to build automated testing into a pipeline to increase the quality of contributions to the infrastructure artifacts. We were able to make use of our existing GitHub and Teamcity systems to provide quality control along the way, just pushing the final configuration into the VMWare platform at the end of the process.
This diagram shows roughly how it all hung together. The green boxes are configuration or other artefacts which we wanted to version control. These were the source of truth for everything. If anyone wanted to make a change they could do local development on their workstation using Vagrant and other tools. Once they were happy with the changes they would submit a pull-request change through GitHub which would be peer-reviewed (not ”approved”) by the community. The accepted pull request would then be built or checked by teamcity depending on what it was. For example chef cookbooks which passed automated testing would be pushed into our chef server, if they failed they would never get to the server which meant we reduced the chances of bad scripts getting into real environments. In the use-case I mentioned earlier with the packer proof of concept, we got to the stage where all of our windows and linux (yes linux now!) VMWare templates were generated 100% automatically each month and were patched with the latest versions of the vendor patches. These were then automatically tested and pushed into our VMWare clusters for consumption by users. Each month the obsolete templates were also cleaned up by the process, eliminating the previous template sprawl and clutter problem. With all of this setup we were then able to use the VMWare vRA platform to provide a self-service request form which would allow a user to request a specific type of VM into any environment from development, through test, performance test and into production. They were able to select from a defined list of roles and cookbooks and set some parameters on the deployment. About 15-20min later they'd receive and email notifying them that their requested VM was ready to go. One of the other really cool things we did was to integrate SolarWinds IPAM (IP Address Management) into our self-service pipeline. The VMWare platform manages IP addresses normally, but it’s quite restrictive in the way it does this and requires the entire address range to be managed by VMWare. Due to the evolution of the infrastructure at Tatts this wasn’t feasible, and we didn’t want to create a huge amount of work to restructure the entire VM farm. Instead, the VM request workflow would request a free IP from SolarWinds dependent upon which cluster and network zone the VM was going into, this meant that the requestor of a VM didn’t need to worry about IP addresses at all, they just needed to specify which zone and environment, and the platform would take care of the rest.
As I mentioned earlier we liked to develop simple, re-usable, patterns and abstractions as part of our work. This is an example of one. On the left, in purple are the four abstraction layers we were using when building up a destination server. On the right hand side in blue are concrete examples to illustrate what I mean by layering. By using a pattern like this it made it easier for us to re-use automation scripts and processes across different VM tools such as Vagrant, VMWare, and AWS. It also meant we were following similar patterns across different operating systems. This helped to reduce the maintenance burden and cognitive load on people who were maintaining pipelines. Imagine if we had used all the native windows tools on windows, and the native Linux tools on Linux – the result would have been two totally separate code-bases. Instead we ended up with only a tiny amount of bootstrapping required for each operating system and then we used tools like Packer and Chef to do the rest. Obviously the specifics of what was loaded on the different servers varied, but the pattern was the same and as a result it made the transition from windows to Linux more of a step than a leap.
We started the infrastructure as code POD in September 2015, and over the next 18-24 months we had almost a full-time group of 3-6 people working on building TattsCloud and the associated supporting tools. The team was involved in working with customer teams and advocating for the use of the new way of working. They were performing training and coaching, accepting feature requests from customer teams and showcasing their progress fortnightly. Now almost all VM instances are deployed using TattsCloud from development through to production. The VM templates are exclusively managed by the pipeline. The way that infrastructure teams and development teams work together has changed totally, to the point where they are both contributing to infrastructure code and chef cookbooks. The same templates and cookbooks are used to build development servers and then to build production servers. This means there is a massively reduced risk of inconsistency as work progresses through the environments. Finally, just before I left Tatts there was a great story of the lotteries team having a need to scale up some of their API infrastructure from 8 old physical servers which were on aging hardware and had been carefully hand crafted and cared for over the years. With a bit of coaching and encouragement the lotteries developers worked with some of the infrastructure engineers to define new API servers including network and load balancer automation. Over a period of a few weeks they built and destroyed hundreds of iterations of these machines in the test environments until they were confident that they could deploy a VM free from human hands. They eventually succeeded in this task and rolled out 30 VMs to production. A few weeks later there was a need to scale up even further and so with no further code or configuration changes they just submitted a request for 30 more VMs… a couple of hours later the job was done and the new VMs were automatically added to the load balancers. I can’t even imagine how long or painful it would have been previously to request and provision an additional 30 VMs – it definitely wouldn’t have been done in hours.
I’d like to leave a few points for you to think about. Start Small, this stuff is quite easy to sell once you get some wins on the board, so just start with a big pain point that people are experiencing and try to do something small to improve that. Once you’ve built some support by solving people’s problems they’ll be more engaged in being part of the change. Patterns allow you to scale adoption without too much pain. We looked at the patterns used by cloud and container systems for inspiration. For example, minimising dependencies to maximise reusability. The layering approach I talked about earlier was also inspired by container technology, rather than build 100 VM templates with every permutation of configuration and environment, why don’t we build a very generic base OS template, and customise it at deployment time. Less complexity, less sprawl. The downside was slightly longer build times, but that wasn’t such big deal in the end. If you’re building large-scale systems or integrations to do infrastructure as code, treat the whole thing as a product. It’s unreasonable to expect to build it and hand it over to a support team and happy days. Like any complex software system, your infrastructure as code pipelines and ecosystem require care and feeding. Consider the formation of a tools team to maintain and evolve the systems – they should be a blended team of systems and software people to ensure that the voices of all customers are heard – a true Dev+Ops team. I would also say don’t underestimate the complexity of this task, of course the benefits are huge, but there are a lot of moving parts and one of the great challenges we faced was with organisational change. You’re changing people’s identities and pushing them outside their comfort zones. Some people will love this and become raving fans, others will resist with all their might. Don’t underestimate the importance of managing this change, if you invest the time into doing it properly it will be a much smoother ride. Having said that, some times you just need to get on with it and show people what’s possible before they’ll be open to accepting the reality. And finally, take the time to retrospect on what HAS improved. It’s really easy when you’re in an enterprise with lots of legacy processes and systems to always be focusing on what can be improved and what needs to be fixed. As I was preparing to leave Tatts I reflected upon what had been achieved in my time there, and it was quite amazing. Not only had we implemented some impressive technical solutions, we had also totally changed the mindsets and ways of working of an entire technology department.

KnowledgeHut - Switching On DevOps

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

Similaire à KnowledgeHut - Switching On DevOps

Similaire à KnowledgeHut - Switching On DevOps (20)

Plus de Shaw Innes

Plus de Shaw Innes (6)

Dernier

Dernier (20)

KnowledgeHut - Switching On DevOps

Notes de l'éditeur