Building and scaling a multi-cloud solution that's enabled for cloud bursting is not a trivial task, and requires a lot of automation. While experiencing hyper-growth on the Adobe Advertising Cloud, our operations engineering team had to frequently update and improve its workflow in order to stay nimble and allow fast delivery of new infrastructure. At TubeMogul/Adobe Advertising Cloud, we implemented a lean Puppet workflow that enables the operations engineering team to deploy and support a broad range of services in a complex environment that supports hundreds of billions of requests a day. With over 150 changes released per day on its production infrastructure, the team had to adjust and tune its processes to enforce quality, standards, to review, and to prevent systems from breaking. In this talk, you will learn how we implemented our infrastructure as code by leveraging tools like Puppet, Gerrit, Terraform, and Jenkins, which together enable our private and public cloud infrastructures across 12 locations and four continents.

1. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
A Lean Puppet Workflow to Support a Multi-Cloud and Cloud-Bursting Infrastructure
Nicolas Brousse | Director of Engineering, Operations

2. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Increased Complexity and Faster Changes

3. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Immutable or Mutable?
3

4. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Tesla remotely extends range of vehicles for free in
Florida to help owners escape Hurricane Irma
4
Source: Electrek.co

5. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Tesla Over-The-Air Software Update
5

6. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Any connected device need to handle updates
6

7. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Connected Home gets more sophisticated
7

8. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Likewise Enterprises Infrastructure Complexity increase
8
• IaaS / PaaS (AWS, Azure, GCP, IBM, Oracle…)
• Cloud Productivity Apps (G Suite, Office 365, AWS...)
• CRM
• Private Cloud (OpenStack, VMware...)
• Containers Orchestration (K8s, Mesos, Swarm, Nomad...)
• Bare-metal servers, mainframes (IBM Z 2017)
• Edge Computing
• Quantum Annealer (D-Wave)
Next Quantum Computers?

9. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 9

10. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Patching (very) quickly matters
10

11. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
The Adobe Advertising Cloud

12. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Technical Challenges
12
Latency Requirement
<50ms @ 95th percentile
High Volume Traffic
+100 billions request a day
Huge Datasets
billions of object to store

13. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
> 6,000 servers
> 10,000 Puppet code deployments a year
> 40,000 Application deployments a year
1 Puppet Master
13

14. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 14

15. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Bare-metal Infrastructure Automation
Fully automated deployment
Rack-And-Roll Infrastructure
Puppet Hiera and eYAML
15

16. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Bare-metal Infrastructure Automation – Rack…
16
Buy Build
Burn
Test
ShipRack, Bolt, Powered onConnect to Network

17. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
And Roll…
17
Jenkins API TFTP PXE boot API puppet
Log centralization Monitoring and Metrics IdM & IDS

18. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Containers Environments
18
git jenkins
docker
terraform puppet K8s

19. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Cloud Bursting
19
git
jenkins
terraform
puppet
puppet
jenkins
Application DeploymentInfrastructure Deployment
git

20. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Final word

21. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Configuration Management is not dead
It’s complicated and diverse
21

22. © 2017 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Key Take Aways
§ Acknowledge the reality of complex infrastructure in the enterprise world
§ Understand your Infrastructures models
§ Strongly Immutable
§ Weakly Immutable
§ Mutable
§ Prevent and detect configuration drift + allow patching
§ Holistic understanding of your product lifecycle
§ Infrastructure As Code (multiple tools involved)
§ Feature Toggles
§ Beta Percentage Rollout
§ Blue / Green Deployments
§ Mean Time to Release Changes at scale
22

23. @AdobeAdCloud @orieg