"VMware vCHS, Puppet, and Project Zombie" by Nicholas Weaver, Cloud Automation Architect, Hybrid Cloud Service, VMware. Speaker Bio: Nicholas Weaver is the Cloud Automation Architect for VMware's vCloud Hybrid Service (vCHS) platform and the primary architect behind the vCHS automation framework (Project Zombie). He is also a co-creator of the Puppet Labs Razor project and many VMware-specific free tools. He previously worked in the CTO office for EMC, in the EMC field as a vSpecialist, and as a infrastructure engineer in financial, media, and retail companies. Nick loves software-driven control, hacking prototypes together, speaking at user groups, and demonstrating automation innovation to the masses. Nick can be found on Twitter and Github as @lynxbat.

1. © 2011 VMware Inc. All rights reserved
Nicholas Weaver - Cloud Automation Architect
VMware vCloud Hybrid Service
VMware vCHS, Puppet, and Project Zombie

2. 2
About me

3. 3
About Nick
 Nicholas Weaver
• VMware – Cloud Automation Architect, vCHS
• (formerly) EMC – Office of the CTO, Global Alliances, vSpecialist, Demo-builder
• (formerly) VMware/EMC Architect: Financial, Retail
• Co-creator Project Razor
• Twitter: @lynxbat
• Blog: nickapedia.com
• LinkedIn: www.linkedin.com/in/nicholasweaver

4. 4
What is vCloud Hybrid Service?

5. 5
vCloud Hybrid Service: Extending To The Cloud
VMware vCloud
Hybrid Service
Your Data Center
Any Application… No Changes
Software-Defined
Data Center
VMware vSphere &
vCloud Suite
Existing & New Apps
Seamless Networking
Common management
One Support call
IaaS cloud owned and operated by VMware based on VMware software

6. 6
What do I do?

7. 7
Automation

8. 8

9. 9

10. 10

11. 11
Effort Evolution

12. 12
Why is automation important for vCHS?

13. 13
What is it really doing?

14. 14

15. 15

16. 16
The problem with success is…

17. 17

18. 18
Nothing stays the same

19. 19

20. 20

21. 21
We are not the first

22. 22

23. 23

24. 24

25. 25

26. 26
We need automation special sauce

27. 27
?

28. 28
project zombie

29. 29
Key principles

30. 30
Scale

31. 31
Extensibility

32. 32
Simplicity

33. 33
Resiliency

34. 34

35. 35

36. 36
What is Project Zombie?

37. 37

38. 38

39. 39

40. 40

41. 41

42. 42
Rez

43. 43
Distributed Resource Management
• Resources
• State
• Configuration
• Location
• Distributed Locking
• Ownership
• Event History

44. 44
Distributed Resource Management
Razor
Razor
Storage
Manager
Storage
Manager

45. 45
Engine

46. 46
Zombie Engine DSL - ZED
-------------- 1 ---------------
Init VPC
- Engine
- API Call
------------- 3 --------------
Install ESXi
Engine Broker Plugin
- Razor
Inputs:
- Razor IP
- Blade
- Hostname
- Network Information
Tasks:
- Create Policy
- Boot Blade
------------- 3 --------------
Prep TLM
Engine Broker Plugin
- Puppet vCenter
Inputs:
- TLM VCSA IP
- TLM VCSA Creds
- TLM DVS
- Portgroup Name
- Portgroup VLAN
- Portgroup Settings
- Ports: 128
- Load Balancing: IP Hash
- Reset at Disconnect: Enabled
Tasks:
- Create PGs
- d#p#v#-dmz-pg-##
- d#p#v#-esx-pg-##
- d#p#v#-pvt-pg-##
----------- 4 ------------
Deploy VPC VCSA
Engine Broker Plugin
- OVFTool
Inputs:
- OVA
- TLM vCenter IP
- TLM vCenter Creds
- Cluster
- Datastore
- VM Name
- Network
- VM Hardware Settings
- Memory: 16384
Tasks:
- Deploy VPC VCSA
- d#p#v#-mgmt-vc0
- Update VM Hardware
----------- 4 ------------
Deploy VPC VSE Pub Pair
Engine Broker Plugin
- Puppet vShield
Inputs:
- TLM VSM IP
- TLM VSM Creds
- Edge Settings
- Name
- Hostname
- Enable HA
- Declare Dead Time: 6
- Interface 1
- Heartbeat IP Addresses
- TLM Datacenter
- TLM Cluster
- TLM Shared Datastore
- Size: Compact
- Interface Settings
- Interface 0
- Name
- Type
- Portgroup
- Subnet Settings
- Interface 1
- Name
- Type
- Portgroup
- Subnet Settings
Tasks:
- Create VPC Pub Edge
- d#p#v#-mgmt-vse-pub
----------- 4 ------------
Deploy VPC VSE Priv Pair
Engine Broker Plugin
- Puppet VMware-vShield
Inputs:
- TLM VSM IP
- TLM VSM Creds
- Edge Settings
- Name
- Hostname
- Enable HA
- Declare Dead Time: 6
- Interface 2
- Heartbeat IP Addresses
- TLM Datacenter
- TLM Cluster
- TLM Shared Datastore
- Size: Compact
- Interface Settings
- Interface 0
- Name: InterVPC
- Type
- Portgroup
- Subnet Settings
- Interface 1
- Name: DMZ
- Type
- Portgroup
- Subnet Settings
- Interface 2
- Name: PVT
- Type
- Portgroup
- Subnet Settings
- Interface 3
- Name: ESX
- Type
- Portgroup
- Subnet Settings
Tasks:
- Create VPC Priv Edge
- d#p#v#-mgmt-vse-priv
----------- 5 ------------
Configure VPC VSE Pub Pair
Engine Broker Plugin
- Puppet vShield
Inputs:
- TLM VSM IP
- TLM VSM Creds
- Edge Settings
- DNS Settings
- Firewall Settings
- IPsets
- Application Groups
- Applications
- Firewall Rules
- Load Balancer Settings
- Enable
- Pools
- VIPs
- Default Route
- Syslog
Tasks:
- Configure VPC VSE Pub Pair
----------- 5 ------------
Configure VPC VSE Priv Pair
Engine Broker Plugin
- Puppet VMware-vShield
Inputs:
- TLM VSM IP
- TLM VSM Creds
- Edge Settings:
- Syslog
- DNS
- Firewall Settings
- IPsets
- Application Groups
- Applications
- Firewall Rules
- Load Balancer Settings
- Enable
- Pools
- VIPs
- Default Route
Tasks:
- Configure VPC VSE Priv Pair
----------- 7 ------------
Init VPC VCSA
Engine Broker Plugin
- Puppet VMware-VCSA
Inputs:
- Credentials
- NTP Settings
- Syslog Settings
- DB Settings
- Sizing: Large
Tasks:
- Init VPC VCSA
----------- 4 ------------
Deploy VPC VSM
Engine Broker Plugin
- OVFTool
Inputs:
- OVA
- TLM vCenter IP
- TLM vCenter Creds
- Cluster
- Datastore
- VM Name
- Network
Tasks:
- Deploy VPC VSM
- d#p#v#-mgmt-vsm0
----------- 9 ------------
Configure VPC VSM
Engine Broker Plugin
- Puppet VMware-vShield
Inputs:
- VPC VSM IP
- VPC VSM Creds
- VPC VCSA IP
- VPC VCSA Creds
- NTP
- Syslog
Tasks:
- Configure VPC VSM
----------- 4 ------------
Deploy VPC vCloud
Engine Broker Plugin
- ??
Inputs:
- TLM vCenter IP
- TLM vCenter Creds
- Cluster
- Datastore
- VM Names
- VM Settings
- Network
Tasks:
- Deploy VPC vCloud Cells
- Deploy VPC vCloud NFS
- Deploy VPC vCloud DB
-------------- 2 -----------------
Reserve VPC
Engine Broker Plugin:
- Rez
Inputs:
- VPC #
----------- 8 ------------
Configure VPC vCenter
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC VCSA IP Address
- VPC VCSA Credentials
- Licenses
- vCenter
- ESXi
- vCloud Net & Sec
- Retention Policies
- Task: Enabled
- Event: Enabled
- Datacenter Name
- Cluster Settings
- Name
- DRS Settings
- EVC Settings?
- DVS Configuration
- Name: d#p#v#
- Settings
- Uplinks: 4
- MTU: 9000
- Enable NIOC
- Portgroup settings
- Name
- VLAN
- Settings
-Tasks:
- Configure VPC VCSA
- Configure DVS
----------- 7 ------------
Init VPC VSM
Engine Broker Plugin
- ??
Inputs:
- VPC VSM Network Settings
Tasks:
- Init VPC VSM
----------- 6 ------------
VPC VCSA Port Check
Engine Broker Plugin
- Util
Inputs:
- VPC VCSA IP
Tasks:
- VPC VCSA Port Check
----------- 6 ------------
VPC VSM Port Check
Engine Broker Plugin
- Util
Inputs:
- VPC VSM IP
Tasks:
- VPC VSM Port Check
----------- 6 ------------
Verify VPC ESXi
Engine Broker Plugin
- Puppet Util?
Inputs:
- VPC ESXi IP
- VPC ESXii Creds
Tasks:
- Verify VPC ESXi
----------- 9 ------------
Add ESXi to VPC Datacenter
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC ESXi IP
- VPC ESXii Creds
- VPC VCSA IP
- VPC VCSA Creds
- VPC Datacenter Name
Tasks:
- Add ESXi to VPC Datacenter
----------- 10 ------------
Configure VPC ESXi
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC VCSA IP
- VPC VCSA Creds
- DNS Settings
- NTP Settings
- Syslog Settings
- VMK Configuration
- vMotion
- Storage
- FT
- ???
Tasks:
- Configure ESXi
- Add to DVS
- Add VMK Networking
----------- 11 ------------
Add Storage to VPC ESXi
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC VCSA IP
- VPC VCSA Creds
- iSCSI Settings
- LUN Information
Tasks:
- Configure iSCSI
- Add LUNs
------------- 3 ---------------
Configure VNX VPC Storage Group
Engine Broker Plugin:
- Storage Controller
Inputs:
- VPC #
- LUN IDs
- ESXi iSCSI information
Tasks:
- Create VPC Storage Group
- Configure VNX Host registrations
- Add LUNs to VPC Storage Group
----------- 6 ------------
VPC vCloud Cell Port Check
Engine Broker Plugin
- Util
Inputs:
- VPC vCloud Cell IP
Tasks:
- VPC vCloud Cell Port Check
----------- 6 ------------
VPC vCloud Cell Port Check
Engine Broker Plugin
- Util
Inputs:
- VPC vCloud Cell IP
Tasks:
- VPC vCloud Cell Port Check
----------- 6 ------------
VPC vCloud NFS Port Check
Engine Broker Plugin
- Util
Inputs:
- VPC vCloud Cell IP
Tasks:
- VPC vCloud NFS Port Check
----------- 6 ------------
VPC vCloud DB Port Check
Engine Broker Plugin
- Util
Inputs:
- VPC vCloud Cell IP
Tasks:
- VPC vCloud DB Port Check
----------- 7 ------------
Configure VPC vCloud NFS
Engine Broker Plugin
- ??
Inputs:
- VPC vCloud NFS IP
- VPC vCloud NFS Creds
- NFS Export Settings
- ???
Tasks:
- Config VPC vCloud NFS
----------- 7 ------------
Configure VPC vCloud DB
Engine Broker Plugin
- ??
Inputs:
- VPC vCloud DB IP
- VPC vCloud DB Creds
- vCloud Database Config
- ???
Tasks:
- Configure VPC vCloud DB
----------- 8 ------------
Configure VPC vCloud Cell
Engine Broker Plugin
- ??
Inputs:
- VPC vCloud Cell IP
- VPC vCloud Cell Creds
- VPC vCloud NFS Config
- VPC vCloud Installation
Responses.properties
- VPC vCloud Cell Cert
- NTP Settings
Tasks:
- Install and Configure NTP
- Install VPC vCloud Cert
- Configure vCD
- Configure vCD Transfer Service
----------- 9 ------------
Configure VPC vCloud Cell
Engine Broker Plugin
- ??
Inputs:
- VPC vCloud Cell IP
- VPC vCloud Cell Creds
- VPC vCloud Installation
Responses.properties
- VPC vCloud NFS Config
- VPC vCloud Cell Cert
- NTP Settings
Tasks:
- Install and Configure NTP
- Install VPC vCloud Cert
- Configure vCD
- Configure vCD Transfer Service
----------- 12 ------------
Create VPC VM Storage Profile
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC VCSA IP
- VPC VCSA Creds
- Storage Profile Name
Tasks:
- Create VPC VM Storage Profile
- Tag VPC Datastores
----------- 13 ------------
Add VPC vCenter to vCloud
Engine Broker Plugin
- Puppet VMware-vCloud
Inputs:
- VPC vCloud IP
- VPC vCloud Creds
- VPC VCSA IP
- VPC VCSA Creds
- ???
Tasks:
- Add VPC vCenter to VPC vCloud
----------- 10 ------------
Configure VXLAN
Engine Broker Plugin
- Puppet VMware-vShield
Inputs:
- VPC VSM IP
- VPC VSM Creds
- Multicast Information
- VPC DVS Information
- VPC Cluster
- ???
Tasks:
- Configure VXLAN
----------- 11 ------------
Reconfigure VXLAN VMK Ports
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC VCSA IP
- VPC VCSA Creds
- VXLAN VMK Network Settings
Tasks:
- Configure VXLAN VMK Ports
----------- 14 ------------
Configure VPC vCloud
Engine Broker Plugin
- Puppet VMware-vCloud
Inputs:
- VPC vCloud IP
- VPC vCloud Creds
- Provider VDC Name
- External Org Network Name
- ???
Tasks:
- Configure VPC vCloud
----------- 12 ------------
Configure VPC Cluster HA
Engine Broker Plugin
- Puppet VMware-vCenter
Inputs:
- VPC VCSA IP
- VPC VCSA Creds
- VPC ESXi Hosts
- VPC Cluster
- HA Settings
- Failure: percentage
Tasks:
- Add VPC ESXi hosts to Cluster
- Configure VPC Cluster HA Settings
----------- 5 ------------
Prep VPC vCloud Cell
Engine Broker Plugin
- Puppet vCenter
- ??
Inputs:
- VPC VCSA IP
- VPC VCSA Cred
- VPC vCloud Cell Name
- Static Routes
Tasks:
- PowerOn VM
- Set Hostname
- Set Static Routes
- Configure Networking
----------- 5 ------------
Prep VPC VCSA
Engine Broker Plugin
- ??
Inputs:
- Credentials
- Network Settings
Tasks:
- Configure Networking
----------- 5 ------------
Prep VPC vCloud Cell
Engine Broker Plugin
- Puppet vCenter
- ??
Inputs:
- VPC VCSA IP
- VPC VCSA Cred
- VPC vCloud Cell Name
- Static Routes
Tasks:
- PowerOn VM
- Set Hostname
- Set Static Routes
- Configure Networking
----------- 5 ------------
Prep VPC VSM
Engine Broker Plugin
- ??
Inputs:
- VPC VCSA IP
- VPC VCSA Cred
- VPC VSM name
- Network Settings
Tasks:
- PowerOn VM
- Configure Networking

47. 47
Zombie Engine DSL - ZED
Puppet
Apply
MCollective
MCollective
OVFTool
Puppet
Apply
MCollective
API Query

48. 48
Zombie Engine DSL - ZED
Reserve Compute
Install ESXi
Add To Cluster
Add Compute
Reserve Compute
Install ESXi
Add To ClusterReplace Compute
Un-Reserve Compute
Remove From Cluster

49. 49
Distributed and location-aware execution
Work to do
Concurrent
WorkConcurrent
WorkConcurrent
Work
Concurrent
WorkConcurrent
WorkConcurrent
Work
P
P
P
B
B B
BB
Datacenter: Dallas
Datacenter: Singapore
Work to do

50. 50
Why we use Puppet
• Greater investment in building modules than scripting
• Much greater return on investment with evolution of the product
• 13 in-house Puppet Modules so far (small example)
• vCloud Director
• vShield Networking
• vSphere
• We use a total of 47 modules for everything
• We have Puppet modules for installing Zombie in
Production, Integration, and Development (including Vagrant + Puppet use
for laptops)
• Project Zombie itself uses Puppet to do work, and is installed and
upgraded using Puppet

51. 51
Why we use Puppet
Puppet 1st Rule

52. 52
Cool things we do with Puppet
• Engine can stage changes using MCollective to an environment
• We can update that environment throughout job execution
• We can apply the environment against a target at any time
• We have the ability to load balance/scale both Engine execution and
Puppet execution
• Processes that would be complex to order in Puppet can be staged much
easier with Zombie Engine + Puppet
• ZED allows for iteration using configuration data into execution
concurrency and queuing. Very cool layer on top of Puppet.

53. 53
Does it work?

54. 54

55. 55
Cloud Build A
Maximum capacity per week:
Man hours: 72
2
Time to delivery: 6 days

56. 56
Maximum capacity per week:
Man hours: 1.5
336 x Zombie Engine instance count
Time to delivery: 2.5 hours*
Cloud Build A

57. 57
Details (per job):
• ~120 individual tasks (plugin calls)
• ~2700 discrete configuration points
• ~1400 managed resources
• Dynamically sized (pick the # of compute and storage)
• Controls: vCloud Director, vCenter, ESXi, EMC
VNX, Razor, vShield Manager, vShield Edge, Linux and
Windows operating systems.
Cloud Build A

58. 58
Cloud Build A

59. 59
So yeah… it works

60. 60
What’s next?

61. 61
Bring Your Own Licenses
Bring Your Own VMs
Bring Your Own Tools
Customer
Components
VMware vCloud
Hybrid Service
Hybrid Service Delivery Components
Infrastructure Hardware & Facilities
Compute Storage
Networking &
Security
OS Catalogs
Application Catalogs
Infrastructure Management
Web Console vCloud API