Len Bellemore from ControlCircle shares his exeriences with CloudStack

1. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
Len Bellemore – Cloud Product Development
4th July 2013
Securing the CloudStack
Management Domain

2. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
Agenda
• Who are ControlCircle
• The design goals
• Network design
• Challenges faced
• Example use case testing
• How did we trouble shoot
• Lessons learned

3. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
• Our focus is delivering mission critical IT
services that are of strategic importance to our
customers - underpinning their business
growth and competitiveness
• We specialize in hybrid capabilities – from
colocation to cloud - aligning our customers’
requirements to the right solution and ensuring
they maintain full visibility and control through
our Max3000™ monitoring and management
platform
Introduction to ControlCircle

4. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
Design Goals
• Security
– Security in depth, not just at the edge
– Containment of any attack
– Identification of any attack
• Traffic Segmentation
– Reduce size of broadcast domains
– Use of high capacity links for storage
• Scale
– Easy addition of physical sites/locations
– Distribute the controller servers across multiple sites

5. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
CloudStack Network Design
Standard design
• Single management
network
– CS Management
Servers
– CS MySQL DB
– Hypervisors
– SSVM
– Console Proxy
• Guest
• Public
• Storage
– NFS/iSCSI Device
What we designed
• DMZ
– Load Balancers
– Web Servers
– Internet Proxy
• Control
– CS Management
Servers
– vCenter
– Windows Domain
Controllers
– DNS Servers (non-
Guest)
• Management
– Hypervisors
– SSVM, Console Proxy
• Guest
• Public
• Database
– CS MySQL DB
• Storage
– NFS/iSCSI
Devices

6. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
High Level Network Diagram
Zone 1
Guest VMs
Guest
Virtual
Routers
Public 1
Secondary
Primary
Storage 1
CS Management vCenterDNS
ProxyWeb Servers
Guest VM Hosts
Management 1
SSVM Console Proxy
MySQL
Guest 1
DMZ
Control
Database
Control VM
Hosts
Control Segment 1
Zone 2
Guest VMs
Guest
Virtual
Routers
Public 2
Secondary
Primary
Storage 2
CS Management vCenterDNS
ProxyWeb Servers
Guest VM Hosts
Management 2
SSVM Console Proxy
MySQL
Guest 2
DMZ
Control
Database
Control VM
Hosts
Control Segment 2
Load Balancers Load Balancers
Control Segment
traffic is routed
between
physical sites

7. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
Challenges we faced
• Traffic flows difficult to ascertain
• Logging detailed but ambiguous
• Documentation scattered across internet
• CloudStack documents only cover the standard single management
network design
• Testing had to be use case led – what are the use cases?

8. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
Example use case
ISO Upload
• Copy ISO to a Web Server
• Register ISO and input URL to ISO
• SSVM Downloads ISO
• SSVM copies ISO to Secondary Storage
• Finished!

9. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
ISO Upload
Test outcome
• Copy ISO to a Web Server
• Register ISO and input URL to ISO
• GUI error
• CS Management Server log file reports name resolution failure
• Troubleshooting is focused on CS Mgmt Server, no luck
• Proxy? DNS Servers? Firewall Rules?
• Confusion…..
• http://www.cloudstack-china.org/wp-content/uploads/downloads/2012/12/ccc-
cloudstacknetworking.pptx

10. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential

11. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
ISO Upload
Final resolution
• Focus back to SSVM
• Realised that DNS was not resolving. Why?
• Open firewall rules for DNS to correct IP
• Retry ISO Upload
• Presto!

12. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
How did we Troubleshoot?
• CS Management Server Logs
• SSVM Logs
• Firewall Logs
• Work closely with the Network team
• users@cloudstack.apache.org mailing list

13. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
Lessons Learned
Debugging and fault finding
• There is no set way of doing
things in CloudStack world
• Official installation guides are
written for simple networks
• Documentation is scattered,
blogs and unofficial sites are
the best source of info
• Logging is your friend, albeit a
troubled one!
• The community is your best
troubleshooting tool
Designing the solution
• Consider agility when
designing your network
• Firewall rules based on
single IP’s is not achievable
– you need host and object
groups
• Understand the use cases
prior to testing
• Document firewall rules
before you start building
based on use cases
• Document as you go

14. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
Documentation and Resources
• http://www.slideshare.net/cloudstack/
• http://www.shapeblue.com/
• http://www.buildacloud.org/
• http://markmail.org/
• http://cloudstack.apache.org/
• http://support.citrix.com/
• http://support.citrix.com/servlet/KbServlet/download/2389-
102-654859/CitrixPorts_by_Port_1103.pdf

15. © Copyright ControlCircle 2013. All rights reserved. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the UK
and other countries. ControlCircle products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of ControlCircle in the United States and in other countries.
Confidential
Any Questions
?