4. Cloud Computing
Virtualization is not Cloud computing
Server Virtualization++ Cloud
Built for traditional enterprise Designed around big data,
apps & client-server compute massive scale & next-gen apps
• Enterprise arch for 100s of hosts • Cloud arch for 1000s of hosts
• Scale-up (pool-based resourcing) • Scale-out (horizontal resourcing)
• IT management-centric • Autonomic management
• 1 administrator for Dozens of servers • 1 administrator for 1,000’s of servers
• Apps assume reliability • Apps assume failure
• Proprietary vendor stack • Open, value-added stack
5. Cloud Computing (contd..)
• Tenets of Cloud
o Shared infrastructure and Multi-tenancy
o Self Service
o Elasticity
o Built for massive Scale
o Service agility
o Pay-as-you-go
o APIs and Extreme Automation
• IAAS/PAAS/SAAS
• Public/Private/Hybrid clouds
6. What is Apache CloudStack
• Turnkey orchestration platform for delivering IAAS clouds
o Secure, multi-tenant
o Self-service
o Service agility and elasticity
o Built for large scale
o Pay-as-you-go
• Deploys on premise (private) or as a hosted (public) cloud
• Can be used for hybrid clouds
• built in java, provides native REST API’s and EC2 API
• Has python, Ruby clients and CLI as well
7. A
bit
of
History
• Original
company
Cloud.com
(2008)
• Open
source
(GPLv3)
as
CloudStack
(2010)
• Acquired
by
Citrix
(July
2011)
• Relicensed
under
ASL
v2
April
3,
2012
• Accepted
as
Apache
IncubaKng
Project
April
16,
2012
• First
Apache
(ACS
4.0)
released
• Many
non-‐Citrix
contributors,
commiRers,
PPMC
members
8. Who is contributing
• Sungard: Unit test cases
• Carnigo: Object store plug-in
• Ceph/Rbd support by Wido
• CLVM/KVM by Marcus
• Nicira NVP: Schuberg Philis
• Basho: Object Store
• Brocade ADX ADC support
• Midokura midonet SDN controller integration
9. How to contribute
• Its not just about code! As community member
you can engage in
o Discussions: Design, Use Case, deployment
issues
o Bug reporting, feature requests
o Code reviews
o Build, tools, infrastructure
o Helping out on the IRC
o Documentation
o Submit bug fixes, features
10. How to contribute (contd..)
• Git repo, bug tracker, wiki are on ASF infra
• Project website
o http://incubator.apache.org/cloudstack/
o http://www.cloudstack.org
• IRC
o #cloudstack on irc.freenode.net
o Wednesday - 10:30 PM IST, 5:00 UTC
• Mailing lists (cloudstack.org/discuss/mailing-
lists.html)
o cloudstack-dev-subscribe@incubator.apache.org
o cloudstack-users-subscribe@incubator.apache.org
• http://www.slideshare.net/cloudstack
11. On-demand infrastructure as a service
Org A
Org B
Admin
Admin
Users
Users
Cloud
Admin
End User
Provision Consume
resources resources
UI Cli EC2 CloudStack managed cloud
REST API Compute Network Storage
CloudStack
Management Server
manage
resources
12. Core CloudStack Components
VM
• Hosts
• Servers onto which services will be provisioned Host
VM
Network
• Primary Storage
Host
• VM storage
• Cluster Primary
• A grouping of hosts and their associated storage Storage
• Pod
• Collection of clusters Cluster
• Network Secondary
Storage Cluster
• Logical network associated with service offerings
• Secondary Storage
• Template, snapshot and ISO storage CloudStack Pod
• Zone
• Collection of pods, network offerings and secondary
CloudStack Pod
storage
• Management Server Farm
Zone
• Responsible for all management and provisioning
tasks
13. CloudStack Deployment Architecture
CloudStack
Management
Internet Ø Hypervisor is the basic unit
Server of scale.
Zone 1 Ø Cluster consists of one ore
more hosts of same
L3 core hypervisor
Ø All hosts in cluster have
Pod 1 Access Layer Pod N access to shared (primary)
Secondary storage
…. Storage
Cluster N Ø Pod is one or more clusters,
usually with L2 switches.
…. Ø Availability Zone has one or
more pods, has access to
Cluster 1 secondary storage.
Host 1 Ø One or more zones
Primary represent cloud
Storage
Host 2
14. CloudStack Managing Multiple Zones
Data Center 1 Data Center 2 Ø Single Management Server can
Data Center 2
Management Data Center 3 manage multiple zones
Server Zone
Zone Ø Zones can be geographically
2 distributed but low latency links
2 Zone
Zone are expected for better
Zone1 Zone
3 performance
4 3
Ø Single MS node can manage up
to 5K hosts.
Data Center 2 Ø Multiple MS nodes can be
Data Center 2 deployed as cluster for scale or
Data Center 2
redundancy
Zone
Zone
2ZoneZone
2
2 Zone
3 Zone
3
3
18. Create Virtual Machines via Offerings
Select Operating System
• Windows, Linux
Select Compute Offering
• CPU & RAM
Select Disk Offering
• Volume Size
Select Network Offering
• Network & Services
Create VM
19. Virtual Machine Management
Users
Change
VM Operations Console Access VM Status
Service Offering
Start
• CPU Utilized 2 CPUs 4 CPUs
Stop 1 GB 4 GB
• Network Read RAM RAM
Restart • Network Writes 20 GB 200 GB
Destroy 20 100
Mbps Mbps
20. Volume & Snapshot Management
VM 1
Add / Delete
Volumes Volume
Create Templates Volume Template
from Volumes
Hourly Weekly
Schedule Now
Snapshots Daily Monthly
….
View Snapshot
History
21. A
Very
Flexible
IaaS
Pla5orm
Compute Hypervisor
XenServer VMware Oracle VM KVM Bare metal
Storage Block & Object
Fiber
Local Disk iSCSI NFS Swift Ceph Riak
Channel
Primary
Storage
Secondary
Storage
Network Network & Network Services
Load
Network Type Isolation Firewall VPN
balancer
22. CloudStack Storage
Primary Storage
• Configured at Cluster-level. Close to hosts for
better performance
• Stores all disk volumes for VMs in a cluster L3 switch
• Cluster can have one or more primary storages
L2 switch
Pod 1
• Local disk, iSCSI, FC or NFS
Cluster 1 Secondary
Storage
Secondary Storage Host 1
Local
• Configured at Zone-level storage Primary
Storage
• Stores all Templates, ISOs and Snapshots Host 2
• Zone can have one or more secondary storages
• NFS, OpenStack Swift Availability zone
Local Storage
• Storage available on hypervisor hist
23. Role of Storage and Templates
• Primary Storage
• Cluster level storage for VMs Host
• Connected directly to hosts
• NFS, iSCSI, FC and Local Host
• Secondary Storage Primary Storage
• Zone level storage for template, ISOs and Cluster
snapshots
• NFS or OpenStack Swift via CloudStack Pod
System VM
• Templates and ISOs
• Imported into CloudStack
• Can be private or public Secondary Storage
Zone
Template
24. Provisioning Process
1. User Requests Instance VM
2. Provision Optional Network Host
Services
Host
3. Copy instance template from Primary Storage
secondary storage to primary Cluster
storage on appropriate cluster
Pod
4. Create any requested data
volumes on primary storage for the
Template
cluster
5. Create instance
Secondary Storage
6. Start instance Zone
25. Object Store
CloudStack Mgmt • Object store used to store
Server templates and snapshots
• VM’s can be distributed
across the availability
zones
• For DR create instances
Availability
Zone
Availability
Zone
Availability
Zone
in different zones
Object Storage
26. Multi-tenancy & Account Management
Resources
Domain
VMs, IPs, Snapshots…
Domain is a unit of isolation
Org A
that represents a customer
Admin
org, business unit or a
reseller
Domain
Reseller A
Domain can have arbitrary
Admin
Sub-Domain
Resources levels of sub-domains
VMs, IPs, Snapshots…
Org C
A Domain can have one or
Admin
more accounts
Account
Group A
An Account represents one
or more users and is the
Account
basic unit of isolation
Group B
Admin can limit resources at
User 1
the Account or Domain
levels
User 2
27. User Dashboard: Consumed Resources
• Running, Stopped &
Total VMs
• Public IPs
• Private networks
• Latest Events
28. Admin Dashboard:
Consumed Resources
• Provides zone wide
resource
consumption
• Also provides latest
alerts and events
29. Edge services with System VMs
• System VMs optimize and scale the datapath on behalf of CloudStack
o Stateless, can be destroyed and recreated from database state
o Highly Available
o Communicates with Management Server over management network
o Usually have 3 interfaces: control, guest and public
• Console Proxy VM
o Provides AJAX-style HTTP-only console viewer
o Grabs VNC output from hypervisor
o Scales out (more spawned) as load increases
o Java-based server Communicates with MS over message bus
• Secondary Storage VM
o Provides image (template) management services
o Download from HTTP file share or Swift
o Copy between zones
o Scale out to handle multiple NFS mounts
o Java-based server communicates with MS over message bus
30. Edge services with System VMs (contd.)
• Virtual Router VM
o Provides multiple network services
o IPAM (DHCP), DNS, NAT, Source NAT, Firewall, PF, VPN
o User-data, Meta-data, SSH keys and password change server
o Redundancy via VRRP
o MS configures VR over SSH
§ Proxied via the hypervisor on XS and KVM
31. Network & Network Services
• Create Networks and attach VMs
• Acquire public IP address for
NAT & load balancing
• Control traffic to VM using
ingress and egress firewall rules
• Set up rules to load balance
traffic between VMs
32. Networking feature overview
• Orchestration of L2 – L7 network services
o IPAM, DNS, Gateway, Firewall, NAT, LB, VPN, etc
• Mix-and-match services and providers
• Out-of-the-box integration with automated deployment of virtual
routers
o Highly available network services using CloudStack HA and VRRP
• Orchestrate external providers such as hardware firewalls and
load balancers
o Devices can provide multiple services
o Admin API to configure external devices
o Plugin-based extensions for network behavior and admin API extensions
• Multiple multi-tenancy [network isolation] options
• Integrated traffic accounting
• Access control
• Software Defined Networking (Nicira NVP)
33. L2 Features
• Choice of network isolation
o Physical, VLAN, L3 (anti-spoof), Overlay[GRE]
o Physical isolation through network labels [limited to # of
nics or bonds]
• Multi-nic
o Deploy instance in multiple networks
o Control default route
• Access control
o Shared networks, project networks
• QoS [max rate]
• Traffic monitoring
• Hot-plug / detach of nics
34. L3 Features
• IPAM [DHCP], Public IP address management
o VR acts as DHCP server
o Can request multiple public IPs per tenant
• Gateway (default gateway)
o Redundant VR (using VRRP)
o Inter-subnet routing
o Static routing control
• Remote Access VPN
o L2TP over IPSec using PSK
o Virtual Router only
• Firewall based on source cidr
• Static NAT [1:1]
o Including “Elastic IP” in Basic Zone
• Source NAT
o Per-network, or interface NAT
• Public Traffic usage
o Monitoring on the Virtual Router / External network device
o Integration with sFlow collectors
• Site-to-Site VPN
o IPSec VPN based on VR
• L3 ACLs
35. L4 Features
• Security groups for L3-isolation
o “Basic Zone” in docs
o Default AWS-style networking
o Scales much better than VLANs
• Stateful firewall for TCP, UDP and ICMP
• Port forwarding [“Advanced Zone”]
o Conserve public Ips
36. L7 features
• Loadbalancer
o VR has HAProxy built in
o External Loadbalancer support
§ Netscaler (MPX/SDX/VPX)
§ F5 BigIP
§ Can dedicate an LB appliance to an account or share it
among tenants
o Loadbalancer supported with L3-isolation as well
o Stickiness support
o SSL support [future]
o Health Checks [future]
• User-data & meta-data
o Fetched from virtual router
• Password change server
37. CloudStack Terminology
• Guest network
o The tenant network to which instances are attached
• Storage network
o The physical network which connects the hypervisor to primary storage
• Management network
o Control Plane traffic between CloudStack management server and hypervisor clusters
• Public network
o “Outside” the cloud [usually Internet]
o Shared public VLANs trunked down to all hypervisors
• All traffic can be multiplexed on to the same underlying physical network
using VLANs
o Usually Management network is untagged
o Storage network usually on separate nic (or bond)
• Admin informs CloudStack how to map these network types to the
underlying physical network
o Configure traffic labels on the hypervisor
o Configure traffic labels on Admin UI
38. CloudStack Network Service Providers
• A Network Service Provider is hardware or virtual
appliance that makes a network service possible
in CloudStack ; for example, a Citrix NetScaler
appliance can be installed in the cloud to provide
Load-Balancing services.
• Administrators can have multiple instances of the
same service provider in a network; for example,
more than one Citrix NetScaler or Juniper SRX
device can be added to CloudStack
• CloudStack supports the following Network
Providers:
o CloudStack Virtual Router (default)
o Citrix NetScaler SDX, VPX and MPX models
o Juniper SRX
o F5 BigIP
39. Network Service Providers Matrix
• Network offerings is basically a definition of what Network Services are
available when this offering is used. The available Network Services
are: VPN, DHCP, DNS, Firewall, Load Balancer, User Data, Source
NAT, Static NAT, Port Forwarding and Security Groups*
Feature Virtual Citrix Juniper F5 BigIP
Router NetScaler SRX
Remote Access VPN YES N/A N/A N/A
Firewall YES N/A YES N/A
Source NAT YES N/A YES N/A
Static NAT YES YES YES N/A
Load Balancing YES YES N/A YES
Port Forwarding YES N/A YES N/A
Elastic IP N/A YES N/A N/A
Elastic LB N/A YES N/A N/A
DHCP/DNS/User Data YES N/A N/A N/A
40. Network Offerings
• Cloud provider defines the
feature set for guest networks
• Toggle features or service
levels
o Security groups on/off
o Load balancer on/off
o Load balancer software/hardware
o VPN, firewall, port forwarding
• User chooses network offering
when creating network
• Enables upgrade between
network offerings
• Default offerings built-in
o For classic CloudStack
networking
41. Add Guest Networks
• Choice to choose L3
subnet, default gateway
• Choice of network
offerings
42. Editing Guest Networks
When editing a guest network users can
change the network offering. They can
either upgrade to a “premium” network
offering (for example offering that uses
hardware Load-balancer) or downgrade to a
“cheaper” network.
43. Restarting/Cleaning Up a Guest Network
• Restarting the network will simply
resend all the LB, Firewall and
Port-Forwarding rules to the
network provider
• Restarting the Network with
“Clean up”:
• restarKng
network
elements
-‐
virtual
routers,
DHCP
servers
• If
virtual
router
is
used,
it
will
be
destroyed
and
recreated
• Reapplying
all
public
IPs
to
the
network
provider
• Reapplying
load-‐Balancing/Port-‐Forwarding/Firewall
rules
44. Deleting a Guest Network
• An Isolated Guest Network can only be deleted if no VMs
are using these network (e.g. Completely destroyed and
expunged)
• Deleting a Network will Destroy the Virtual Router (if used)
and will release the Public IPs back to the IP Pool
45. Basic vs Advanced Networking
• Segmentation based on feature set and ease-of-
deployment
• Both are feature-rich
• Basic implements true AWS-style L3-isolation
o Tenants do not get contiguous IP addresses or subnets
o Network segmentation based on Security Groups
o Tremendous scale (tens of thousands)
• Advanced Zone offers full L3 subnets and L2
isolation
o VLANs are default implementation (4K limit)
o More features (source NAT, PF, LB, VPN)
46. Physical Network in Zone
Core (L3) Network
Pod 1 Pod 2 Pod N
Cloudstack
Access
Switch(es)
Server
Cloudstack
Servers
CLUSTER 1 …
Hypervisor
1
VM Traffic …
Hypervisor
8
Control Plane Traffic
Storage Traffic Storage 2
Storage 1
Public Traffic
…
CLUSTER 4
Hypervisor
N
Hypervisor
N+1
Storage k
47. Layer 3 cloud networking
Web DB Web
VM VM VM
Web DB
Security Security
Group Group
Web Web DB
VM VM VM
… … …
Web Web
VM VM
48. Guest Networks with L3 isolation
Public
Public
IP
Guest
Guest
Internet address
1
VM
1 address
65.37.141.11
10.1.0.2
10.1.0.1 Guest
65.37.141.24
Pod
1
L2
Guest
65.37.141.36
Switch 2
VM
1 address
65.37.141.80
10.1.0.3
Guest
Guest
1
VM
2 address
L3
Core
Switch
Pod
2
L2
Switch
10.1.8.1
… 10.1.0.4
Guest
Guest
10.1.16. 2
VM
2 address
Load
Pod
3
L2
Balancer 1 10.1.16.12
Switch
Guest
2
VM
3 Guest
address
10.1.16.21
… Guest
1
VM
3
Guest
address
10.1.16.47
Guest
Guest
1
VM
4
address
10.1.16.85
49. Guest Networks with L2 isolation
Core (L3) Network
Pod K Pod M Pod N
Access
Switch(es)
V V
Hypervisor
R V
Hypervisor
CLUSTER 1 …
Hypervisor
1
R
VM Traffic …
Hypervisor
8
Public Traffic
…
CLUSTER 4
V V
Hypervisor
N
V Tenant VM
Hypervisor
N+1
V
R Tenant Virtual Router
50. L2 isolation: VLAN networking
User
1
User
1 User
User 1
2
User
1
User
User
2
1
User
2
… … …
User
1
51. SDN at Work
CloudStack Mgmt Server
SDN
Controlle
r
Host 1 OVS Host 3
OVS
VM VM VM V
1 1 3 R
GRE Tunnel GRE Tunnel
Host 2 OVS Host 4
OVS
VM VM VM V
2 2 3 R
GRE Tunnel GRE Tunnel
52. Guest virtual layer-2 network
Guest
Virtual
Network
10.1.1.0/24
Public
Public
IP
Guest
Gateway
Guest
Network address
1
VM
1
address
address
65.37.141.11
10.1.1.1 10.1.1.2
65.37.141.36
Guest
1
Guest
Guest
Public
Virtual
1
VM
2 address
Internet Router 10.1.1.3
NAT
Guest
Guest
DHCP
1
VM
3 address
Load
10.1.1.4
Balancing
Guest
Guest
VPN 1
VM
4 address
10.1.1.5
Guest
Virtual
Network
Public
IP
10.1.1.0/24
address
Gateway
Guest
Guest
65.37.141.24
address
2
VM
1 address
65.37.141.80 10.1.1.1 10.1.1.2
Guest
2
Guest
Guest
Virtual
2
VM
2 address
Router 10.1.1.3
NAT
Guest
Guest
DHCP
2
VM
3 address
Load
10.1.1.4
Balancing
VPN
53. Layer-2 Guest Virtual Network
CS Virtual Router provides Network Services External Devices provide Network Services
Guest
Virtual
Network
10.1.1.1/8
Guest
Virtual
Network
10.1.1.1/8
VLAN
100 VLAN
100
Public
Public
Network/ Network/
Internet Guest Internet Guest
Public
IP
Private
IP
10.1.1.1
10.1.1.1 VM 1 10.1.1.111 VM 1
Gateway
65.37.141.111 Juniper
Public
IP
SRX
address
65.37.141.11 CS Firewall
10.1.1.1 Guest Guest
Virtual
10.1.1.3 VM 2 10.1.1.3 VM 2
Router
Public
IP
Private
IP
DHCP,
DNS
65.37.141. NetScaler 10.1.1.112
NAT
Guest 112 Load Guest
Load
Balancing
10.1.1.4 VM 3 Blancer VM 3
10.1.1.4
VPN
Guest Guest
10.1.1.5 VM 4 10.1.1.5 VM 4
CS
DHCP,
Virtual
Router
DNS
54. Layer-3 Guest Network
Network Services Managed Externally Network Services Managed by CS
Public
Network
65.11.0.0/16
Security
Group
Security
Group
Public
Network/ 1
1
Internet 10.1.2.3
65.11.1.2 Guest
Guest
VM 1
VM 1
10.2.12.4
65.11.1.3 65.11.1.2 NetScaler
L3
Guest
Guest
65.11.1.3 Load
switch
VM 2
VM 2 Blancer
65.11.1.4
EIP,
ELB
10.5.2.99
65.11.1.4 Guest
Guest
VM 3
VM 3
10.1.2.18
65.11.1.5 Guest
Guest
VM 4
VM 4
CS CS
Virtual DHCP,
Virtual Security
Group
DHCP,
Route Security
Group
Router
DNS
2
DNS
r 2
55. Multi-tier network
Internet
IPSec or SSL site-to-site VPN
CS Customer
Virtual Router
Loadbalancer Premises
Monitoring VLAN
Virtual Router Services
App VM
• IPAM 10.1.2.31 1
• DNS 10.1.1.1
Web VM
1
• LB [intra]
• S-2-S VPN App VM
10.1.2.24
• Static Routes Web VM 2
• ACLs 10.1.1.3 2
• NAT, PF
• FW [ingress & egress]
Web VM DB VM
• BGP 10.1.1.4 3 10.1.3.24 1
Web VM
10.1.1.5 4
Virtual
Network
Virtual
Network
Virtual
Network
10.1.1.0/24
10.1.2.0/24
10.1.3.0/24
VLAN
100 VLAN
1001 VLAN
141
57. Problem Definition
• Offer a scalable, flexible, manageable IAAS platform
that orchestrate physical and virtual resources to
offer self-service infrastructure provisioning and
monitoring
• Flexible
o Handle new physical resource types
§ Hypervisors, storage, networking
o Add new APIs
o Add new services
o Add new networking models
58. Problem Definition (contd..)
• Manageable
o Hide complexity of underlying resources
o Rich functional end-user and admin UI
o Admin API to automate operations
o Easy install, upgrade for small -> large clouds
o Simple scaling, automated resilience
• Scalable architecture
o 1 -> N hypervisors / VMs / virtual resources
o 1 -> N end users
59. Problem Definition (contd..)
• Resource Allocation
o Hypervisor CPU, Memory
o Storage space
o Avoid set of pods, clusters, hosts
• Capacity scanning
o Snapshot of resources consumed
o Trigger capacity threshold violations
• Garbage collection
o Network resources (IP, VLAN, CIDR etc)
o Compute (VM, CPU, memory)
o Storage (volumes)
• Synchronizing the resource states
• Infrastructure resource failures
• Fencing
60. Scaling: Horizontal Scaling
Single-node Multi-node
Deployment Deployment
Manage
ment
Server
User API User API
Manage Manage
ment MySQL Load
ment
Server DB Balancer
Server
Admin API Admin API
Manage MySQL
ment DB
Server
Back Up
DB
Replication
Ø MS is stateless. MS can be
deployed as physical server or
VM
Infrastructure Infrastructure
Ø Single MS node can manage up
Resources Resources
to 10K hosts. Multiple nodes can
be deployed for scale or
redundancy
61. Resource Load Balancing
• As management server is added into the cluster, resources are
rebalanced seamlessly.
o MS2 signals to MS1 to hand over a resource
o MS1 wait for the commands on the resources to finish
o MS1 holds further commands in a queue
o MS1 signals to MS2 to take over
o MS2 connects
o MS2 signals to MS1 to complete transfer
o MS1 discards its resource and flows the commands being held to MS2
• Listeners are provided to business logic to listen on connection
status and adjusts work based on who’s connected.
• By only working on resources that are connected to the
management server the process is on, work is auto-balanced
between management servers.
• Also reduces the message routing between the management
servers.
62. Cloud Other
CLI UI Clients
Portal
Management Server
REST API
End User Other Pluggable Service API
OAM&P API EC2 API
API APIs Engine
Console Proxy ACL & Authentication Security Adapters
Management - Accounts, Domains, and Projects
- ACL, limits checking Account Management
Connectors
Template Services API
Access
Plugin API
Deployment Planning
HA
Kernel Job
Services API
- Drives long running VM Network Configurations Queue
Usage operations
Calculations - Syncs between resources
managed and DB Network Elements
Additional - Generates events
Services
Hypervisor Gurus
Cluster Resource
Job Alert & Event Database
Managemen Managemen
Management Management Access
DB
t t
Event Bus
Message Bus
Hypervisor Network Storage Image Snapshot
Resources Resources Resources Resources Resources
63. Interactions
OVM Cluster Primary
Storage
vcenter
Monitoring Primary
CS API vSphere Cluster
Storage
End
User
UI Primary
XS Cluster Storage
Admin
UI Clustered
CloudStack XAPI
Domai CS Admin & CloudStack
CloudStack
n End-user API Primary
Admin Management JSON KVM Cluster Storage
UI Server
NetConf
Juniper SRX
Cloud user Nitro API
{API client (Fog/etc)} VNC
JSON
ec2 API JSON Netscaler
Cloud user Console
Console
{ec2 API client } Proxy VM
Proxy VM NFS
MySQL Server
Sec.
{Proxied} SSH Sec.
Storage NFS NFS
Storage
VM
Ajax HTTPS VM
Console
Router VM HTTP (Template Download)
Router VM HTTP (Template Copy)
Router VM
Cloud user HTTP (Swift)
65. Balancing Incoming Requests
• Each management server has two worker thread pools for
incoming requests: effectively two servers in one.
o Executor threads provided by tomcat
o Job threads waiting on job queue
• All incoming requests that requires mostly DB operations are
short in duration and are executed by executor threads because
incoming requests are already load balanced by the load
balancer
• All incoming requests needing resources, which often have long
running durations, are checked against ACL by the executor
threads and then queued and picked up by job threads.
• # of job threads are scaled to the # of DB connections available
to the management server
• Requests may take a long time depending on the constraint of
the resources but they don’t fail.
66. Inside a Management Server
CloudStack
API
API
Servlet
Plugins
Commands cmd.execute() Plugins
Plugins
Async
Job
Queue Services Agent API
Kernel (Cmds)
Mgr API
Responses
Mess
age Resources
Bus Local
Or
Remote
Agent
Manager
Hypervisor Network
Native Device
APIs API
MySQL
67. CloudStack API Sync/Async commands
• Package and Location
cloudstack-oss/api/src/com/cloud/api/…
• BaseCmd (base class)
All commands descend from the BaseCmd base class
68. CloudStack API
Configuration
Commands are configured in cloudstack-oss/client/command.properties.in
Format:
<command name>=<java classname>;<ACL>
*note* ACL is calculated as a bitmap with the following, 1 = ADMIN, 2 =
RESOURCE_DOMAIN_ADMIN, 4 = DOMAIN_ADMIN, 8 = USER
Example:
### snapshot commands!
createSnapshot=com.cloud.api.commands.CreateSnapshotCmd;15!
listSnapshots=com.cloud.api.commands.ListSnapshotsCmd;15!
deleteSnapshot=com.cloud.api.commands.DeleteSnapshotCmd;15!
createSnapshotPolicy=com.cloud.api.commands.CreateSnapshotPoli
cyCmd;15!
deleteSnapshotPolicies=com.cloud.api.commands.DeleteSnapshotPo
liciesCmd;15!
listSnapshotPolicies=com.cloud.api.commands.ListSnapshotPolici
esCmd;15!
69. CloudStack API: adding API
Adding a new command
Determine type of command
Synchronous
Synchronous List Based
Asynchronous
Asynchronous Create based
Create your command
Define request parameters
Implement the execute() method
Implement an appropriate ResponseObject
Add new command to command.properties.in
70. Management Layer
• Management layer is collection of Managers
o Managers are responsible for directing a specific area of the
cloud
§ Storage Manager
• Manages primary storage server (allocation, life-cycle,
attach, detach, user volumes, life-cycle of the primary
storage server itself)
§ Network Manager
• Manages network configurations, IP Allocations, Port
Forwarding, Load Balancers etc.
§ User Vm Manager
• Manages life-cycle of VMs created in the cloud
§ And many more!!!
• Managers coordinate with each other to achieve a task
71. Management Layer: Adapters
• Modularization and customization within the CloudStack
management server is achieved through the use of the
Adapter framework.
• Each Adapter is uniquely identified by the interface it exposes
and represents the boundary between CloudStack and the
individual component and/or processes that can be configured
into the system
• Adapters provide extensibility and in many cases device
specific implementation details while maintaining a simple and
consistent interface.
72. Management Layer: Adapters
• Adapters are executed as a chain in the order that they are
configured
• Defined in cloudstack-oss/client/tomcatconf/components.xml.in
<adapters key="com.cloud.network.guru.NetworkGuru”>
<adapter name="StorageNetworkGuru” class="com.cloud.network.guru.StorageNetworkGuru"/>
<adapter name="ExternalGuestNetworkGuru" class="com.cloud.network.guru.ExternalGuestNetworkGuru"/>
<adapter name="PublicNetworkGuru" class="com.cloud.network.guru.PublicNetworkGuru"/>
<adapter name="PodBasedNetworkGuru" class="com.cloud.network.guru.PodBasedNetworkGuru"/>
<adapter name="ControlNetworkGuru" class="com.cloud.network.guru.ControlNetworkGuru"/>
<adapter name="DirectNetworkGuru" class="com.cloud.network.guru.DirectNetworkGuru"/>
<adapter name="DirectPodBasedNetworkGuru" class="com.cloud.network.guru.DirectPodBasedNetworkGuru"/>
<adapter name="OvsGuestNetworkGuru" class="com.cloud.network.guru.OvsGuestNetworkGuru"/>
</adapters>
74. Adapters: VM orchestration
• Deployment Planner
o First Fit planner
• Host Allocator
o First Fit
o Random
• Storage Allocator
o First Fit
o Random
75. Adapters: Network Orchestration
• Network Guru (Responsible for L2-L3)
o Design
o Implement
o Allocate
o Release
o Shutdown
e.g. guest network guru, OVS network guru etc
• Network Element (Responsible for L4-L7)
o Implement
o Shutdown
e.g. F5, SRX, NetScaler, Virtual Router
76. Extending CloudStack Networking
2. prepare (Network, Nic, DeployDestination, VmInfo)
1. prepare (part of start vm)
Network PluggableServi
Network Element ce
Manager
Device Configuration
MyDnsDeviceS Admin API (CRUD)
DnsService
ervice
3. addDnsRecord(ip, fqdn)
Demonstrates one way to MyDnsDeviceM MySQL
MyDnsElement
inform an external DNS anager
server when an instance
starts. AgentMana
4.Enqueue AddDnsRecord ger Queue
Classes shaded blue form
a plugin / service bundle
to integrate an external MyDnsDeviceR
DNS server. Clients of the esource
instance can then use
DNS names to access the 5.API call to Dns Device
instance.
77. Sequence Flow for VM Creation
Deployme Server
Job Services User VM VirtualMac Network Storage Network Network Templat nt Resource
Threads API Mgr hine Mgr Mgr Mgr Guru Element e Mgr Planner s
Start VM
Start User
VM
Start
VM
Get a Deployment Plan (Host and StoragePool)
Prepare Nics
Reserve resources for
Nic
Notify that Nic is about to be started in
network
Agent Calls
Prepare Volumes
Prepare template on Primary
Storage
Agent Calls
Agent Start VM
Call
Stores job result
79. Server Resources
Agent • Resources are carried in
service VMs to be in close
Hypervisor Resources network proximity to the
physical resources it manages
Resource API
Network Resources
• Easily scales to utilize the most
abundant resource in data
Storage Resources center (CPU & RAM)
Image & Template • Communicates with
Resources Orchestration Server over
message bus (JSON)
Snapshot Resources
• Can be replicated for fault
tolerance
• Control gateway to resources
within data center
81. Working toward 4.1 release
• 4.1 is next major release
o Moving away from monolithic architecture to loosely
coupled subsystems
o Spring for IOC container and AOP
o Storage subsystem refactoring
o Network subsystem refactoring
o New orchestration engine
o Regions support
85. DevCloud
• Several use cases
o Try CloudStack in an isolated sandbox. Runs within
the appliance
o Develop CloudStack on own machine, build locally
and deploy new version in DevCloud (Build and test)
o Develop and Run locally, use DevCloud as Xen hosts
