Contenu connexe
Similaire à Open stack architecture overview-meetup-6-6_2013 (20)
Open stack architecture overview-meetup-6-6_2013
- 1. © MIRANTIS 2013 PAGE 1© MIRANTIS 2013
Module 1:
OpenStack
Architecture
Overview
Mirantis, 2013
- 3. © MIRANTIS 2013 PAGE 3
Goals
• Understand OpenStack purpose and use cases
Understand the OpenStack ecosystem
• Definition
• History
• Projects
- 4. © MIRANTIS 2013 PAGE 4
Goals
• Understand OpenStack purpose and use cases
Understand the OpenStack ecosystem
• Definition
• History
• Projects
BACKGROUND
- 5. © MIRANTIS 2013 PAGE 5
Goals
• Understand OpenStack purpose and use cases
Understand the OpenStack ecosystem
• Definition
• History
• Projects
• Understand OpenStack architecture
• Logical architecture
• Provision VM request flow
• Components details
BACKGROUND
- 6. © MIRANTIS 2013 PAGE 6
Goals
• Understand OpenStack purpose and use cases
Understand the OpenStack ecosystem
• Definition
• History
• Projects
• Understand OpenStack architecture
• Logical architecture
• Provision VM request flow
• Components details
BACKGROUND
THEORY
- 7. © MIRANTIS 2013 PAGE 7
What is OpenStack?
As described by the OpenStack Foundation:
"Open source software for building
private and public clouds“
- 10. © MIRANTIS 2013 PAGE 10
OpenStack Capabilities
• VMs on demand
• provisioning
• snapshotting
- 11. © MIRANTIS 2013 PAGE 11
OpenStack Capabilities
• VMs on demand
• provisioning
• snapshotting
• Volumes
- 12. © MIRANTIS 2013 PAGE 12
OpenStack Capabilities
• VMs on demand
• provisioning
• snapshotting
• Volumes
• Networks
- 13. © MIRANTIS 2013 PAGE 13
OpenStack Capabilities
• VMs on demand
• provisioning
• snapshotting
• Volumes
• Networks
• Object storage for VM images and arbitrary files
- 14. © MIRANTIS 2013 PAGE 14
OpenStack Capabilities
• VMs on demand
• provisioning
• snapshotting
• Volumes
• Networks
• Object storage for VM images and arbitrary files
• Multi-tenancy
• quotas for different tenants
• user can be associated with multiple tenants
- 15. © MIRANTIS 2013 PAGE 15
OpenStack History
Date Rel Projects Type Note
Jul 2010 N/A PoC
* Rackspace Hosting & NASA
joint launch
Oct 2010 Austin Nova, Swift PoC
Feb 2011 Bexar Nova, Glance, Swift PoC
Apr 2011 Cactus Nova, Glance, Swift PoC
** 6 month development
cycle starts
Sep 2011 Diablo Nova, Glance, Swift Prod
1st
production release
(Cactus) at Internap (10/27)
Apr 2012 Essex
Nova, Glance, Swift, Horizon,
Keystone
Prod
Common web UI and shared
authentication mechanism
added
Sep 2012 Folsom
Nova, Glance, Swift, Horizon,
Keystone, Quantum, Cinder, Oslo
Prod
OpenStack Foundation
Established
Apr 2013 Grizzly
Nova, Glance, Swift, Horizon,
Keystone, Quantum, Cinder, Oslo
Prod
Ceilometer and Heat
integration projects added
Oct 2013 Havana
Nova, Glance, Swift, Horizon,
Keystone, Quantum, Cinder, Oslo,
Heat, Ceilometer
Prod Coming Soon!
** Follows similar Ubuntu 6 month release cycle
* Pre-July 2010 is predicated by Rackspace Cloud Files project (Swift), NASA Nebula project (Nova)
- 16. © MIRANTIS 2013 PAGE 16
OpenStack Grizzly Projects
• Core Projects:
• Nova (Compute Service)
• Glance (Image Service)
• Quantum (Network Service)
• Cinder (Block Storage Service)
• Swift (Object Store Service)
• Common Projects:
• Keystone (Identity Service)
• Horizon (Dashboard)
• Library Projects:
• Oslo (Shared Infrastructure Code)
• Incubated Projects (Coming in Havana)
• Ceilometer (Metering/Monitoring)
• Heat (Orchestration)
- 18. © MIRANTIS 2013 PAGE 18
Each OpenStack Project
• Is also a “top-level” OpenStack component
- 19. © MIRANTIS 2013 PAGE 19
Each OpenStack Project
• Is also a “top-level” OpenStack component
• Has an elected “Project Technical Lead” (PTL)
- 20. © MIRANTIS 2013 PAGE 20
Each OpenStack Project
• Is also a “top-level” OpenStack component
• Has an elected “Project Technical Lead” (PTL)
• Has separate developers and design teams
- 21. © MIRANTIS 2013 PAGE 21
Each OpenStack Project
• Is also a “top-level” OpenStack component
• Has an elected “Project Technical Lead” (PTL)
• Has separate developers and design teams
• Has a well defined public API
• With the exception of Horizon, which is the Web GUI, all
other projects have a RESTfull (JSON/HTTP) API
- 22. © MIRANTIS 2013 PAGE 22
Each OpenStack Project
• Is also a “top-level” OpenStack component
• Has an elected “Project Technical Lead” (PTL)
• Has separate developers and design teams
• Has a well defined public API
• With the exception of Horizon, which is the Web GUI, all
other projects have a RESTfull (JSON/HTTP) API
• Has a separate database and isolated
persistent layer
- 25. © MIRANTIS 2013 PAGE 25
Dev Trends
• Decoupling of features (e.g. Nova-volume
became Cinder)
• API-s to communicate
• Common generic API/Infrastructure (Oslo)
• Backends & drivers (everything’s pluggable)
- 29. © MIRANTIS 2013 PAGE 29
OpenStack Projects Relationships
MySQL
Database
Not a project, but
important to
understand the
relationship
- 30. © MIRANTIS 2013 PAGE 30
OpenStack Projects Relationships
MySQL
Database
Not a project, but
important to
understand the
relationship
RabbitMQ Not a project, but
important to
understand the
relationship
- 31. © MIRANTIS 2013 PAGE 31
Deployment – Pick up What
You Want
• The components can be mixed & matched
• Base:
• Nova
• Keystone
• Dashboard
• Glance
• Mutually exclusive scenarios
• Some components can conflict
- 32. © MIRANTIS 2013 PAGE 32
OpenStack: “Typical” Deployment Topology
Control Cluster
Horizon Keystone
GlanceQuantum
Cinder
Cloud Ctrl.
(nova)
Compute Cluster
compute
node
compute
node
compute
node
compute
node
compute
node
compute
node
Storage Cluster (Swift)
storage
node
storage
node
storage
node
storage
node
storage
node
storage
node
- 33. © MIRANTIS 2013 PAGE 33
OpenStack: “Typical” Deployment Topology
Control Cluster
Horizon Keystone
GlanceQuantum
Cinder
Cloud Ctrl.
(nova)
Compute Cluster
compute
node
compute
node
compute
node
compute
node
compute
node
compute
node
Storage Cluster (Swift)
storage
node
storage
node
storage
node
storage
node
storage
node
storage
node
Heavy CPU and
RAM
- 34. © MIRANTIS 2013 PAGE 34
OpenStack: “Typical” Deployment Topology
Control Cluster
Horizon Keystone
GlanceQuantum
Cinder
Cloud Ctrl.
(nova)
Compute Cluster
compute
node
compute
node
compute
node
compute
node
compute
node
compute
node
Storage Cluster (Swift)
storage
node
storage
node
storage
node
storage
node
storage
node
storage
node
Heavy CPU and
RAM Heavy Disk and I/O
- 35. © MIRANTIS 2013 PAGE 35
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
OpenStack Projects:
Communication Types
HTTP
AMQP
SQL
Native API
iSCSI
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
- 36. © MIRANTIS 2013 PAGE 36
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
OpenStack Projects:
Communication Types
HTTP
AMQP
SQL
Native API
iSCSI
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Every OpenStack service exposes access
to restful API via HTTP
UI: Horizon or CLI
- 37. © MIRANTIS 2013 PAGE 37
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
OpenStack Projects:
Communication Types
HTTP
AMQP
SQL
Native API
iSCSI
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Every OpenStack service exposes access
to restful API via HTTP
UI: Horizon or CLI
Each action treated as distributed
transaction, state built as MQ messages
- 38. © MIRANTIS 2013 PAGE 38
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
OpenStack Projects:
Communication Types
HTTP
AMQP
SQL
Native API
iSCSI
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Every OpenStack service exposes access
to restful API via HTTP
Each service updates it’s own DB with
state information as actions are performed
UI: Horizon or CLI
Each action treated as distributed
transaction, state built as MQ messages
- 39. © MIRANTIS 2013 PAGE 39
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
OpenStack Projects:
Communication Types
HTTP
AMQP
SQL
Native API
iSCSI
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Every OpenStack service exposes access
to restful API via HTTP
Each service updates it’s own DB with
state information as actions are performed
UI: Horizon or CLI
Each action treated as distributed
transaction, state built as MQ messages
Direct
access calls,
ex. Plugins,
NetApp,
Nicira, etc.
- 40. © MIRANTIS 2013 PAGE 40
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
OpenStack Projects:
Communication Types
HTTP
AMQP
SQL
Native API
iSCSI
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Cinder Block
storage
provided as
iSCSI storage
to VMs
Every OpenStack service exposes access
to restful API via HTTP
Each service updates it’s own DB with
state information as actions are performed
UI: Horizon or CLI
Each action treated as distributed
transaction, state built as MQ messages
Direct
access calls,
ex. Plugins,
NetApp,
Nicira, etc.
- 41. © MIRANTIS 2013 PAGE 41
OpenStack REST API
• OpenStack public API is a RESTful API
- 42. © MIRANTIS 2013 PAGE 42
OpenStack REST API
• OpenStack public API is a RESTful API
• REST stands for Representational State Transfer
- 43. © MIRANTIS 2013 PAGE 43
OpenStack REST API
• OpenStack public API is a RESTful API
• REST stands for Representational State Transfer
• REST is a stateless client/server protocol with a uniform
interface for accessing the object model
- 44. © MIRANTIS 2013 PAGE 44
OpenStack REST API
• OpenStack public API is a RESTful API
• REST stands for Representational State Transfer
• REST is a stateless client/server protocol with a uniform
interface for accessing the object model
• OpenStack RESTful API is implemented using HTTP
GET/PUT/POST/DELETE in combination with JSON for
data
- 46. © MIRANTIS 2013 PAGE 46
Part 1 Recap
• OpenStack – open source software for building
IaaS
- 47. © MIRANTIS 2013 PAGE 47
Part 1 Recap
• OpenStack – open source software for building
IaaS
• OpenStack release cycle is every 6 month
- 48. © MIRANTIS 2013 PAGE 48
Part 1 Recap
• OpenStack – open source software for building
IaaS
• OpenStack release cycle is every 6 month
• OpenStack is an umbrella over multiple
independent projects (components)
- 49. © MIRANTIS 2013 PAGE 49
Part 1 Recap
• OpenStack – open source software for building
IaaS
• OpenStack release cycle is every 6 month
• OpenStack is an umbrella over multiple
independent projects (components)
• All OpenStack Components talk RESTful API
- 50. © MIRANTIS 2013 PAGE 50
Part 1 Recap
• OpenStack – open source software for building
IaaS
• OpenStack release cycle is every 6 month
• OpenStack is an umbrella over multiple
independent projects (components)
• All OpenStack Components talk RESTful API
• Most OpenStack Components have dedicated DB
(SQL) and MQ (QP), some talk to 3rd party
components using their native APIs
- 51. © MIRANTIS 2013 PAGE 51
Use case: Provision VM
• Most common and complex process
• Interacts with all OpenStack components
- 52. © MIRANTIS 2013 PAGE 52
Initial State
Assumes Tenant is created,
provisioning quota is available, user
has an access to Horizon/CLICloud Operator, DevOp, etc.
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
- 54. © MIRANTIS 2013 PAGE 54
Step 1: Request Provisioning
– From UI
• Login to Horizon
- 55. © MIRANTIS 2013 PAGE 55
Step 1: Request Provisioning
– From UI
• Login to Horizon
• Specify params of VM
• VM Name
• Image (OS type)
• Flavor (specifies CPU, Memory, Disk)
• Network (required for Folsom or later)
• Optional (SSH Keys, Persistent volumes, comments, etc.)
- 56. © MIRANTIS 2013 PAGE 56
Step 1: Request Provisioning
– From UI
• Login to Horizon
• Specify params of VM
• VM Name
• Image (OS type)
• Flavor (specifies CPU, Memory, Disk)
• Network (required for Folsom or later)
• Optional (SSH Keys, Persistent volumes, comments, etc.)
• Hit "Create" button
- 57. © MIRANTIS 2013 PAGE 57
What is Horizon
"The OpenStack Dashboard
(Horizon) provides a
baseline user interface
for managing OpenStack services.“
- 60. © MIRANTIS 2013 PAGE 60
Horizon Notes
• "Stateless“, no DB
• Error handling is delegated to back-end
- 61. © MIRANTIS 2013 PAGE 61
Horizon Notes
• "Stateless“, no DB
• Error handling is delegated to back-end
• Doesn't support all API functions
- 62. © MIRANTIS 2013 PAGE 62
Horizon Notes
• "Stateless“, no DB
• Error handling is delegated to back-end
• Doesn't support all API functions
• Can use memcached or database to store
sessions
- 63. © MIRANTIS 2013 PAGE 63
Horizon Notes
• "Stateless“, no DB
• Error handling is delegated to back-end
• Doesn't support all API functions
• Can use memcached or database to store
sessions
• Gets updated via Nova API polling
- 64. © MIRANTIS 2013 PAGE 64
Horizon Internals
• Subprojects
• Horizon – generic Python Django libraries and
components to work with REST-based back-end / restful
web service
• Openstack Dashboard - web app itself, exposes UI for
OpenStack with styles, locale, etc.
• Dashboard for each component = Individual
nested Django app
• Easily modifiable
• Modularly developed
- 65. © MIRANTIS 2013 PAGE 65
Step 1: Request VM
Provisioning via UI/CLI
Cloud Operator, DevOp, etc.
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
User logs in to UI
Specifies VM params: name,
flavor, keys, etc. and hits
"Create" button
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
- 67. © MIRANTIS 2013 PAGE 67
Step 1: Request Provisioning
– Under the Hood
• Form params are converted to POST data
- 68. © MIRANTIS 2013 PAGE 68
Step 1: Request Provisioning
– Under the Hood
• Form params are converted to POST data
• "Create" request initiate HTTP POST request to
back-end
• To Keystone if auth token is not cached – step 2
- 69. © MIRANTIS 2013 PAGE 69
What is Keystone?
"Keystone provides
Identity, Token, Catalog and
Policy services
for use specifically by projects in the
OpenStack family.“
Keystone was developed by the OpenStack community but is written as a “generic” authentication /
authorization mechanism for any 2 or more restful API services to communicate
- 70. © MIRANTIS 2013 PAGE 70
Keystone Architecture
Deploys with it’s own DB but can
also be integrated with LDAP or
other EAS
Contains user,
role, and
tenant data
Contains
temporary
tokens
Rule management
interface and rule-
based
authorization
Contains
endpoint
registry
- 71. © MIRANTIS 2013 PAGE 71
Keystone Data Model
• User: has account credentials, is associated with one or more
tenants
• Tenant: unit of ownership in OpenStack, contains one or more
users
• Role: a first-class piece of metadata associated with many
user-tenant pairs
• Token: identifying credential associated with a user or user
and tenant
• Extras: bucket of key-value metadata associated with a user-
tenant pair
• Rule: describes a set of requirements for performing an action
- 72. © MIRANTIS 2013 PAGE 72
Keystone Key Concept
• What service exposes
• http://myservice/instances/* - GET/POST/PUT
• http://myservice/images/* - GET
• How RBAC mapping works
• Role X in Tenant Y can do actions A,B,C
• User: GET/POST/PUT on instances, images
• Admin: GET/POST/PUT on tenants, users, quotas
• Each API Service has it’s own RBAC
enforcement through policy files
- 73. © MIRANTIS 2013 PAGE 73
Step 2: Validate Auth Data
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Horizon sends HTTP request to
Keystone. Auth info is specified
in HTTP headers.
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
- 74. © MIRANTIS 2013 PAGE 74
Step 2: Validate Auth Data
• Horizon sends HTTP request to Keystone
• Keystone parses HTTP header info and verifies
that
• The credentials are valid (Authentication)
• User-Tenant-Role mapping is valid (Access Control)
• The requested action is available for this user
(Authorization)
- 75. © MIRANTIS 2013 PAGE 75
Step 2: Validate Auth Data - Success
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Keystone sends temporary token
back to Horizon via HTTP.
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
- 76. © MIRANTIS 2013 PAGE 76
Step 1: Request Provisioning
– Under the Hood
• Form params are converted to POST data
• "Create" request initiate HTTP POST request to
back-end
• To Keystone if auth token is not cached – step 2
- 77. © MIRANTIS 2013 PAGE 77
Step 1: Request Provisioning
– Under the Hood
• Form params are converted to POST data
• "Create" request initiate HTTP POST request to
back-end
• To Keystone if auth token is not cached – step 2
• To Nova API if auth token hasn't expired yet – step 3
- 78. © MIRANTIS 2013 PAGE 78
Nova API
“Nova API is a
RESTful API web service
which is used to interact with Nova"
- 80. © MIRANTIS 2013 PAGE 80
Nova API Characteristics
• Exposes REST API via HTTP
- 81. © MIRANTIS 2013 PAGE 81
Nova API Characteristics
• Exposes REST API via HTTP
• Provides system for managing multiple APIs on
different sub-domains
• EC2-compatible – Starting to be deprecated
• Compute API – all innovation happens here
- 82. © MIRANTIS 2013 PAGE 82
Nova API Characteristics
• Exposes REST API via HTTP
• Provides system for managing multiple APIs on
different sub-domains
• EC2-compatible – Starting to be deprecated
• Compute API – all innovation happens here
• The only "allowed" way to interact with Nova
- 83. © MIRANTIS 2013 PAGE 83
Nova API Characteristics
• Exposes REST API via HTTP
• Provides system for managing multiple APIs on
different sub-domains
• EC2-compatible – Starting to be deprecated
• Compute API – all innovation happens here
• The only "allowed" way to interact with Nova
• Stateless - HA-ready
- 84. © MIRANTIS 2013 PAGE 84
Nova API Clients
Active effort in the community to make
one CLI to “rule them all”, currently
multiple CLIs available
OpenStack dashboard is
currently the only “unified”
OpenStack API client
- 85. © MIRANTIS 2013 PAGE 85
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 3: Send API Request to Nova API
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Horizon sends POST request to
Nova API (signed with given token).
- 86. © MIRANTIS 2013 PAGE 86
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 4: Validate API Token
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Nova API sends HTTP request to
validate API token to Keystone.
- 88. © MIRANTIS 2013 PAGE 88
Keystone /w PKI - Token
Validation
• User gets one-time-password on creation
• User uses it to establish a key-pair
• Public key is signed and stored on Keystone
• From this point user uses client certificate to login
• Nova API performs offline check of the validity of token using CA&Cert it has from Keystone
Keystone
Key
generate_cms_token(
meta, keystone_key)user/pass/tenant
signed_cms_token
signed_cms_token
Nova
CA&Cert from
Keystone
verify(signed_cms_toke
n, Cafile,certfile)
- 89. © MIRANTIS 2013 PAGE 89
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 4: Validate API Token - Sucess
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Keystone validates API token and
sends HTTP response with token
acceptance/rejection info.
- 91. © MIRANTIS 2013 PAGE 91
Step 5:
Process API Request Process
• Validate request params
• Typographical errors are verified on code level
• Cloud-related params are validated via DB requests
- 92. © MIRANTIS 2013 PAGE 92
Step 5:
Process API Request Process
• Validate request params
• Typographical errors are verified on code level
• Cloud-related params are validated via DB requests
• If request cannot be processed then throw an
exception
- 93. © MIRANTIS 2013 PAGE 93
Step 5:
Process API Request Process
• Validate request params
• Typographical errors are verified on code level
• Cloud-related params are validated via DB requests
• If request cannot be processed then throw an
exception
• If request can be processed
• Save initial state to the Database
- 94. © MIRANTIS 2013 PAGE 94
Nova Database
“Nova Database stores current
state of all objects in compute
cluster."
- 95. © MIRANTIS 2013 PAGE 95
Nova Database
• In theory can be any relational database
• Most of the deployments are done with MySQL or
PostgreSQL
• Nova API talks to DB via SQLAlchemy (python
ORM (Object Related Mapper))
• DB HA should be done via external tools (like
Galera or Multi-Master replication Model for
MySQL)
- 96. © MIRANTIS 2013 PAGE 96
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 5: Process API Request
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Nova API parses request to
python object model and
validates it by fetching data
from Nova DB. If request is
valid, it saves initia db entry
about VM to the database.
- 97. © MIRANTIS 2013 PAGE 97
Step 5:
Process API Request Process
• Validate request params
• Typographical errors are verified on code level
• Cloud-related params are validated via DB requests
• If request cannot be processed then throw an
exception
• If request can be processed
• Save initial state to the database
- 98. © MIRANTIS 2013 PAGE 98
Step 5:
Process API Request Process
• Validate request params
• Typographical errors are verified on code level
• Cloud-related params are validated via DB requests
• If request cannot be processed then throw an
exception
• If request can be processed
• Save initial state to the database
• Send message with next actions to MQ – step 6
- 99. © MIRANTIS 2013 PAGE 99
Message Queue
"Message Queue is a unified way for
collaboration between nova
components."
- 100. © MIRANTIS 2013 PAGE 100
Messaging Process Example
• 2 modes:
• rpc.cast - don't wait for result (fire and forget)
• rpc.call - wait for result (when there is something to
return)
- 101. © MIRANTIS 2013 PAGE 101
Messaging Process Example
• 2 modes:
• rpc.cast - don't wait for result (fire and forget)
• rpc.call - wait for result (when there is something to
return)
Ex. Nova API Ex. Nova Scheduler
- 102. © MIRANTIS 2013 PAGE 102
Messaging Process Example
• 2 modes:
• rpc.cast - don't wait for result (fire and forget)
• rpc.call - wait for result (when there is something to
return)
Ex. Nova API Ex. Nova Scheduler
- 103. © MIRANTIS 2013 PAGE 103
Messaging Process Example
• 2 modes:
• rpc.cast - don't wait for result (fire and forget)
• rpc.call - wait for result (when there is something to
return)
Ex. Nova API Ex. Nova Scheduler
- 104. © MIRANTIS 2013 PAGE 104
Messaging Process Example
• 2 modes:
• rpc.cast - don't wait for result (fire and forget)
• rpc.call - wait for result (when there is something to
return)
Ex. Nova API Ex. Nova Scheduler
- 105. © MIRANTIS 2013 PAGE 105
Messaging Process Example
• 2 modes:
• rpc.cast - don't wait for result (fire and forget)
• rpc.call - wait for result (when there is something to
return)
Ex. Nova API Ex. Nova Scheduler
- 107. © MIRANTIS 2013 PAGE 107
OpenStack Messagings Notes
• Uses multiple queues within single RabbitMQ
instance
• Used by services to build machine state
• Each compute node has a queue for scheduling
- 108. © MIRANTIS 2013 PAGE 108
OpenStack Messagings Notes
• Uses multiple queues within single RabbitMQ
instance
• Used by services to build machine state
• Each compute node has a queue for scheduling
• Messages traffic is not intensive
- 109. © MIRANTIS 2013 PAGE 109
OpenStack Messagings Notes
• Uses multiple queues within single RabbitMQ
instance
• Used by services to build machine state
• Each compute node has a queue for scheduling
• Messages traffic is not intensive
• Doesn't send broadcast messages, e.g. for
monitoring, uses API polling instead
- 110. © MIRANTIS 2013 PAGE 110
OpenStack Messagings Notes
• Uses multiple queues within single RabbitMQ
instance
• Used by services to build machine state
• Each compute node has a queue for scheduling
• Messages traffic is not intensive
• Doesn't send broadcast messages, e.g. for
monitoring, uses API polling instead
• HA should be configured separately, e.g.
mirrored queues, not handled by OpenStack
- 111. © MIRANTIS 2013 PAGE 111
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 6a: Publish Provisioning Request
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova API makes rpc.call to
Scheduler. It publishes a
short message to scheduler
queue with VM info. UI: Horizon or CLI
Request has been validated, but no action has
been taken yet, i.e. which host, IP address, etc.
- 112. © MIRANTIS 2013 PAGE 112
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 7: Pick up Provisioning Request
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Scheduler picks up the
message from MQ.
UI: Horizon or CLI
- 113. © MIRANTIS 2013 PAGE 113
Nova Scheduler
“Nova Scheduler is a daemon, which
determines, on which compute host
the request should run.“
• Only provisioning time component, i.e. not like VMware’s DRS
• Typically co-located with the Cloud Controller
- 114. © MIRANTIS 2013 PAGE 114
VM Scheduling:
Typical Requirements
• provision VM to particular host
• provision VMs of the particular tenant to
isolated hosts
• provision all VMs on different hosts
• provision VMs to "higher density" hosts
- 115. © MIRANTIS 2013 PAGE 115
Nova Scheduler: Available
Schedulers
Scheduler Description Behavior
Chance Picks a host that is up Random
Simple
Picks a host that is up and
has the fewest running
instances
Least Loaded
Filter
Picks the best-suited host
which satisfies selected
filter
Custom JSON Filters
Multi (Deprecated, to be
replaced by cells)
A scheduler that holds
multiple sub-schedulers
Collection of filters
commonly used for
multi-site or customized
deployments
- 116. © MIRANTIS 2013 PAGE 116
Nova Scheduler: Filtering
Affinity, Anti-affinity,
etc.
Eliminate
inapplicable hosts
- 117. © MIRANTIS 2013 PAGE 117
Nova Scheduler: Filters
Filter Description
affinity Same host or different host
availability zone Least cost inside selected availability zone
core Least CPU core utilization
ram Only return hosts with sufficient RAM
json Allows simple JSON based grammar. Can be used to
build custom schedulers.
i/o filter out hosts with too many concurrent I/O
operations
compute capabilities match attributes with compute node's capabilities
(e.g. CPU arch.)
- 118. © MIRANTIS 2013 PAGE 118
Nova Scheduler: Filters
Filter Description
aggregate specs match the attributes for the instance with those
provided by aggregate
image properties find compute nodes with capabilities matching
image specification from glance
isolated host match given image with a group of compute nodes
trusted host (by Intel) finds only "attested" hosts
type find only compute nodes which do not run any
instances
… A lot more
- 119. © MIRANTIS 2013 PAGE 119
Nova Scheduler: Filters
• Filters are statically configured in nova.conf
• Multiple filters can be specified
• It is possible to create custom filter
• Inherit from BaseHostFilter class
• Override host_passes(self, host_state, filter_properties)
- 120. © MIRANTIS 2013 PAGE 120
Nova Scheduler: Filtering
RAM, CPU, etc.
Integer values
- 121. © MIRANTIS 2013 PAGE 121
Nova Scheduler: Weights and
Costs
• Cost - integer value
• Every compute host can have several cost
functions associated with it
• If no cost functions associated - use default
from nova.conf
• weight = sum(costi + weigth_fni)
- 122. © MIRANTIS 2013 PAGE 122
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 8a: Schedule Provisioning
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Scheduler fetches
information about the
whole cluster from
database, filters, and
selects compute node and
updates DB with its ID
UI: Horizon or CLI
- 123. © MIRANTIS 2013 PAGE 123
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 8b: Provision Scheduled
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Scheduler publishes
message to the compute
queue (based on host ID) to
trigger VM provisioning UI: Horizon or CLI
- 124. © MIRANTIS 2013 PAGE 124
VM Provisioning Algorithm
• Step 9 – Nova Compute gets message from MQ and asks
Nova Conductor for VM info from database
• Step 10 – Nova Compute queries Quantum (previously Nova-
Network in Essex) to allocate networking information
• Step 11 – Nova Compute queries Cinder to allocate volume
information (optional step for persistent data)
• Steps 12-14 – Nova Compute fetches VM image (base OS)
from Glance
• Step 15 – Nova Compute passes all information about VM (in
a single message) to Hypervisor and Hypervisor (KVM / Xen)
creates an instance
- 125. © MIRANTIS 2013 PAGE 125
Nova Compute
“Nova Compute is a worker
daemon, which primarily creates
and terminates VMs via
Hypervisor API."
- 126. © MIRANTIS 2013 PAGE 126
Nova Compute Drivers
Nova Compute
XCP
VM
VM
VMWare
VM
VM
HyperV
VM
VM
LPAR
VM
VM
libvirt
KVM
VM
VM
Xen
VM
VM
Qemu
VM
VM
LXC
VM
VM
Today only 1 hypervisor type per
cloud instance. Libvirt / KVM is most
common deployment
Maintained
by Citrix
Maintained by
VMWare
Maintained
by Microsoft
Maintained
by IBM
Native or
through libvirt
Bare
Metal
VM
VM
Experimental
at this point
- 127. © MIRANTIS 2013 PAGE 127
Nova Compute Drivers
(Continued)
• Functionality is not 100% similar
• Exact "run_instance" flow depends on driver
implementation
• Most of the features are developed and tested
on KVM
- 128. © MIRANTIS 2013 PAGE 128
Nova Compute Config
(nova.conf on each host)
• --libvirt_type
• Hypervisor being used. In this deployment ‘kvm’ is specified.
• --libvirt_uri
• URI to use for connection to hypervisor. In this deployment ‘qemu+tcp:///system’ is specified.
• --sql_connection
• Database connection string in SQLAlchemy format. This is used for connecting to state database (if
Nova Conductor is not used)
• --rabbit_host
• IP address for RabbitMQ host. Non-standard port also can be specified
• --glance_host
• IP address and port of Glance Image Service host. This is needed for streaming virtual boot images.
• --glance_api_server
• IP address and port of Glance API server. This is needed for getting virtual boot images meta-data.
- 129. © MIRANTIS 2013 PAGE 129
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 9a: Start VM Provisioning
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute gets
message from MQ
UI: Horizon or CLI
- 130. © MIRANTIS 2013 PAGE 130
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 9b: Start VM Provisioning
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Keystone
Keystone Server
Keystone DB
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute makes
rpc.call to Nova Conductor
for Information on VM from
DB
UI: Horizon or CLI
- 131. © MIRANTIS 2013 PAGE 131
Nova Conductor
“The Nova Conductor service is key
to completing no-db-compute"
- 132. © MIRANTIS 2013 PAGE 132
Nova Conductor Notes
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()
- 133. © MIRANTIS 2013 PAGE 133
Nova Conductor Notes
• Eliminites remote DB access (security)
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()
- 134. © MIRANTIS 2013 PAGE 134
Nova Conductor Notes
• Eliminites remote DB access (security)
• Horizontal scalability (performance)
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()
- 135. © MIRANTIS 2013 PAGE 135
Nova Conductor Notes
• Eliminites remote DB access (security)
• Horizontal scalability (performance)
• Hides DB implementation/schema from the Nova Compute (upgrades)
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()
- 136. © MIRANTIS 2013 PAGE 136
Nova Conductor Notes
• Eliminites remote DB access (security)
• Horizontal scalability (performance)
• Hides DB implementation/schema from the Nova Compute (upgrades)
• Possible offloading of long-running operations from other services, not just Nova Compute
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()
- 137. © MIRANTIS 2013 PAGE 137
Nova Conductor Notes
• Eliminites remote DB access (security)
• Horizontal scalability (performance)
• Hides DB implementation/schema from the Nova Compute (upgrades)
• Possible offloading of long-running operations from other services, not just Nova Compute
• Beneficial for operations that cross multiple compute nodes (migration, resizes)
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()
- 138. © MIRANTIS 2013 PAGE 138
Nova Conductor Notes
• Eliminites remote DB access (security)
• Horizontal scalability (performance)
• Hides DB implementation/schema from the Nova Compute (upgrades)
• Possible offloading of long-running operations from other services, not just Nova Compute
• Beneficial for operations that cross multiple compute nodes (migration, resizes)
• “This is just one (major) step along the path”
controller node
DB
nova-conductor
compute node
nova-compute
rpc.call()
- 139. © MIRANTIS 2013 PAGE 139
VM Provisioning Algorithm
• Step 9 – Nova Compute gets message from MQ and asks
Nova Conductor for VM info from Nova Conductor
• Step 10 – Nova Compute queries Quantum (previously Nova-
Network in Essex) to allocate networking information
• Step 11 – Nova Compute queries Cinder to allocate volume
information (optional step for persistent data)
• Steps 12-14 – Nova Compute fetches VM image (base OS)
from Glance
• Step 15 – Nova Compute passes all information about VM (in
a single message) to Hypervisor and Hypervisor (KVM / Xen)
creates an instance
- 142. © MIRANTIS 2013 PAGE 142
Quantum Notes
• Provides a flexible API (POST / GET) for service providers or
their tenants to manage OpenStack network topologies
• Create networks, associate VMs, set routers, etc.
- 143. © MIRANTIS 2013 PAGE 143
Quantum Notes
• Provides a flexible API (POST / GET) for service providers or
their tenants to manage OpenStack network topologies
• Create networks, associate VMs, set routers, etc.
• Presents a logical API and a corresponding plug-in
architecture that separates the description of network
connectivity from its implementationion
- 144. © MIRANTIS 2013 PAGE 144
Quantum Notes
• Provides a flexible API (POST / GET) for service providers or
their tenants to manage OpenStack network topologies
• Create networks, associate VMs, set routers, etc.
• Presents a logical API and a corresponding plug-in
architecture that separates the description of network
connectivity from its implementationion
• API evolves independently of the compute API, allowing to
introduce more advanced network capabilities (e.g. QoS,
ACLs, etc.)
- 145. © MIRANTIS 2013 PAGE 145
Quantum Notes
• Provides a flexible API (POST / GET) for service providers or
their tenants to manage OpenStack network topologies
• Create networks, associate VMs, set routers, etc.
• Presents a logical API and a corresponding plug-in
architecture that separates the description of network
connectivity from its implementationion
• API evolves independently of the compute API, allowing to
introduce more advanced network capabilities (e.g. QoS,
ACLs, etc.)
• In Folsom/Grizzly one can choose to stay with nova-network
(Essex approach) or to go with Quantum
- 146. © MIRANTIS 2013 PAGE 146
Quantum Architecture -
“Bird's Eye" View
3rd Party plug-in,
networking data is stored
outside of OpenStack /
Quantum
Quantum native
functionality
- 147. © MIRANTIS 2013 PAGE 147
Network Configuration Flow
• Allocate MAC addresses
• Allocate IPs (for each network)
• Associate IP and MAC with VM (DB)
• Setup network - L2:
• configure L2 via a quantum plugin
• actual action can be variable, depending on the plugin used
(with OVS plugin the action is: plugging an instance into the
integration bridge on the hypervisor)
• Setup network - L3
• Update DHCP config
• Initialize gateway
Allocation during
cloud setup
Association and
Setup during VM
provisioning
- 148. © MIRANTIS 2013 PAGE 148
Available Quantum Plugins
• Linux Bridge
• OpenVSwitch (most common)
• Nicira NVP
• Cisco (UCS Blade + Nexus)
• Ryu OpenFlow controller
• NEC ProgrammableFlow Controller
- 149. © MIRANTIS 2013 PAGE 149
Step 10: Configure Network
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute makes a call to
Quantum API to provision
network for the instance
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
- 150. © MIRANTIS 2013 PAGE 150
Step 10: Configure Network (Continued)
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Quantum configures IP, gateway,
DNS name, L2 connectivity, etc.
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Static
Dynamic
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
- 151. © MIRANTIS 2013 PAGE 151
VM Provisioning Algorithm
• Step 9 – Nova Compute gets message from MQ and asks
Nova Conductor for VM info from database
• Step 10 – Nova Compute queries Quantum (previously Nova-
Network in Essex) to allocate networking information
• Step 11 – Nova Compute queries Cinder to allocate volume
information (optional step for persistent data)
• Steps 12-14 – Nova Compute fetches VM image (base OS)
from Glance
• Step 15 – Nova Compute passes all information about VM (in
a single message) to Hypervisor and Hypervisor (KVM / Xen)
creates an instance
- 155. © MIRANTIS 2013 PAGE 155
Cinder Notes
• Optional
• iSCSI solution which can plug into a number of
storage backends
- 156. © MIRANTIS 2013 PAGE 156
Cinder Notes
• Optional
• iSCSI solution which can plug into a number of
storage backends
• Volume can be attached only to 1 instance at a
time
- 157. © MIRANTIS 2013 PAGE 157
Cinder Notes
• Optional
• iSCSI solution which can plug into a number of
storage backends
• Volume can be attached only to 1 instance at a
time
• Persistent volumes keep their state
independent of instances
- 159. © MIRANTIS 2013 PAGE 159
Cinder Drivers
• iSCSI
• Fibre Channel
• Xen Storage Manager
• Nexenta
• NetApp
• Zadara VPSA
• SAN
• NFS (volumes as sparse files)
• RBD Ceph
• IBM Storwize / XIV
• HP 3PAR
• Coraid
• Huawei
• Scality SOFS
• GlusterFS
• LVM thin provisioning support
• Mirrored LVM
• XenAPINFS
• EMC VNX/VMAX arrays
• Solidfire
- 160. © MIRANTIS 2013 PAGE 160
Step 11: Request Volume
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
It is assumed a volume is
already created. Nova
Compute contacts Cinder to
get volume data. Can also
attach volumes after VM is
built.
- 161. © MIRANTIS 2013 PAGE 161
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 11: Request volume (Continued)
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute sets up
iSCSI initiator & instructs
the Hypervisor to mount
iSCSI vol. as a new block
device UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
- 162. © MIRANTIS 2013 PAGE 162
VM Provisioning Algorithm
• Step 9 – Nova Compute gets message from MQ and asks
Nova Conductor for VM info from database
• Step 10 – Nova Compute queries Quantum (previously Nova-
Network in Essex) to allocate networking information
• Step 11 – Nova Compute queries Cinder to allocate volume
information (optional step for persistent data)
• Steps 12-14 – Nova Compute fetches VM image (base OS)
from Glance
• Step 15 – Nova Compute passes all information about VM (in
a single message) to Hypervisor and Hypervisor (KVM / Xen)
creates an instance
- 163. © MIRANTIS 2013 PAGE 163
Glance
"The Glance project provides
services for discovering,
registering, and retrieving virtual
machine images."
- 164. © MIRANTIS 2013 PAGE 164
Glance Summary
• Images-as-a-Service
• Can use multiple back-ends for image storage
• Supports multiple image formats
- 167. © MIRANTIS 2013 PAGE 167
Glance Capabilities
• CRUD images (Create, Read, Update, Delete)
- 168. © MIRANTIS 2013 PAGE 168
Glance Capabilities
• CRUD images (Create, Read, Update, Delete)
• Search images via filters
• name
• container format
• disk format
• size_min, size_max
• status
- 169. © MIRANTIS 2013 PAGE 169
Glance Capabilities
• CRUD images (Create, Read, Update, Delete)
• Search images via filters
• name
• container format
• disk format
• size_min, size_max
• status
• Caches images
• uses SQLite or FS that supports xattrs for caching
• queues images for prefetching
• prefetches images
• prunes images
• cleans invalid cache entries
- 170. © MIRANTIS 2013 PAGE 170
Glance Image Formats
Disk Format Description
raw This is an unstructured disk image format
vhd This is the VHD disk format, a common disk format used by virtual machine monitors from
VMWare, Xen, Microsoft, VirtualBox, and others
vmdk Another common disk format supported by many common virtual machine monitors
vdi A disk format supported by VirtualBox virtual machine monitor and the QEMU emulator
iso An archive format for the data contents of an optical disc (e.g. CDROM).
qcow2 A disk format supported by the QEMU emulator that can expand dynamically and supports Copy
on Write
aki This indicates what is stored in Glance is an Amazon kernel image
ari This indicates what is stored in Glance is an Amazon ramdisk image
ami This indicates what is stored in Glance is an Amazon machine image
- 171. © MIRANTIS 2013 PAGE 171
Custom Image Creation
• Get installation ISO
• Create VM (qemu-img create)
• Start VM and connect to it via VNC console
• Install image without LVM
• Create default iptables rules
• Install and configure cloud-init
• With cloud-init configure image
• Prepare image for OpenStack
• Extract root partition, kernel and ramdisk
• cleanup
• package
- 172. © MIRANTIS 2013 PAGE 172
Step 12: Request VM Image from Glance
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute requests VM
image from Glance via Image ID
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
- 173. © MIRANTIS 2013 PAGE 173
Step 13: Get Image URI from Glance
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
If image with given image ID can be
found - return URI – HTTP Get URI
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
- 174. © MIRANTIS 2013 PAGE 174
Step 14: Download Image from Swift
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute downloads image using URI,
given by Glance, from Swif
(or Glance's back-end)
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
- 175. © MIRANTIS 2013 PAGE 175
VM Provisioning Algorithm
• Step 9 – Nova Compute gets message from MQ and asks
Nova Conductor for VM info from database
• Step 10 – Nova Compute queries Quantum (previously Nova-
Network in Essex) to allocate networking information
• Step 11 – Nova Compute queries Cinder to allocate volume
information (optional step for persistent data)
• Steps 12-14 – Nova Compute fetches VM image (base OS)
from Glance
• Step 15 – Nova Compute passes all information about VM (in
a single message) to Hypervisor and Hypervisor (KVM / Xen)
creates an instance
- 176. © MIRANTIS 2013 PAGE 176
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 15: Start VM Rendering via Hypervisor
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Nova Compute fetches
information about VM from
DB, creates a command to
Hypervisor and delegates
VM rendering to Hypervisor. UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
In case of KVM / libvirtd this is
a single XML VM config file
- 177. © MIRANTIS 2013 PAGE 177
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 16: VM is UP
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
iSCSI communication
begins for volume
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
Nova Compute sends a
message to Nova
Conductor to update
DB with VM state
- 178. © MIRANTIS 2013 PAGE 178
Nova
Nova DB
Queue
Nova:
ControllerNova API
Scheduler
Conductor
Nova Cell
Step 17: User is Happy
Swift
Object Store
Proxy Server
Quantum
Quantum DB
Plugin / Agent
Quantum Server
Cinder
Cinder API
Cinder DB
Scheduler
Queue
Cinder Vol
Glance
Glance API
Glance Registry
Glance DB
Compute Node
Hypervisor
Network
VM
Nova:
Computenova-compute
Block Storage
Node
Storage
Network Node
DHCP / IPAM
Router / GW
Horizon polls Nova API for
VM status and power state,
which is taken from
Database.
UI: Horizon or CLI
Keystone
Keystone Server
Keystone DB
- 180. © MIRANTIS 2013 PAGE 180
Recap:
• Users logs into Horizon and initiates a VM create
- 181. © MIRANTIS 2013 PAGE 181
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
- 182. © MIRANTIS 2013 PAGE 182
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
- 183. © MIRANTIS 2013 PAGE 183
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host
- 184. © MIRANTIS 2013 PAGE 184
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host
• Quantum configures networking
- 185. © MIRANTIS 2013 PAGE 185
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host
• Quantum configures networking
• Cinder provides block device
- 186. © MIRANTIS 2013 PAGE 186
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host
• Quantum configures networking
• Cinder provides block device
• Image URI is looked up through Glance
- 187. © MIRANTIS 2013 PAGE 187
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host
• Quantum configures networking
• Cinder provides block device
• Image URI is looked up through Glance
• Image is retrieved via Swift
- 188. © MIRANTIS 2013 PAGE 188
Recap:
• Users logs into Horizon and initiates a VM create
• Keystone authorizes
• Nova initiates provisioning and saves state to DB
• Nova Scheduler finds appropriate host
• Quantum configures networking
• Cinder provides block device
• Image URI is looked up through Glance
• Image is retrieved via Swift
• VM is rendered