8. 无状态服务（Stateless services）
◦ 请求之间没有依赖
◦ 比如: Nova API, Nova Scheduler, etc.
有状态服务（Stateful services）
◦ 一次操作需要多个关联请求完成
◦ 比如: MySQL, Qpid, etc.
8
9. Active/Passive
◦ Redundant instances of stateless services are load
balanced
◦ For Stateful services a replacement resource can be
brought online.
Active/Active
◦ Redundant instances of stateless services are load balanced
◦ Stateful services are managed in such a way that services are redundant,
and that all instances have an identical state.
◦ Updates to one instance of a database would also update all
other instances.
9
10. Failover
◦ Migration of a service from the “primary” to the
“secondary”
Failback
◦ Migration of service back to the “primary”
11. Compute HA
◦ Instance HA
Controller HA
◦ MySQL
◦ Qpid
◦ OpenStack APIs (keystone, nova-api etc.)
◦ Nova, Neutron, Cinder, Swift, and so on
Overview HA solution
12. 12
Evacuation
Evacuation
Without Shared Storage
◦ The instance will be booted
from a new disk, but will
preserve the configuration,
e.g. id, name, uid, ip...etc.
With Shared Storage
◦ The instance will be booted
from same disk and data
will be preserved
15. Compute HA
◦ Instance HA
Controller HA
◦ MySQL
◦ Qpid
◦ OpenStack APIs (keystone, nova-api etc.)
◦ Nova, Neutron, Cinder, Swift, and so on
Overview HA solution
20. 20
Pacemaker
◦ high availability and load
balancing stack for the Linux
platform.
◦ Interacts with applications
through Resource Agents (RA)
Corosync
◦ Totem single-ring ordering and
membership protocol
◦ UDP and InfiniBand based
messaging, quorum, and
cluster membership to
Pacemaker.
DRBD (Distributed
Replication Block
Device)
◦ Synchronizes Data at the
block device
◦ Uses a journaling system
(such as ext3 or ext4)
http://dev.mysql.com/doc/mysql-ha-
scalability/en/ha-drbd.html
21.
22. Synchronous multi-master
cluster technology for
MySQL/InnoDB
◦ MySQL patched for wsrep
(Write Set REPlication)
◦ Active/active multi-master
topology
◦ Read and write to any
cluster node
◦ True parallel replication, in
row level
◦ No slave lag or integrity
issues
22
23. 23
Database Replication
method
Strengths Weakness/Limit
ations
Keepalived/HAPr
oxy/VRRP
Works on MySQL
master-master
replication
Simple to
implement and
understand.
Works for any
storage system.
Master-master
replication does
not work beyond
2 nodes.
Pacemaker/Coro
sync/DRBD
Mirroring on
Block Devices
Well tested More complex
to setup. Split
Brain possibility
Galera Based on write-
set Replication
(wsrep)
No Slave lag Needs at least 3
nodes. Relatively
new.
Others MySQL Cluster,
RHCS with
DAS/SAN
storage
Well tested More complex
setup.
24. Pacemaker managed without clustering
Clustered without pacemaker
Pacemaker managed with clustering
30. HAProxy
◦ Load Balancing and Proxying for HTTP and TCP Applications
◦ Works over multiple connections
◦ Used to load balance API services
VRRP (Virtual Router Redundancy Protocol)
◦ Eliminates SPOF in a static default routed environment
Keepalived
◦ Based on Linux Virtual Server (IPVS) kernel module to
provide layer 4 Load Balancing
◦ Implements a set of checkers to check service status and to
maintain health
◦ Leverage the VRRP Protocol to remap VIPS in event of failure
30
40. Active/active
◦ dhcp-agent / openvswitch-agent/neutron-server
support active/ passive
◦ L3-agent and metadata-agent
41. Compute HA
◦ Instance HA
Controller HA
◦ MySQL
◦ Qpid
◦ OpenStack APIs (keystone, nova-api etc.)
◦ Nova, Neutron, Cinder, Swift, and so on
Overview HA solution
42. 42
.…
Availability
Zone 1
Dedicated Firewalls
BOND0
BOND1
BOND0
BOND1
Controller
API Services
API & Horizon
Cinder API
Nova Scheduler
Keystone
Glance
RabbitMQ
MYSQL
Chef Server
Recipes
Load Balancers
Redundant Network Switches
Storage
EMC, NetApp, or Solidfire Vols
BOND2
Redundant Network Switches
Inside LB VLAN
Storage Network (private)
Fixed Network (private)
Compute 1
KVM
G2
G1
G4
G3
Compute N
KVM
G6
G5
G7
BOND0
BOND1
BOND2
.…
Availability
Zone 2
BOND0
BOND1
BOND2
Compute 1
KVM
Compute N
KVM
G16
G15
G17
BOND0
BOND1
BOND2
BOND0
BOND1
BOND2
G12
G11
G14
G13
BOND2
Controller
API Services
API & Horizon
Cinder API
Nova Scheduler
Keystone
Glance
RabbitMQ
MYSQL
Chef Server
Recipes
43.
44. OpenStack HA
◦ HA基本概念
◦ 计算节点 HA
◦ 控制节点 HA
OpenStack 性能调优
OpenStack 日志分析与排错技巧
46. I/O :
◦ All instances use local file system to host file systems.
Scarce resources as more instances are run per server by
increasing cores. First to be hit.
Memory/RAM:
◦ Second factor to be hit after disks. Most VMs use RAM
more extensively than CPU. With increasing cores and
larger VMs RAM contention becomes a problem
CPU:
◦ Usually the last to be hit. Not as much of a problem any
more because of Hyper threading and multiple cores.
47. Flavors:
◦ Only allow sensible Flavors for the users. Example on a
compute node with 8 CPU cores and 96 GB ram avoid
creating a Flavors that uses 1 vCPU and 64 GB RAM
Quotas:
◦ Used to limit the number of resources used by a particular
tenant: number of instances, block volume number and
space, or number of snapshots and images kept in Glance.
Consider the potential number of tenants and available
hardware.
Over provisioning:
◦ Use technologies like thin provisioning, hyper threading to
over provision resources but have to be careful about
performance hits.
49. Kernel I/O scheduler to “Deadline”.
◦ Default is cfq, good enough for most work loads
but for over provisioning use “deadline”
Huge pages enabled
Kernel same-page merging enabled (KSM)
Hyper threading turned on
Place guest file systems directly on hypervisor
block devices instead of in files.
50.
51. There are couple of important benefits of
Huge Pages:
• Page size is set 2MB instead of 4KB
• Memory used by Huge Pages is locked and cannot
be swapped.
52. The ksm tuned process work in the following
way:
◦ scans through the memory finding duplicate pages
◦ Merges duplicate page to single page
◦ Map to all virtual machine locations
◦ Set copy on write
◦ Separate page when individual guests write to it.
53. OpenStack HA
◦ HA基本概念
◦ 计算节点 HA
◦ 控制节点 HA
OpenStack 性能调优
OpenStack 日志分析与排错技巧
