HA in OpenStack service - meetup #9

1. High Available controller core services
VIETOPENSTACK 9th MEETUP
Ha Noi, 07/05/2016

2. Click to edit Master title style
OpenStack release: Liberty
OS version: Ubuntu 14.04.3 LTS, Trusty Tahr
VMware ESXi 5.5
Lab information
2High Available controller core services

3. Click to edit Master title styleHigh availability concepts
3High Available controller core services
System downtime
Occurs when a user-facing service is unavailable beyond a specified maximum amount of time.
Data loss
Accidental deletion or destruction of data.
Stateless service
nova-api, nova-conductor, glance-api, keystone-api, neutron-api and nova-scheduler,…
Stateful service
OpenStack database and message queue. Making stateful services highly available can depend on
whether you choose an active/passive or active/active configuration.
One uses a cluster manager such as Pacemaker to co-ordinate the actions of the various services
across a set of machines
HAProxy provides a fast and reliable HTTP reverse proxy and load balancer for TCP or HTTP
applications

4. Click to edit Master title style
Keystone
Glance
Nova
Neutron
L3HA
MySQL
RabbitMQ
Controller core services to be HA
4High Available controller core services

5. Click to edit Master title style
Each instance of HAProxy configures its front end to accept
connections only from the virtual IP (VIP) address and to terminate
them as a list of all instances of the corresponding service under
load balancing, such as any OpenStack API service.
This makes the instances of HAProxy act independently and fail over
transparently together with the network endpoints (VIP addresses)
failover and, therefore, shares the same SLA
HAproxy service
5High Available controller core services
Load distribution
API isolation
Simplified process for adding/removing of nodes
Enhanced failure detection

6. Click to edit Master title style
Pacemaker to manage VIP & Haproxy service.
Pacemaker relies on the Corosync messaging layer for reliable
cluster communications.
Corosync implements the Totem single-ring ordering and
membership protocol. It also provides UDP and InfiniBand based
messaging, quorum, and cluster membership to Pacemaker.
Pacemaker relies on resource agents (RAs), scripts that encapsulate
the knowledge of how to start, stop, and check the health of each
application managed by the cluster
Pacemaker cluster stack & corosync service
6High Available controller core services

7. Click to edit Master title styleArchitecture
7High Available controller core services

8. Click to edit Master title style
Use Galera Cluster to provide replication between
instances of the database on controller nodes.
Galera Cluster is a synchronous multi-master database
cluster, based on MySQL and the InnoDB storage engine.
It is a high-availability service that provides high system
uptime, no data loss, and scalability for growth.
Mariadb Galera cluster
8High Available controller core services

9. Click to edit Master title style
Memcached is a general-purpose distributed memory caching system. It
is used to speed up dynamic database-driven websites by caching data
and objects in RAM to reduce the number of times an external data
source must be read.
Memcached is a memory cache demon that can be used by most
OpenStack services to store ephemeral data, such as tokens.
Access to memcached is not handled by HAproxy because replicated
access is currently only in an experimental state. Instead OpenStack
services must be supplied with the full list of hosts running memcached.
Memcached
9High Available controller core services

10. Click to edit Master title style
RabbitMQ is AMQP broker to coordinate the execution of jobs entered into the system
(communication service)
Configure RabbitMQ for HA queues
Build a cluster of RabbitMQ nodes to construct a RabbitMQ broker,
which is a logical grouping of several Erlang nodes.
Mirrored queues in RabbitMQ improve the availability of service
since it is resilient to failures.
Configure OpenStack services to use Rabbit HA queues
We have to configure the OpenStack components to use at least two
RabbitMQ nodes.
RabbitMQ cluster
10High Available controller core services

11. Click to edit Master title style
1, Stop Pacemaker or Corosync service
2, Stop 1 of 3 nodes
3, Stop 2 of 3 nodes
4, Other recommended
Failover & failback senarios
11High Available controller core services

12. Click to edit Master title style
VPN to lab
Demo
12High Available controller core services

13. Click to edit Master title styleThe end
13High Available controller core services
Source: http://docs.openstack.org/ha-guide/
Support team: VietOpenStack