L3HA-VRRP-20141201

日本OpenStackユーザ会第20回勉強会
Neutron L3 HA (VRRP)
2014-12-01
Manabu Ori

自己紹介
● 名前: 織学 (おりまなぶ)
● 所属: レッドハット
● Twitter: @orimanabu

自己紹介
● 名前: 織学 (おりまなぶ)
● 所属: レッドハット
● Twitter: @orimanabu
● slideshareに上げた資料たち
– POWER + PostgreSQL
– POWER + KVM

Neutron L3 agentとは
● OpenStackのネットワーク仮想化で使用するエージェ
ントのひとつ
– 他にはMetadata Proxy agentとかDHCP agentとか
● テナントネットワークのサブネット間接続、テナント
VMと外部ネットワークとの接続等を制御する
● 複数ノードで起動可能
– 仮想ルータはどこか1ノード上で実行される
仮想ルータ #1 仮想ルータ #2
L3 Agent #1 L3 Agent #2
VM
VM
VM
パブリック
ネットワーク
テナント
ネットワーク

これまでのL3 agentの冗長化
● コントローラーノードによるリスケジューリング
– allow_automatic_l3agent_failover = True
– Juno以降で使用可能
– https://review.openstack.org/#/c/110893/
– https://bugs.launchpad.net/neutron/+bug/1174591
● L3 Agent Healthcheck
– OpenStackと独立して開発 (by eNovance)
– Grizzly, Havana, Icehouseで使用可能
– https://github.com/enovance/neutron-l3-healthcheck
● Pacemaker等でHAクラスター化
フェイルオーバーに
時間がかかる
フェイルオーバーに
時間がかかる
設定が大変

L3 HA概要
● L3 agentの冗長化機能を拡張/ドライバの形で実装
● 今の実装はkeepalivedを使用
– VRRPを用いて死活監視
– フェイルオーバーはRPC layerから独立
● テナントごとにひとつVRRPハートビートネットワークを作
成
– ハートビートネットワークも通常のNeutronネットワーク
– 物理的にはテナントネットワークを通る
● 全てのテナント分割方式 (VXLAN, VLAN, GRE, ...)で使用可能
● フェイルオーバー時は、ゲートウェイのIPアドレスおよび
Floating IPを仮想IPアドレスとして引き継ぐ

VRRPとは
● VRRP: Virtual Router Redundancy Protocol
● RFC3768
● 複数のルータから仮想的なルータを構成
– マスタールータ
● 一時点に1台のみ
● 実際にフォワーディング処理を行う
– バックアップルータ
● 通常は待機
● 障害時にマスターになる
● それぞれのルータは、ステータスの確認メッセージを交換するこ
とで障害を検知
● First Hop Redundancy Protocol
– サーバ、PC等のデフォルトルートとしてIPアドレス、MACアドレスを
提供

ネットワークノードの構成
- non-HAの場合 -
Network Node #1 Network Node #2
eth2
br-ex
qg-BBB
qr-AAA
br-int
IP address
int-br-tenant
phy-br-tenant
br-tenant
eth1
eth2
br-ex
qg-EEE
qr-DDD
br-int
IP address
int-br-tenant
phy-br-tenant
br-tenant
eth1
パブリックネットワーク
テナントネットワーク
IP address
IP address
qrouter-xxxx qrouter-yyyy

ネットワークノードの構成
- HAの場合 -
Network Node #1 Network Node #2
eth2
br-ex
qg-BBB
br-int
qr-AAA
qrouter-xxxx qrouter-yyyy
ha-CCC
int-br-tenant
phy-br-tenant
br-tenant
eth1
eth2
br-ex
qg-EEE
br-int
ha-FFF
qr-DDD
int-br-tenant
phy-br-tenant
br-tenant
eth1
パブリックネットワーク
VRRPハートビート
テナントネットワーク
VIP
VIP VIP
VIP
Gateway
Floating IP
Fixed IP
keepalived keepalived
IP address IP address

制限事項
● VRRPハートビート用のネットワークは1つのみ
– テナントネットワークを流れる
● テナント当たりの仮想ルータは最大256個
– ハートビートネットワークが1つかつ VRIDが8ビット
● セッションの引き継ぎは現時点では未サポート
● DVRとの同時使用は現時点では未サポート
● (LB|FW|VPN)aaSとの統合は現時点では未サポート
● 今後の改良予定
– どれがマスターがわかりにくい問題
– 手動フェイルオーバーの方法問題
– L2populationと仲良くなる

Gerrit Topics
● https://review.openstack.org/#/q/topic:bp/l3-h
igh-availability,n,z
●

関連する設定パラメータ
最低限の設定
● /etc/neutron/neutron.conf
– [Default]
● l3_ha = True
その他のパラメータ
● /etc/neutron/neutron.conf
– [Default]
● max_l3_agents_per_router 3
● min_l3_agents_per_router 2
● l3_ha_net_cidr 169.254.192.0/18
● /etc/neutron/l3_agent.ini
– [Default]
● agent_mode legacy
● ha_confs_path $state_path/ha_confs
● ha_vrrp_auth_type PASS
● ha_vrrp_auth_password openstack
● ha_vrrp_advert_int 2

動き
● neutron router-createすると
– ハートビート用のNeutronネットワークを作成
– ハートビート用のインタフェースをルータの
Namespace内に作成
– keepalivedを起動
Public Network
Management/API Network
eth0 eth0 eth2 eth0 eth2 eth0
controller network1 network2 compute1
eth1 eth1 eth1 eth1
Tenant Network

ルータ作成 (1)
● neutron router-createすると...
# neutron router-create --tenant-id $(keystone tenant-list | awk '/demo/ {print $2}') router_demo
Created a new router:
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| admin_state_up | True |
| distributed | False |
| external_gateway_info | |
| ha | True |
| id | e7e98ef7-36d5-4544-be07-c83ddf7e67ea |
| name | router_demo |
| routes | |
| status | ACTIVE |
| tenant_id | 328ffb467eb24c3595b549516b8acbd0 |
+-----------------------+--------------------------------------+

ルータ作成 (2)
● 両ノードのL3 agentで動いている
# neutron l3-agent-list-hosting-router router_demo
+--------------------------------------+---------------+----------------+-------+
| id | host | admin_state_up | alive |
+--------------------------------------+---------------+----------------+-------+
| 1c407f95-5a62-4577-a86d-623943ab0283 | juno-network1 | True | :-) |
| 8ad4e4df-cd8e-4c6e-bea2-8101a1225019 | juno-network2 | True | :-) |
+--------------------------------------+---------------+----------------+-------+
● ハートビート用のネットワークができている
# neutron net-list
+--------------------------------------+----------------------------------------------------+----------------------------------| id | name | subnets +--------------------------------------+----------------------------------------------------+----------------------------------| 616a2195-162c-4e09-8afa-a347c56b9c11 | external | 3c705be7-6d51-| 218b2842-1832-4ec6-a979-5efdf01453bf | HA network tenant 328ffb467eb24c3595b549516b8acbd0 | 288d389b-33c9-+--------------------------------------+----------------------------------------------------+----------------------------------

ルータ作成 (3)
● ハートビート用ネットワークの情報
# neutron net-show 218b2842-1832-4ec6-a979-5efdf01453bf
+---------------------------+----------------------------------------------------+
| Field | Value |
+---------------------------+----------------------------------------------------+
| id | 218b2842-1832-4ec6-a979-5efdf01453bf |
| name | HA network tenant 328ffb467eb24c3595b549516b8acbd0 |
| provider:network_type | vlan |
| provider:physical_network | physnet-tenant |
| provider:segmentation_id | 2001 |
| router:external | False |
| shared | False |
| status | ACTIVE |
| subnets | 288d389b-33c9-4559-8ec8-98b4faac2274 |
| tenant_id | |
+---------------------------+----------------------------------------------------+
# neutron subnet-show 288d389b-33c9-4559-8ec8-98b4faac2274
+-------------------+------------------------------------------------------+
| Field | Value |
+-------------------+------------------------------------------------------+
| allocation_pools | {"start": "169.254.192.1", "end": "169.254.255.254"} |
| cidr | 169.254.192.0/18 |
| dns_nameservers | |
| enable_dhcp | False |
| gateway_ip | |
| host_routes | |
| id | 288d389b-33c9-4559-8ec8-98b4faac2274 |
| ip_version | 4 |
| ipv6_address_mode | |
| ipv6_ra_mode | |
| name | HA subnet tenant 328ffb467eb24c3595b549516b8acbd0 |
| network_id | 218b2842-1832-4ec6-a979-5efdf01453bf |
| tenant_id | |
+-------------------+------------------------------------------------------+
● network_type、physical_networkは
テナント用ネットワークと同じ
● 独立したsegmentation_idが振られる
● テナントIDは空
● /etc/neutron/neutron.confで変更可能
[DEFAULT]
l3_ha_net_cidr = 169.254.192.0/18

ルータ作成 (4)
● ハートビート用のポートの確認
# neutron port-list
+--------------------------------------+-------------------------------------------------+-------------------+--------------------------------------------------------------------------------------+
| id | name | mac_address | fixed_ips |
+--------------------------------------+-------------------------------------------------+-------------------+--------------------------------------------------------------------------------------+
| 00ab5988-dedd-421d-921a-61698de65155 | HA port tenant 328ffb467eb24c3595b549516b8acbd0 | fa:16:3e:7c:52:0d | {"subnet_id": "288d389b-33c9-4559-8ec8-98b4faac2274", "ip_address": "169.254.192.2"} |
| ef4a5a34-17a1-4008-916a-72ff19424b61 | HA port tenant 328ffb467eb24c3595b549516b8acbd0 | fa:16:3e:16:91:0f | {"subnet_id": "288d389b-33c9-4559-8ec8-98b4faac2274", "ip_address": "169.254.192.1"} |
+--------------------------------------+-------------------------------------------------+-------------------+--------------------------------------------------------------------------------------+
# for port_id in $(neutron port-list | awk '/HA port/ {print $2}'); do neutron port-show ${port_id}; done
+-----------------------+--------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------------------------------------------------------+
| allowed_address_pairs | |
| binding:host_id | juno-network1 |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| device_id | 6ef3ebbe-dcd2-4af3-a4d9-def49792f09b |
| device_owner | network:router_ha_interface |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "9dc14d95-7588-4a02-b5f8-5436b0669916", "ip_address": "169.254.192.2"} |
| id | 208ace4e-077c-4b23-86a3-d67a7d5ca612 |
| mac_address | fa:16:3e:99:54:58 |
| name | HA port tenant 328ffb467eb24c3595b549516b8acbd0 |
| network_id | ca16daee-6272-421a-ba9d-6859a10ffce4 |
| security_groups | |
| status | ACTIVE |
| tenant_id | |
+-----------------------+--------------------------------------------------------------------------------------+
+-----------------------+--------------------------------------------------------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------------------------------------------------------+
| allowed_address_pairs | |
| binding:host_id | juno-network2 |
| binding:profile | {} |
| binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": true} |
| binding:vif_type | ovs |
| binding:vnic_type | normal |
| device_id | 6ef3ebbe-dcd2-4af3-a4d9-def49792f09b |
| device_owner | network:router_ha_interface |
| extra_dhcp_opts | |
| fixed_ips | {"subnet_id": "9dc14d95-7588-4a02-b5f8-5436b0669916", "ip_address": "169.254.192.1"} |
| id | e6a1529c-6a52-418e-8507-7531d343a84a |
| mac_address | fa:16:3e:a3:bc:e6 |
| name | HA port tenant 328ffb467eb24c3595b549516b8acbd0 |
| network_id | ca16daee-6272-421a-ba9d-6859a10ffce4 |
| security_groups | |
| status | ACTIVE |
| tenant_id | |
+-----------------------+--------------------------------------------------------------------------------------+

keepalived (1)
● keepalived関連のファイル置き場
# ls -F /var/lib/neutron/
dhcp/ external/ ha_confs/ lock/ metadata_proxy=
● keepalived関連のファイルたち
# find /var/lib/neutron/ha_confs/ -type f
/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_master.sh
/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_backup.sh
/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_fault.sh
/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/keepalived.conf
/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state
/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid
/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid-vrrp
router id
状態変化に応じて
呼ばれるスクリプト
設定ファイル
master or backup

keepalived (2)
● 設定ファイル
# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/keepalived.conf
vrrp_sync_group VG_1 {
group {
VR_1
}
notify_master "/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_master.sh"
notify_backup "/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_backup.sh"
notify_fault "/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_fault.sh"
}
vrrp_instance VR_1 {
state BACKUP
interface ha-ef4a5a34-17
virtual_router_id 1
priority 50
nopreempt
advert_int 2
track_interface {
ha-ef4a5a34-17
}
virtual_ipaddress {
172.16.99.100/24 dev qg-a6ee9669-8f
}
virtual_ipaddress_excluded {
172.16.99.101/32 dev qg-a6ee9669-8f
192.168.10.1/24 dev qr-3d8666f0-c4
}
virtual_routes {
0.0.0.0/0 via 172.16.99.1 dev qg-a6ee9669-8f
}
}
● パブリックネットワークの
ゲートウェイ
● Floating IP
● テナントネットワークの
ゲートウェイ

keepalived (3)
● 通知スクリプトたち
マスターになったら
metadata proxyも起動する
# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_backup.sh
#!/usr/bin/env bash
kill -9 $(cat /var/lib/neutron/external/pids/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid)
echo -n backup > /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state
# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_fault.sh
#!/usr/bin/env bash
kill -9 $(cat /var/lib/neutron/external/pids/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid)
echo -n fault > /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state
# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_master.sh
#!/usr/bin/env bash
neutron-ns-metadata-proxy --pid_file=/var/lib/neutron/external/pids/e7e98ef7-36d5-4544-be07-c83ddf7e67ea.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy --router_id=e7e98ef7-36d5-4544-be07-c83ddf7e67ea
--state_path=/var/lib/neutron --metadata_port=9697 --verbose --log-file=neutron-ns-metadata-proxy-e7e98ef7-
36d5-4544-be07-c83ddf7e67ea.log --log-dir=/var/log/neutron
echo -n master > /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state
マスターかバックアップかを
stateファイルに書いておく

keepalived (4)
● keepalivedはプロセスが2
個見える
– VRRP用に1個 fork(2) する
– 親はVRRP用の子供を監視
して、死んでいたら
respawnする
# pgrep keepalived
2948
5130
# pstree -p -N net
[4026531956]
systemd(1)─┬─agetty(908)
├─agetty(909)
├─auditd(594)───{auditd}(605)
├─avahi-daemon(624)───avahi-daemon(635)
├─chronyd(674)
├─crond(701)
├─dbus-daemon(680)───{dbus-daemon}(698)
├─httpd(1189)─┬─httpd(1553)
│ ├─httpd(1554)
│ ├─httpd(1555)
│ ├─httpd(1556)
│ ├─httpd(1557)
│ ├─httpd(1558)
│ ├─httpd(1559)
│ └─httpd(1560)
├─iprdump(745)
├─iprinit(668)
├─iprupdate(671)
├─irqbalance(644)
├─lvmetad(471)
├─master(1676)─┬─pickup(27341)
│ └─qmgr(1690)
├─monitor(704)───ovsdb-server(705)
├─monitor(727)───ovs-vswitchd(729)─┬─{ovs-vswitchd}(732)
│ ├─{ovs-vswitchd}(733)
│ ├─{ovs-vswitchd}(734)
│ └─{ovs-vswitchd}(735)
├─neutron-dhcp-ag(2291)
├─neutron-l3-agen(2290)
├─neutron-metadat(1192)─┬─neutron-metadat(2167)
│ └─neutron-metadat(2168)
├─neutron-meterin(1191)
├─neutron-openvsw(2293)───sudo(2378)───neutron-rootwra(2380)───ovsdb-client(2382)
├─nrpe(1198)
├─rsyslogd(626)─┬─{rsyslogd}(646)
│ └─{rsyslogd}(647)
├─sshd(1199)───sshd(1842)───bash(1969)───pstree(28359)
├─systemd-journal(470)
├─systemd-logind(679)
├─systemd-udevd(473)
└─tuned(642)─┬─{tuned}(774)
├─{tuned}(775)
├─{tuned}(776)
└─{tuned}(788)
[4026532345]
dnsmasq(3834)
[4026532262]
keepalived(2948)───keepalived(5130)
neutron-ns-meta(4043)

マスタールータの確認
● VIPがついている方がマスター
juno-network1 juno-network2
# ip netns list
qrouter-e7e98ef7-36d5-4544-be07-c83ddf7e67ea
# ip netns list
qdhcp-802e69c8-b064-4eb1-90b9-327e712472d8
qrouter-e7e98ef7-36d5-4544-be07-c83ddf7e67ea
ip netns exec qrouter-e7e98ef7-36d5-4544-be07-c83ddf7e67ea ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
9: ha-ef4a5a34-17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:16:91:0f brd ff:ff:ff:ff:ff:ff
inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-ef4a5a34-17
inet6 fe80::f816:3eff:fe16:910f/64 scope link
10: qg-a6ee9669-8f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:7a:d9:16 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f816:3eff:fe7a:d916/64 scope link
11: qr-3d8666f0-c4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:43:43:8c brd ff:ff:ff:ff:ff:ff
inet6 fe80::f816:3eff:fe43:438c/64 scope link
ip netns exec qrouter-e7e98ef7-36d5-4544-be07-c83ddf7e67ea ip addr show
9: ha-00ab5988-de: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:7c:52:0d brd ff:ff:ff:ff:ff:ff
inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-00ab5988-de
inet6 fe80::f816:3eff:fe7c:520d/64 scope link
inet 172.16.99.100/24 scope global qg-a6ee9669-8f
inet6 fe80::f816:3eff:fe7a:d916/64 scope link tentative dadfailed
inet 192.168.10.1/24 scope global qr-3d8666f0-c4
[root@juno-network1 ~]# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-
c83ddf7e67ea/state; echo
backup
[ro[root@juno-network2 ~]# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-
c83ddf7e67ea/state; echo
master
● ここにも書いてある

フェイルオーバー
● 初期状態
– master: juno-network1
– backup: juno-network2
● keepalivedを殺す
[root@juno-network1 ~]# date; killall keepalived
Mon Dec 1 12:51:59 JST 2014
[root@juno-controller rdo-juno(keystone_demo)]# nova show vm1
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2014-12-01T03:37:44.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2014-12-01T03:37:01Z |
| demo_net network | 192.168.10.100, 172.16.99.101 |
| flavor | m1.tiny (1) |
| hostId | 826d63fe7c11232775d45dd1edc8bf9db15b01d01ce908c143434750 |
| id | da10da5e-6b61-40f7-b5ab-18409b4d4267 |
| image | cirros (69d97e8f-24b5-4f0b-8977-7295d8ba8edb) |
| key_name | sshkey |
| metadata | {} |
| name | vm1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | ACTIVE |
| updated | 2014-12-01T03:37:44Z |
| user_id | 01c4dc5c8ec0494eaaecb84fd4170556 |
+--------------------------------------+----------------------------------------------------------+
[root@juno-controller rdo-juno(keystone_demo)]# nova ssh --login cirros vm1
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether fa:16:3e:be:02:20 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.100/24 brd 192.168.10.255 scope global eth0
inet6 fe80::f816:3eff:febe:220/64 scope link
$ ping 172.16.99.11
PING 172.16.99.11 (172.16.99.11): 56 data bytes
64 bytes from 172.16.99.11: seq=0 ttl=63 time=2.659 ms
^C
--- 172.16.99.11 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.687/1.442/2.659 ms
$ exit
Connection to 172.16.99.101 closed.
[root@juno-controller rdo-juno(keystone_demo)]#

● OSのログ
juno-network1
12月 01 12:51:59 juno-network1 Keepalived[3668]: Stopping Keepalived v1.2.10 (06/10,2014)
12月 01 12:51:59 juno-network1 Keepalived_vrrp[5025]: VRRP_Instance(VR_1) sending 0 priority
juno-network2
12月 01 12:52:00 juno-network2 Keepalived_vrrp[5038]: VRRP_Instance(VR_1) Transition to MASTER STATE
12月 01 12:52:00 juno-network2 Keepalived_vrrp[5038]: VRRP_Group(VG_1) Syncing instances to MASTER state
12月 01 12:52:00 juno-network2 Keepalived_vrrp[5038]: Opening script file
/var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/notify_master.sh
12月 01 12:52:02 juno-network2 Keepalived_vrrp[5038]: VRRP_Instance(VR_1) Entering MASTER STATE

● VMへの疎通
[Mon Dec 1 12:51:52 JST 2014] 64 bytes from 172.16.99.101: icmp_seq=417 ttl=62 time=1.13 ms

フェイルオーバー前
● ip netns
[root@juno-network1 ~]# ip netns exec $(ip netns list | grep qrouter) ip a
[root@juno-network1 ~]# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state
master
[root@juno-network2 ~]# cat /var/lib/neutron/ha_confs/e7e98ef7-36d5-4544-be07-c83ddf7e67ea/state
backup

フェイルオーバー後
● ip netns

● VRRP
12:51:51.991662 IP 169.254.192.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 50, authtype none, intvl 2s, length 20

● Gratuitous ARP
12:52:02.476000 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:02.476075 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:02.476121 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:02.476133 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:02.476144 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477325 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477376 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477403 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477429 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:52:07.477439 fa:16:3e:43:43:8c (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 2002, p 0, ethertype ARP, Request who-has 192.168.10.1 (Broadcast) tell 192.168.10.1, length 12:51:59.679351 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28
12:51:59.679373 fa:16:3e:7a:d9:16 (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Request who-has 172.16.99.101 (Broadcast) tell 172.16.99.101, length 28

● network nodes (external)
12:52:02.475961 ARP, Request who-has 172.16.99.100 (Broadcast) tell 172.16.99.100, length 28
12:52:03.331280 IP 172.16.99.254 > 172.16.99.101: ICMP echo request, id 6456, seq 428, length 64
12:52:03.334090 ARP, Request who-has 172.16.99.254 tell 172.16.99.101, length 28
12:52:03.334767 ARP, Reply 172.16.99.254 is-at 52:54:00:9e:a3:85 (oui Unknown), length 28
12:52:03.335070 IP 172.16.99.101 > 172.16.99.254: ICMP echo reply, id 6456, seq 428, length 64
juno-network1
juno-network2

● network nodes (tenant)
12:51:59.675765 IP 169.254.192.1 > 224.0.0.22: igmp v3 report, 1 group record(s)
12:52:03.332823 ARP, Reply 192.168.10.100 is-at fa:16:3e:be:02:20 (oui Unknown), length 28

素朴な疑問 (1)
● Q: non-HA構成で作ったルータをHA構成に変更
できる？
● A: できませんでした
# neutron router-create --tenant-id $(keystone tenant-list | awk '/demo/ {print $2}') --ha False router_demo
Created a new router:
+-----------------------+--------------------------------------+
| Field | Value |
+-----------------------+--------------------------------------+
| distributed | False |
| external_gateway_info | |
| ha | False |
| id | f747119c-6183-4ae4-b6a5-397628a3cc4a |
| name | router_demo |
| routes | |
| status | ACTIVE |
+-----------------------+--------------------------------------+
# neutron router-list
+--------------------------------------+-------------+-----------------------+-------------+-------+
| id | name | external_gateway_info | distributed | ha |
+--------------------------------------+-------------+-----------------------+-------------+-------+
| f747119c-6183-4ae4-b6a5-397628a3cc4a | router_demo | null | False | False |
+--------------------------------------+-------------+-----------------------+-------------+-------+
[root@juno-controller rdo-juno(keystone_admin)]# neutron router-update router_demo --ha True
Bad Request (HTTP 400) (Request-ID: req-f7f3a8db-cb4e-4f2e-b3e1-ab99809fb15c)
2014-11-30 23:19:41.715 3526 INFO neutron.wsgi [-] (3526) accepted ('10.0.1.111', 54480)
2014-11-30 23:19:41.776 3526 INFO neutron.wsgi [req-f2181436-4a6c-4ab8-b63d-f12c5770d6f2 None] 10.0.1.111 - - [30/Nov/2014 23:19:41] "GET
/v2.0/routers.json?fields=id&name=router_demo HTTP/1.1" 200 250 0.059557
2014-11-30 23:19:41.785 3526 ERROR neutron.api.v2.resource [req-f7f3a8db-cb4e-4f2e-b3e1-ab99809fb15c None] update failed
2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource Traceback (most recent call last):
2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/resource.py", line 87, in resource
2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource result = method(request=request, **args)
2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 501, in update
2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource allow_bulk=self._allow_bulk)
2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource File "/usr/lib/python2.7/site-packages/neutron/api/v2/base.py", line 623, in
prepare_request_body
2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource raise webob.exc.HTTPBadRequest(msg)
2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource HTTPBadRequest: Cannot update read-only attribute ha
2014-11-30 23:19:41.785 3526 TRACE neutron.api.v2.resource

素朴な疑問 (2)
● Q: マスタールータはどっち？
● A: VIPがついている方
● でもたまに動きがあやしい...
● この辺りから総合的に判断するのが確実
– VIPがついている
– /var/lib/neutron/ROUTER_ID/stateに "master" と書かれて
いる
– keepalivedのプロセスが見える

素朴な疑問
● Q: セッションの引き継ぎ？
● A: 今後conntrackdと組み合わせてやる予定
– Wikiにはconntrackdの設定テンプレートとかが
載っていていかにもできそうですが、まだできませ
ん
● https://wiki.openstack.org/wiki/Neutron/L3_High_Avai
lability_VRRP
– Specには "Phase 2でやるよ" と書いてあります
● https://github.com/openstack/neutron-specs/blob/maste
r/specs/juno/l3-high-availability.rst

素朴な疑問
● HAルータを2個作ると？
[root@juno-network1 ~]# for ns in $(ip netns list); do echo "=> ${ns}"; ip netns exec ${ns}
ip a; done
=> qrouter-767318c2-8dfa-4713-9018-27ef072383dd
12: ha-12860419-7f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:65:92:ed brd ff:ff:ff:ff:ff:ff
inet 169.254.192.4/18 brd 169.254.255.255 scope global ha-12860419-7f
inet6 fe80::f816:3eff:fe65:92ed/64 scope link
13: qg-6214b83a-d7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:bc:8d:2f brd ff:ff:ff:ff:ff:ff
inet 172.16.99.101/24 scope global qg-6214b83a-d7
inet6 fe80::f816:3eff:febc:8d2f/64 scope link tentative dadfailed
14: qr-56c2e065-77: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:66:8d:63 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.1/24 scope global qr-56c2e065-77
inet6 fe80::f816:3eff:fe66:8d63/64 scope link
=> qrouter-b0402579-0a6e-42c3-8d3b-33af9a51d3cf
9: ha-51456562-ca: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:7b:10:1d brd ff:ff:ff:ff:ff:ff
inet 169.254.192.2/18 brd 169.254.255.255 scope global ha-51456562-ca
inet6 fe80::f816:3eff:fe7b:101d/64 scope link
10: qg-5f3287c0-e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:e7:7d:5c brd ff:ff:ff:ff:ff:ff
inet6 fe80::f816:3eff:fee7:7d5c/64 scope link
11: qr-48ea3761-b0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:ab:52:f0 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f816:3eff:feab:52f0/64 scope link
[root@juno-network2 ~]# for ns in $(ip netns list); do echo "=> ${ns}"; ip netns exec ${ns}
ip a; done
=> qrouter-767318c2-8dfa-4713-9018-27ef072383dd
12: ha-fe1f12b3-8c: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:ae:78:91 brd ff:ff:ff:ff:ff:ff
inet 169.254.192.3/18 brd 169.254.255.255 scope global ha-fe1f12b3-8c
inet6 fe80::f816:3eff:feae:7891/64 scope link
13: qg-6214b83a-d7: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:bc:8d:2f brd ff:ff:ff:ff:ff:ff
inet6 fe80::f816:3eff:febc:8d2f/64 scope link
14: qr-56c2e065-77: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:66:8d:63 brd ff:ff:ff:ff:ff:ff
inet6 fe80::f816:3eff:fe66:8d63/64 scope link
=> qrouter-b0402579-0a6e-42c3-8d3b-33af9a51d3cf
9: ha-1dff8969-fc: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:db:a4:68 brd ff:ff:ff:ff:ff:ff
inet 169.254.192.1/18 brd 169.254.255.255 scope global ha-1dff8969-fc
inet6 fe80::f816:3eff:fedb:a468/64 scope link
10: qg-5f3287c0-e0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:e7:7d:5c brd ff:ff:ff:ff:ff:ff
inet 172.16.99.100/24 scope global qg-5f3287c0-e0
inet6 fe80::f816:3eff:fee7:7d5c/64 scope link tentative dadfailed
11: qr-48ea3761-b0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether fa:16:3e:ab:52:f0 brd ff:ff:ff:ff:ff:ff
inet 192.168.10.1/24 scope global qr-48ea3761-b0
inet6 fe80::f816:3eff:feab:52f0/64 scope link

Packstackのハマりどころ
● Cinder
– Bug 1164359 - KeyError: 'CONFIG_CINDER_VOLUMES_SIZE'
● https://bugzilla.redhat.com/show_bug.cgi?id=1164359
– Fix LVM Cinder Volume Creation
● https://review.openstack.org/#/c/131803/
– Avoid KeyError: 'CONFIG_CINDER_VOLUMES_SIZE' error
● https://review.openstack.org/#/c/136766/
● mongodb
– Bug 1167888 - [CENTOS7] packstack all in one install fails at MongoDB
Service[mongodb]: Execution of '/usr/bin/systemctl start mongod' returned
1: Job for mongod.service failed
● https://bugzilla.redhat.com/show_bug.cgi?id=1167888
– mongodb.pp error when installing RDO on CentOS 7
● https://ask.openstack.org/en/question/54015/mongodbpp-error-when-installing-rdo
-on-centos-7/

SELinux問題
● SELinux=enforcingの場合...
– HA Routerを起動するとエラーになります
2014-11-29 15:28:11.800 2269 ERROR neutron.agent.l3_agent [-]
Command: ['sudo', 'neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'ip', 'netns', 'exec', 'qrouter-
002a56ba-beda-43ad-a5d6-adff4dc1a725', 'keepalived', '-P', '-f', '/var/lib/neutron/ha_confs/002a56ba-beda-
43ad-a5d6-adff4dc1a725/keepalived.conf', '-p', '/var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6-
adff4dc1a725.pid', '-r', '/var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6-adff4dc1a725.pid-vrrp']
Exit code: 99
Stdout: ''
Stderr: '/usr/bin/neutron-rootwrap: Unauthorized command: ip netns exec qrouter-002a56ba-beda-43ad-a5d6-
adff4dc1a725 keepalived -P -f /var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6-
adff4dc1a725/keepalived.conf -p /var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6-adff4dc1a725.pid
-r /var/lib/neutron/ha_confs/002a56ba-beda-43ad-a5d6adff4dc1a725.pid-vrrp (no filter matched)n'
"no filter matched" と言われている
→ 一見rootwrap関連の不具合にみえる
→ だが実は...

SELinux問題
● SELinux関連が原因だったのでした
– keepalivedの起動に失敗している
– permissiveにすると問題なし
# ausearch -m avc
----
time->Sat Nov 29 15:28:11 2014
type=SYSCALL msg=audit(1417242491.790:1992): arch=c000003e syscall=21 success=no exit=-13 a0=1b17a00
a1=1 a2=7fc4466fcf88 a3=7fff3728eb20 items=0 ppid=3390 pid=3391 auid=4294967295 uid=0 gid=0 euid=0
suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="neutron-rootwra"
exe="/usr/bin/python2.7" subj=system_u:system_r:neutron_t:s0 key=(null)
type=AVC msg=audit(1417242491.790:1992): avc: denied { execute } for pid=3391 comm="neutron-rootwra"
name="keepalived" dev="dm-1" ino=9038320 scontext=system_u:system_r:neutron_t:s0
tcontext=system_u:object_r:keepalived_exec_t:s0 tclass=file
(snip)
avc: access vector cache

参考文献
● Blueprint
– https://blueprints.launchpad.net/neutron/+spec/l3-high-availability
● Spec
– https://github.com/openstack/neutron-specs/blob/master/specs/juno/l3-high-availability
.rst
● Neutron/L3 High Availability VRRP
– https://wiki.openstack.org/wiki/Neutron/L3_High_Availability_VRRP
● Atlanta Summit: High Availability in Neutron – Getting the L3 Agent Right
– https://www.youtube.com/watch?v=8kBPlIBq0Dc
– http://techs.enovance.com/wp-content/uploads/2014/05/Neutron-L3-HA-Summit-eNova
nce-template.pdf
● Paris Summit: Neutron Network Node High Availability
– https://www.youtube.com/watch?v=go4fOYOUkmE
– https://assafmuller.files.wordpress.com/2014/11/l3-ha.pdf
● Layer 3 High Availability
– http://assafmuller.com/2014/08/16/layer-3-high-availability/
● Keepalived Documentation
– http://www.keepalived.org/documentation.html

L3HA-VRRP-20141201

Recommandé

Recommandé

Contenu connexe

Tendances

Tendances (20)

En vedette

En vedette (20)

Similaire à L3HA-VRRP-20141201

Similaire à L3HA-VRRP-20141201 (20)

Dernier

Dernier (20)

L3HA-VRRP-20141201