Contenu connexe
Similaire à OpenStack Neutron's Distributed Virtual Router (20)
OpenStack Neutron's Distributed Virtual Router
- 1. Distributed Virtual Router
Introduced in the Juno Release of OpenStack Neutron
Carl Baldwin
DVR Illustrations courtesy of Jack McCann
© Copyright 2013 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 2. DVR Architecture
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 3. Neutron deployment architecture without DVR
neutron-server
API
auth
ML2 plug-in
database
Network Service Node(s)
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 3 to change without notice.
Compute hosts
DHCP agent
L3 agent
message
queue
Metadata agent
ovs agent
Nova metadata
ovs agent
Open
vSwitch
Open
vSwitch
- 4. Neutron deployment architecture with DVR
neutron-server
API
auth
ML2 plug-in
database
Metadata agent
L3 agent
message
queue
Network Service Node(s)
Compute hosts
DHCP agent
Metadata agent Nova metadata
L3 agent agent_mode = dvr
ovs agent
w/dvr agent
Nova metadata
ovs agent
Open
vSwitch
Open
vSwitch
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 4 to change without notice.
enable_distibuted_routing = True
programs DVR flow handling
external network
compute nodes on external
network
- 5. API extension
Adds ‘distributed’ attribute to ‘router’ object
• Can be set by admin user through the API
• Global default is set as “router_distributed” in neutron.conf
• Default is False
• The attribute is only visible to admin tenant in GET
• Cannot be updated
• Work in progress to allow update from False to True
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 5 to change without notice.
- 6. DVR – East-West (subnet-to-subnet)
“VM1-1
QRouter-X
S1.1 S2.1
br-int
patch-tun
eth0
ARP for gw
kept local
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 6 to change without notice.
“VM2-1
br-int
patch-tun
eth0
QRouter-X
S1.1 S2.1
same gw IP/MAC
on each node
no remote bcast
in to routers
- 7. DVR – North-South (floating IP)
“VM1-1
“VM2-1
br-int
QRouter-Y
qr rfp-y
floating IP namespace
br-ex
external-vlan
eth0
QRouter-X
qr rfp-x
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 7 to change without notice.
floating IP NAT in router ns
default route via FIP-NS
local addressing connects QR to FIP-NS
floating IP host routes pointing to QR
proxy-arp for floating IPs on br-ex
(future option: BGP route injection)
fpr-x fpr-y
fg-u
- 8. DVR – North-South (default SNAT)
br-int
snat-Y
qr qg-x
br-ex
external-vlan
eth0
snat-X
qr qg-x
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 8 to change without notice.
default SNAT in snat namespace
default route via br-ex
- 9. Database
router_extra_attributes
router_id string uuid
distributed boolean
dvr_host_macs
host string 255
mac_address string 32
ml2_dvr_port_bindings - port binding for all the
ports associated to a DVR identified by router_id
port_id string uuid
host string
router_id string uuid
vif_type string
vif_details string
vnic_type string
profile string
cap_port_filter boolean
driver string
segment string
status string
csnat_l3_agent_bindings
router_id string uuid
l3_agent_id string uuid
host_id string
csnat_gw_port_id string uuid
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 9 to change without notice.
- 10. config file options
neutron.conf
[DEFAULT]
router_distributed = False
dvr_base_mac = fa:16:3f:00:00:00
ovs_neutron_plugin.ini
[agent]
enable_distributed_routing = False # Make the l2 agent run in dvr mode
l3_agent.ini
[DEFAULT]
agent_mode = legacy # legacy, dvr, or dvr_snat
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 10 to change without notice.
- 11. OVS Flow Handling
How to Distribute the Router’s Internal Port
• ARP Requests to Router Port are Blocked from the Tunnel
• These ARP requests should only be seen by the local port
• Source Mac is Mapped to Host Mac on Overlay Network
• All traffic generated by the
• A mac address is allocated for each compute host
• Mapping must be done on both ends of the tunnel
• Destination Mac Blocked from Overlay
• These should go to the local port
• They would create mac ambiguity in the overly
• L2 Pre-Population is Required
• “Prevent(s) multiple unicast of routed packets destined to remove VMs.”
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 11 to change without notice.
- 12. DVR Limitations
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 13. Default SNAT still centralized
snat-Y
qr qg-x
br-ex
external-vlan
eth1
“VM1-1
qrouter-X
S1.1 S2.1
br-int
patch-tun
eth0
br-int
patch-tun
eth0
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 13 to change without notice.
- 14. Floating IP Namespace
• Pros
• Eliminates Need for Public Address/IR
• Keeps IR Macs Off External Net
• Cons
• Extra Complexity in L3 Agent
• Still Consumes a Public Address / CN
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 14 to change without notice.
QRouter-Y
qr rfp-y
fpr-x fpr-y
floating IP namespace
fg-u
eth0
QRouter-X
qr rfp-x
br-ex
external-vlan
- 15. Heavy L2 Integration
• Led to Initial Dependence on OVS and Tunnel Protocols
• Mitigate Scope Creep
• Distributed Port Concept Needs to be Abstracted
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 15 to change without notice.
- 16. Firewall as a Service (FWaaS) Complexity
• External Net Connects to Hypervisors
• FWaaS is Needed There Too.
• Asymmetric Routing Problem
• E/W Routing
“VM1-1
QRouter-X
S1.1 S2.1
br-int
patch-tun
eth0
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 16 to change without notice.
“VM2-1
br-int
patch-tun
eth0
QRouter-X
S1.1 S2.1
- 17. Contributing DVR
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
- 18. Initial Development on Havana
Pros and Cons
• Stable Code Base
• No Risk of External Regression
• Very Large Effort to Integrate
• Upstream Moves Quickly
• Subject to Regression
• Comm. Standard Enforcment
• Code Style
• No Demand for Unit or Functional Tests
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 18 to change without notice.
- 19. Initial Development on Havana
If We had to do it Over Again…
• Contribute Smaller, More Focused Patches to Trunk
• Start with Pure Refactoring Needed to Ease Development
• Develop Unit Tests for Code that will be Modified
• Move Gradually Beyond Refactoring to Other Improvements
• Divide Implementation According to Themes
• Develop Unit Tests (TDD) and Functional Tests to Prevent Regression
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 19 to change without notice.
- 20. Divided in to 7 Patches
• Division According to Component
• Patches Added Unused Code for Later Patches
• Indicates there are themes that cross patch boundaries
• Each Patch Had Multiple Active Authors
• Indicates possibly more than one theme in the patch
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 20 to change without notice.
- 21. Handling Multiple Changes
• Dependency Order Not Linear
• Should it be?
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 21 to change without notice.
- 22. Handling Multiple Changes
It is Never Linear!!!
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 22 to change without notice.
- 23. Handling Multiple Changes
Enemy Number One!
• Automatic Rebase Feature
• Default behavior of “git review”
• Most of the Time it is Disruptive
• Sometimes it is Destructive
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 23 to change without notice.
- 24. Handling Multiple Changes
Enemy Number One!
• Clobbered API Extension!!
• Happened More than Once
• Use --no-rebase Always
• Rebase on Merge Conflict
• Work from the bottom up
• Merge Faster
• Smaller, more focused patches
• Continuous community involvement
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject 24 to change without notice.