This document discusses troubleshooting VXLAN and network services in a virtualized environment using VMware NSX. It covers VXLAN packet flow, NSX enhancements to the data and control planes, configuration and consumption demos, packet walks in unicast mode, troubleshooting demos using NSX Manager tools, dynamic routing details and demos, and network virtualization operations. The key takeaways are that multicast is not required in the physical network for VXLAN, NSX provides tools to troubleshoot networks and services, and NSX integrates with operations tools for analysis and alerting.

1. Troubleshooting VXLAN and Network Services in a
Virtualized Environment
Vyenkatesh (Venky) Deshpande, VMware
Sachin Thakkar, VMware
NET5654
#NET5654

2. 2
Objectives of the Session
 Explain VXLAN Packet Flow in a Controller based environment
 Introduce various tools and commands that help monitor and
troubleshoot network and network services.

3. 3
Recommended Sessions & Labs
 NET5790 – Operational Best Practices for NSX in VMware
Environments
 SEC5894 - Deploying, Troubleshooting, and Monitoring VMware
NSX Distributed Firewall
 NET5266 – Bringing Network Virtualization to VMware
Environments with NSX
 NET5584 – Deploying VMware NSX Network Virtualization
 You can actually test all the workflows in the lab HOL-SDC-1303

4. 4
Agenda
 VMware NSX Overview
 VXLAN
 Enhancements – Data and Control Plane
 Configuration Demo
 Packet Walk
 Troubleshooting Demo
 Dynamic Routing
 Enhancements
 Routing Demo
 Network Virtualization Operations
 Demo

5. 5
VMware NSX – Networking & Security Capabilities
Any Application
(without modification)
Virtual Networks
VMware NSX Network Virtualization Platform
Logical L2
Any Network Hardware
Any Cloud Management Platform
Logical
Firewall
Logical
Load Balancer
Logical L3
Logical
VPN
Any Hypervisor
Logical Switching – Layer 2 over Layer 3,
decoupled from the physical network
Logical Routing – Routing between virtual
networks without exiting the software
container
Logical Firewall – Distributed Firewall,
Kernel Integrated, High Performance
Logical Load Balancer – Application Load
Balancing in software
Logical VPN – Site-to-Site & Remote
Access VPN in software
NSX API – RESTful API for integration into
any Cloud Management Platform
Partner Eco-System

6. 6
VMware NSX Functional System Overview
vSphere vSphere vSphere vSphere
vSwitch vSwitch vSwitch vSwitch
Hosts
Data Plane
Operations
UI
Logs/Stats
CMP
Consumption
Tenant UI
API
Control Plane Run-time state
Management Plane
API
API, config, etc.
HA, scale-out
NSX Manager
NSX Controller

7. 7
VXLAN

8. 8
VXLAN Protocol Overview
 Ethernet in IP overlay network
 Entire L2 frame encapsulated in
UDP
 50+ bytes of overhead
 Decouples Physical network
from the Logical
 VMs do NOT see VXLAN ID
 Physical Network devices don’t see
VMs MAC and IP address
 VTEP (VXLAN Tunnel End
Point)
 VMkernel interface which serves as
the endpoint for encapsulation/de-
encapsulation of VXLAN traffic
 VXLAN can cross Layer 3
network boundaries
 Technology submitted to IETF
for standardization
• With Cisco, Citrix, Red Hat,
Broadcom, Arista and Others

9. 9
Inner
Dest
MAC
Inner
Source
MAC
Optional
Ether
Type
Optional
Inner
802.1Q
Original
Ethernet
Payload
Inner Ethernet Frame
Outer
Dest
MAC
Outer
Source
MAC
Optional
VXLAN
Type
Optional
Outer
802.1Q
IP
Header
Data*
IP
Proto
-col
Header
Check
Sum
Outer
Source
IP
Source
Port
Dest Port
(8472)
UDP
Length
UDP
Check
Sum
VXLAN
Flags
RSVD
VXLAN
NI
(VNI)
FCS
RSVD
VXLAN Encapsulated Frame
Outer
Ethernet
Header
14 bytes
Outer IP Header
20 bytes
Outer
UDP
Header
8 bytes
VXLAN
Header
8 bytes
Ether
Type
Outer
Dest
IP
*IP Header Data = Version, IHL, TOS, Length, ID
VXLAN Frame Format

10. 10
NSX VXLAN Enhancements – Data Plane
 Support for multiple VXLAN vmknics
per host to provide additional
options for uplink load balancing
 DSCP & COS Tag from internal frame
copied to external VXLAN
encapsulated header
 Support for Guest VLAN tagging
 Dedicated TCP/IP stack for VXLAN
and Static IP addressing for VTEPs
vSphere Cluster B
UWA VTEP
UWA VTEP
UWA VTEP
vSphere Cluster A
UWA VTEP
UWA VTEP
UWA VTEP

11. 11
NSX VXLAN Enhancements – Control Plane
 A highly available and secure
control plane to distribute VXLAN
network information to vSphere
hosts
 Removes dependency on multicast
routing/PIM in the physical network
 Suppress broadcast traffic in VXLAN
networks
• ARP Directory Service & Cache
Controller
Cluster
Controller
VXLAN Directory
Service
MAC table
ARP table
VTEP table

12. 12
vSphere Cluster B
UWA VTEP
UWA VTEP
UWA VTEP
NSX VXLAN – Management Plane
 NSX Manager deploys Controllers
and prepares vSphere Clusters for
VXLAN
 After the configuration any new
logical networks are created
through NSX manager
 Platform requirements: vSphere
(vCenter and ESXi) 5.5 and VDS
5.5
NSX Manager
Controller
Cluster
vSphere Cluster A
UWA VTEP
UWA VTEP
UWA VTEP

13. 13
Configuration Demo
 Simplified configuration
 Prepare Cluster
 Expand Transport zone

14. 14
VXLAN NSX for vSphere – Multicast Mode
vSphere Host
VM1
vSphere Distributed Switch
VXLAN Transport Network
vSphere Host
VM2
vSphere Host
VXLAN 5001
VTEP1 10.20.10.10 VTEP2 10.20.10.11 VTEP3 10.20.11.10
vSphere Host
VTEP4 10.20.11.11
VM3 VM4
Multicast Traffic
L3 - PIML2 - IGMP L2 - IGMP
VXLAN Transport Subnet A 10.20.10.0/24 VXLAN Transport Subnet B 10.20.11.0/24

15. 15
VXLAN NSX for vSphere – Unicast Mode
vSphere Host
VM1
vSphere Distributed Switch
VXLAN Transport Network
vSphere Host
VM2
vSphere Host
VXLAN 5001
VTEP1 10.20.10.10 VTEP2 10.20.10.11 VTEP3 10.20.11.10
vSphere Host
VTEP4 10.20.11.11
VM3 VM4
Unicast Traffic
Controller
Cluster
VXLAN Transport Subnet A 10.20.10.0/24 VXLAN Transport Subnet B 10.20.11.0/24

16. 16
VXLAN NSX for vSphere – Hybrid Mode
vSphere Host
VM1
vSphere Distributed Switch
VXLAN Transport Network
vSphere Host
VM2
vSphere Host
VXLAN 5001
VTEP1 10.20.10.10 VTEP2 10.20.10.11 VTEP3 10.20.11.10
vSphere Host
VTEP4 10.20.11.11
VM3 VM4
Unicast Traffic
L2 - IGMP L2 - IGMP
Controller
Cluster
Multicast Traffic
VXLAN Transport Subnet A 10.20.10.0/24 VXLAN Transport Subnet B 10.20.11.0/24

17. 17
Consumption Demo
 Create Logical Networks in few Clicks
 Logical network
 Connect Virtual Machines
 Ability to configure advanced services on the logical network

18. 18
Unicast Mode Packet Walk

19. 19
VXLAN Packet Flow – VTEP Report
vSphere Host
VM
vSphere Distributed Switch
MAC1
Management
Network
10.20.10.10
vSphere Host
VM
MAC2
vSphere Host
Send VNI,VTEP
Mapping to
Controller
1
2
5
10
VXLAN 5001
3
VNI VTEP IP
5001 10.20.10.10
9
IP1 IP2
Controller
VNI VTEP IP
5001 10.20.10.10
VNI VTEP IP
5001 10.20.10.11
10.20.10.11 10.20.10.12
11
6
7
VNI VTEP IP
5001
10.20.10.10
10.20.10.11
VNI VTEP IP
5001
10.20.10.10
10.20.10.11
48
10
VNI VTEP IP
5001
10.20.10.10
10.20.10.11
11
Report the new
VNI,VTEP
Mapping to the
Hosts

20. 20
VXLAN Packet Flow – MAC Report
vSphere Host
VM
vSphere Distributed Switch
MAC1
Management
Network
10.20.10.10
vSphere Host
VM
MAC2
vSphere Host
Send VNI,VM
MAC Mapping
and VTEP IP to
Controller
1
2
5 VXLAN 5001
3
IP1 IP2
Controller
VNI VM MAC
5001 MAC1
VNI VM MAC
5001 MAC2
10.20.10.11 10.20.10.12
6
7
48
VNI VM MAC VTEP
5001 MAC1 10.20.10.10
VNI VM MAC VTEP
5001 MAC1 10.20.10.10
5001 MAC2 10.20.10.11

21. 21
VXLAN Packet Flow – IP Report
vSphere Host
VM
vSphere Distributed Switch
MAC1
Management
Network
10.20.10.10
vSphere Host
VM
MAC2
vSphere Host
Send VM MAC,
IP Mapping and
VNI to Controller
1
2
5 VXLAN 5001
3
IP1 IP2
Controller
VNI
VM
IP
VM
MAC
5001 IP1 MAC1
VNI
VM
IP
VM
MAC
5001 IP2 MAC2
10.20.10.11 10.20.10.12
6
7
48
VNI VM IP VM MAC
5001 IP1 MAC1
VNI VM IP VM MAC
5001 IP1 MAC1
5001 IP2 MAC2

22. 22
VXLAN Packet Flow – ARP Request
vSphere Host
VM
vSphere Distributed Switch
MAC1
Management
Network
10.20.10.10
vSphere Host
VNI
VM
IP
VM
MAC
VTEP
5001 IP1 MAC1 10.20.10.10
VM
MAC2
vSphere Host
1
2
VXLAN 5001
3
IP1 IP2
Controller
10.20.10.11 10.20.10.12
6 5
VNI
VM
IP
VM
MAC
VTEP
5001 IP1 MAC1 10.20.10.10
5001 IP2 MAC2 10.20.10.11
4
PayloadL2
DA: Broadcast
SA: MAC1
ARP Request for
VM IP2 sent to
Controller
ARP Report for
VM IP2, MAC2
sent to VTEP
10.20.10.10
VNI
VM
IP
VM
MAC
VTEP
5001 IP1 MAC1 10.20.10.10
5001 IP2 MAC2 10.20.10.11

23. 23
VXLAN Packet Flow – Communication After ARP Resolution
vSphere Host
VM
vSphere Distributed Switch
MAC1
VXLAN Transport
Network
10.20.10.10
vSphere Host
VM
MAC2
vSphere Host
7
8
VXLAN 5001
IP1 IP2
Controller
VNI
VM
IP
VM
MAC
VTEP
5001 IP2 MAC2 10.20.10.11
10.20.10.11 10.20.10.12
9
VNI
V
M
IP
VM
MAC
VTEP
5001 IP1 MAC1 10.20.10.10
5001 IP2 MAC2 10.20.10.11
PayloadL2
DA: MAC2
SA: MAC1
L2 IP UDP VXLAN PayloadL2
DA: 10.20.10.11
SA: 10.20.10.10
5001
10
DA: MAC1
SA: MAC2
VNI
VM
IP
VM
MAC
VTEP
5001 IP1 MAC1 10.20.10.10
5001 IP2 MAC2 10.20.10.11

24. 24
Troubleshooting Demo
NSX Manager – Visibility and Monitoring tools
Host level troubleshooting
 Monitor different tables
 Packet capture

25. 25
Dynamic Routing Details

26. 26
Routing Enhancements
NSX Edge Pair
Active-Standby
172.16.10.0/24
172.16.20.10
192.168.100.3
172.16.10.10
192.168.10.1
192.168.10.2
External Network
VM VM VM VM
172.16.20.0/24 172.16.30.0/24
172.16.10.11 172.16.30.10
 OSPF
 eBGP/iBGP
 IS-IS
 Route Re-distribution/Filtering
 Graceful Restart
 HA
Dynamic Routing
Logical Distributed
Router

27. 27
Routing Demo

28. 28
Network Virtualization - Operations
Highlights
• Per VM flow visibility
• Traffic Analysis – Packet
Capture
• Transport health
• Inventory & Fault Management
Aggregate Operational Views
• Statistics collections
• Alarms & Health Monitoring
• Network Performance & Resource
Utilization
• Manage & Monitor through infrastructure
management tool such as vCenter
Operations Manager

29. 29
Operations Demo
 vCenter Operations
 Statistics
 Health Monitoring

30. 30
Key Takeaways
 No Multicast required in the Physical Network for VXLAN
 VMware NSX provides various tools to help you troubleshoot your
network and network services
 Integration with next generation operations/monitoring tools for
analysis and alerting of NSX constructs.

31. 31
Q&A
http://blogs.vmware.com/vsphere/networking
@VMWNetworking
@Sachin_t

32. 32
Other VMware Activities Related to This Session
 HOL:
HOL-SDC-1303
VMware NSX Network Virtualization Platform
 Group Discussions:
NET1003-GD
VMware Network Services with Arun Goel

33. THANK YOU

35. Troubleshooting VXLAN and Network Services in a
Virtualized Environment
Vyenkatesh Deshpande, VMware
Sachin Thakkar, VMware
NET5654
#NET5654