1. Verizon‘s Requirements for IP/MPLS-Based
Carrier Ethernet Networks
Andrew G. Malis & Drew Rexrode
Verizon Communications
andrew.g.malis@verizon.com
charles.a.rexrode@verizon.com
Future-Net 2010
2. Introduction
• Public Ethernet services are exploding in popularity
• External Ethernet interface to the customer does not necessarily
mean ―Ethernet inside‖
– The Internet Engineering Task Force (IETF) has standardized
mechanisms for providing point-to-point and multipoint public Ethernet
services over IP/MPLS-based infrastructures
• This talk discusses Verizon‘s requirements for such solutions,
including functionality, conformance to Metro Ethernet forum (MEF)
service definitions, reliability, scalability, QoS, performance
monitoring, OAM, testing, and certification
2
3. Why Are Ethernet Services
Popular?
• Ubiquity and low cost of Ethernet interfaces in customer equipment,
universal experience with Ethernet in LANs, and perceived simplicity
• Successful marketing of the ―Ethernet‖ brand by vendors, IEEE,
MEF, and others
– Little resemblance with original DIX Ethernet specifications, from
physical layer on up (e.g., today‘s Ethernet is mostly point-to-point or
ring-based rather than CSMA-CD at the physical layer)
– Most everything has changed except for the basic frame format – and
jumbograms (large frames up to 9K bytes) change even that
• Plenty of competition and favorable pricing by service providers
3
4. MEF Carrier Ethernet Service
Definitions
Port-Based VLAN-Based
Connectivity Model
(All to One Bundling) (EVC identified by VLAN ID)
E-Line Ethernet Private Line Ethernet Virtual Private Line
(point-to-point EVC) (EPL) (EVPL)
E-LAN Ethernet Private LAN Ethernet Virtual Private LAN
(multipoint-to-multipoint EVC) (EP-LAN) (EVP-LAN)
E-Tree Ethernet Private Tree Ethernet Virtual Private Tree
(rooted multipoint EVC) (EP-Tree) (EVP-Tree)
• Three service types based on the three Ethernet Virtual Connection (EVC) types
• Two ―UNI Types‖ determine whether services are ‗private‘ or ‗virtual‘
– Port-based (All to One Bundling) single EVC (transparency, but uses an entire port per service)
– VLAN-based ‗N‘ EVCs per UNI (not as transparent, but multiple services per port)
• Services are defined by combination of connectivity model and ‗UNI Type‘
• Also Ethernet-based access services to Layer 3 VPNs or dedicated Internet access
4
5. ―Enterprise-Class‖ Ethernet
Limitations
• ―Enterprise-class‖ Ethernet switching has shortcomings as a basic
for public Ethernet services
– Few features for high availability in protocols or equipment
– Scaling limits on MAC addresses, VLAN IDs, and spanning tree
topology limit the size of native Ethernet networks
– Spanning tree routing may take seconds to (occasionally) minutes to re-
converge
• Early Ethernet providers found that enterprise-class Ethernet cannot
naively be deployed for reliable carrier services
5
6. Evolving and Scaling Ethernet
Services
• A typical ―early‖ public Ethernet service provider probably uses
Ethernet switches and Q-in-Q for customer separation
• Typical end user services are
– Ethernet Private LAN (EP-LAN)
– Ethernet Virtual Private LAN (EVP-LAN)
– Ethernet Private Line (EPL)
– Ethernet Virtual Private Line (EVPL)
– Each of these services requires the use of a provider VLAN tag
• As the service becomes successful, the provider will encounter the
usual Ethernet scaling limitations
– MAC address scaling
– VLAN tag scaling (4K customer limit)
– Switching capacity limits
6
7. Typical ―Early‖ Ethernet Service
Network
• Characterized by organic
PE growth driven by customer
PE
location
PE
GigE/LAG
PE • All switches are ―edge
switches‖
PE PE • May be some number of
PE
redundant links
PE
PE
PE
• 802.3ad Link Aggregation may
also be used for resiliency or
PE
PE
PE for additional BW between
PE switches
PE
PE • Flat network with spanning
PE
tree routing
– Network diameter is limited,
often to metro scope
PE – Provider Edge Switch
7
8. Emergence of ―Carrier Ethernet‖
• Limitations in enterprise-class Ethernet have led to the development
of ―Carrier Ethernet‖
• Meant to address unique requirements for carrier Ethernet services,
including Verizon‘s services
– Scaling to support a large number of customers
– Scaling to support large numbers of switches and customer interfaces
– Support both point-to-point (E-Line) and multipoint (E-LAN and E-Tree)
services
– Support for both port-based and VLAN-based services
– Support for QoS other than best-effort to support QoS-based SLAs
– Sub-second outage restoration and routing convergence to support
availability SLAs
– Policing and shaping to support sub-rate services (e.g., 200 Mbps
service on a physical GigE interface)
8
9. IETF Ethernet Services Support
• Point-to-point pseudowires (PWs) to carry layer two frames,
including Ethernet, over IP/MPLS networks
• Extends the MPLS LDP protocol to signal pseudowire establishment
• IETF extended PWs to a multipoint Ethernet service, VPLS (Virtual
Private LAN Service)
• PWs and VPLS extremely popular, implemented by most every
router vendor and in wide use by service providers world-wide
• Verizon uses both point-to-point PWs and VPLS to provide customer
Ethernet services
9
10. IP/MPLS Forum Ethernet Services
Support
• Extended IETF-defined PWs to support non-similar endpoint
interworking
– Supports point-to-point Ethernet-to-Frame Relay, Ethernet-to-ATM, and
ATM-to-Frame Relay interworking over MPLS PWs
– Very useful for multiservice convergence, and to support customers with
a variety of access methods
– Can support applications such as hub location with GigE access, and
low-speed Frame Relay spokes
– Supports interworking of IP packets via ARP Mediation, and bridged
services by interworking native Ethernet with Ethernet frames
encapsulated by FR or ATM
– Can also support VPLS endpoints with FR or ATM-attached customer
equipment
10
11. H-VPLS vs. VPLS+PBB
• VPLS and H-VPLS as originally defined by the IETF cannot meet
Carrier Ethernet service scaling requirements:
– 10s to 100s of thousands of EVCs
– Number of E-LAN bridging instances per edge switch/LER
– Up to millions of customer MAC addresses
• For these reasons, the IETF is now defining the combination of
VPLS in the core with Provider Backbone Bridges (PBB, 802.1ah) at
the edge
11
12. Scalable Network Architecture –
PBB + MPLS
PB PB
N-PE
BEB PBEB BEB
PB N-PE BCB BEB PB
BEB BCB N-PE
PBEB
PBEB
P P
BEB
BEB
/PE
PB
PB PB
• Metro Network Dedicated to Ethernet • Less touch points for cross-metro • MPLS core leveraged across
Service services multiple services (e.g., Ethernet, L3
VPNs)
• Investment Protection • PBB (B-VID) VPLS instance (reduce
PW Meshiness) • Scalable and mature control plane
• Hierarchy with PBB
• Broadcast containment per service • Leverage control plane to ease
• Administrative Traffic Eng. across core (via MMRP/BGP-AD) administration (BGP-Auto Discovery,
TE)
• Operations skill set / OSS Leverage • PBB MAC hiding
12
13. PBB-VPLS— MAC Scaling and
Customer-Addressing Awareness
PE-rs PE-rs
MTU-s MTU-s
MPLS
MPLS
MTU-s
H-VPLS PBB-VPLS MTU-s
No. of MAC addresses/node No. of MAC addresses/node
100,000s
100,000s
1000s Customer MACs
1000s
Backbone MACs
MTU-s PE-rs
0 MTU-s PE-rs
0
• ―Hub‖ PE-rs get visibility of 100,000s of MACs • MAC tables reduced: one B-MAC per
• High customer-addressing awareness node
• No customer-addressing awareness
13
14. PBB-VPLS Benefits — Service/Pseudowire
Scaling and Customer-Service Awareness
B
B
PE-rs PE-rs
B
B
B
B
B B B
MTU-s B
MTU-s MTU-s
H-VPLS
VPLS + PBB
No. of services-PW/node No. of services-PW/node
100,000s 100,000s
Customer services
10,000s 10,000s
Customer PWs
1000s 1000s
Backbone services
0 100s
Backbone PWs
0
MTU-S MTU-S PE-rs PE-rs MTU-S MTU-S PE-rs PE-rs
14
15. OAM Specifications
• The IEEE, ITU-T, and MEF have defined Ethernet OAM (Operations,
Administration and Maintenance) specifications to allow fault
detection and correction. These include:
– Link OAM: IEEE 802.3-2005, Clause 57
• Enables monitoring and troubleshooting of native Ethernet links
– Ethernet Local Management Interface (E-LMI): MEF 16
• Provides EVC status
• Enables automatic configuration of Customer Equipment (CE)
– Connectivity Fault Management (CFM): IEEE 802.1ag
• Enables monitoring and troubleshooting of VLANs within a network
• Supports multiple views (Customer, Service Provider, Operator)
– Service OAM: ITU-T Y.1731
• Extends CFM to include additional FM capabilities
• Performance Monitoring (PM)
15
18. Carrier Ethernet over MPLS
Certification
• ROI • Man Hours
• Time to Market • Resources
18
19. Conclusions
• Verizon‘s Carrier Ethernet services must meet stringent
requirements for:
– Conformance to Metro Ethernet forum (MEF) service definitions
– Scalability to support customer growth
– Reliability, resilience, OAM for troubleshooting and performance
monitoring, to support high service availability
– Standards-based certification
– Pre-deployment and post-deployment testing
19
