This document discusses how carrier-grade Ethernet can ensure reliable communications for utility networks transitioning to support smart grid applications. It covers Ethernet mechanisms that provide carrier-grade performance such as quality of service, resiliency, monitoring and timing synchronization. Choosing between IP, MPLS and Ethernet options is discussed. The document also addresses network security considerations and introduces RAD's carrier-grade Ethernet product portfolio for power utilities.
AWS Community Day CPH - Three problems of Terraform
Carrier ethernet-for-power-utilities-presentation
1. Carrier-Grade Ethernet
for Power Utilities
Ensuring Reliable Communications in
the Smart Grid
Carrier-Grade Ethernet for Utilities Slide 1
2. Agenda
• Towards a Smarter Grid: Utility Networks in Transition
• Carrier-Grade Ethernet Mechanisms
• Timing over Packet Synchronization
• Choosing the Right Packet Network
• Security
• RAD’s Carrier-Grade Ethernet Solutions for Power Utilities
Carrier-Grade Ethernet for Utilities Slide 2
4. Utility Networks in Transition
• Ethernet, IP, MPLS replace SDH/SONET; higher throughput and lower
OpEx for:
Upgrades to Smart Grid
New applications: Substation automation, IEC 61850 IEDs, WASA
synchrophasors, IP video surveillance
• $3.2 billion spent on utilities telecom equipment in the US – transport
networks second largest category (UTC, 2011)
• RAD survey: 43% of utilities begin migration with backbone comm
network; 28% with SCADA system
Carrier-Grade Ethernet for Utilities Slide 4
5. Migration Performance Requirements
• Need service assurance for mission critical apps in a
PSN environment:
Low end-to-end delay
High availability (99.99% - 99.999% )
SDH/SONET-level resiliency
• Requirements vary depending on application:
Teleprotection: < 10ms symmetrical delay, minimal
“jitter”
SCADA: May tolerate 1s latency
• Cyber security:
More interconnected devices = more entry points to
protect
Need scalable protection tools
Carrier-Grade Ethernet for Utilities Slide 5
6. Traffic Types and Transmission over PSN
• Ethernet and IP data and signals from SA IEDs → sent natively over
packet
• TDM traffic, analog voice, serial SCADA, Teleprotection → require
“packetization” or pseudowire emulation (PWE):
Synchronous bit stream is segmented; headers are added to form packet
Packets are forwarded to destination over the PSN network
At destination, headers are removed, original bit stream is reconstructed,
timing regenerated
Common pseudowire emulation standards: CESoPSN, SAToP, TDMoIP
Carrier-Grade Ethernet for Utilities Slide 6
8. Ethernet is Now Carrier-Grade
• Ethernet is no longer a “Best Effort” technology
Engineered by industry to provide SLA-based
performance guarantees, reliability schemes, service
management tools
• Full set of standards developed by the IEEE, ITU-T,
MEF
• Metro Ethernet Forum = a consortium of carriers,
service providers, equipment vendors promoting
Carrier Ethernet adoption
• Extensive deployments of premium Ethernet
services with double digit growth worldwide
Can now provide deterministic quality of service, predictable latency and jitter
performance for mission-critical substation applications: SCADA, IEC 61850
GOOSE, etc
Carrier-Grade Ethernet for Utilities Slide 8
9. Extensive Traffic Management and
Quality of Service Toolset
• Classification of incoming traffic into flows, e.g., IEC 60870-5-
104, DNP3 per DSCP, 61850 GOOSE traffic per PCP, VLAN-ID
• Metering and policing to regulate traffic with different bandwidth
profiles
• Advanced scheduling and queue management to ensure minimal
latency and jitter
• Shaping to smooth out bursts and avoid buffer overruns in
subsequent network elements
• Packet editing and marking to signal proper handling instructions for
subsequent network elements
Carrier-Grade Ethernet for Utilities Slide 9
10. Performance Monitoring and Testing
• A wealth of carrier-grade Ethernet tools to remotely test, monitor
and troubleshoot the operation of communications links
• Utility network operators anticipate service degradation ahead of
time, as well as cut down truck-rolls and on-site technician calls
Fault
Connectivity Fault Performance
Stress Testing Propagation
Verification Detection Monitoring
& Isolation
Carrier-Grade Ethernet for Utilities Slide 10
11. Communications Channel Resiliency
• Hardware redundancy:
No single point of failure (NSPF) design with redundant, hot-swappable
power supplies
Redundant control plane and switch fabric cards
• Link redundancy:
1+1 protection topology with automatic switchover between links
Link aggregation group (LAG) per IEEE 802.3-2005 LACP (link
aggregation control protocol) for Ethernet-based services
• Path protection:
Ethernet Linear protection Switching (G.8031) , AKA “EVC (Ethernet
Virtual Connection) protection”
Ethernet Ring Protection Switching (G.8032 ERPS) to provide Five Nines
(99.999%) availability
Carrier-Grade Ethernet for Utilities Slide 11
12. Timing over Packet (ToP)
Synchronization
• PSNs require complementary clock transfer solutions with a high
level of precision
• ~1ms highest accuracy level required today for utility
applications
Upcoming Smart Grid apps will require 1μs and better
Time of Day (ToD) required for synchrophasors and to avoid
cascading blackouts
• GPS is accurate but costly, susceptible to jamming
• ITU-T Synchronous Ethernet (Sync-E)
Uses the Ethernet physical layer to accurately distribute frequency
• IETF NTP and IEEE 1588-2008 Precision Time Protocol (PTP)
Exchange timestamp information to deliver frequency and ToD
• Many network operators use Sync-E for frequency distribution
and PTP for time synchronization
Carrier-Grade Ethernet for Utilities Slide 12
13. Choosing the Right Packet Network
Carrier-Grade Ethernet for Utilities Slide 13
14. Comparing Technology Options
Utility network operators need to choose which technology to employ:
• IP
• MPLS (Multi-Protocol Label Switching)
• Carrier-grade Ethernet
Protocol OAM/APS Security
IP No standard end-to-end Strong (IPsec)
mechanisms
MPLS Recently developed for No built-in security
MPLS-TP
Ethernet Carrier-grade Several security
mechanisms defined
(802.1X, MACsec)
Carrier-Grade Ethernet for Utilities Slide 14
15. Option 1: Combining Ethernet Access
with MPLS Core
Ethernet GOOSE delivered directly = No
Lower cost per port
tunneling/conversion over MPLS core
Richer OAM, PM tools
Advanced protection
mechanisms
Maintain existing
access media installed
base
Carrier-Grade Ethernet for Utilities Slide 15
16. Option 2: Replace SDH/SONET with
Ethernet Operational Network
Scalable IP/MPLS for IT/enterprise network
VoIP, IP video, internet connectivity, billing
IT/Enterprise Substation
Center IT/Enterprise Multiservice RTU
IP/MPLS Network Access RS-232
Node
Teleprotection
VoIP Operational X.21
Ethernet Network PBX
E1
Ethernet
Service
Aggregation
Platform
1 GbE / 10 GbE
Ring
Ethernet Service
Aggregation
Platform
Simplified architecture Operational
Substation
and management Center Multiservice
RS-232
RTU
Access
Increased security NMS Node
Teleprotection
Lower latency Replace SDH/SONET X.21
SCADA
Assured QoS PBX
operational network with E1
Ongoing performance
monitoring
Carrier-Grade L2 Ethernet
SCADA, Teleprotection,
Automation
Carrier-Grade Ethernet for Utilities Slide 16
17. Security
Carrier-Grade Ethernet for Utilities Slide 17
18. Layered Security Approach
• Utility networks require “Defense in Depth” implementing multiple
levels of protection:
• Should be deployed in the network without adding too many
dedicated security appliances on top of the communications
infrastructure
Carrier-Grade Ethernet for Utilities Slide 18
20. Full Portfolio for Power Utility Networks
IPmux
Megaplex ETX-A
TDM Pseudowire
Multiservice Access Carrier Ethernet
Gateway
Platforms over Fiber
ETX-5300A RICi-GE RADiFlow
Multiservice Ethernet over NG-SDH Service-Aware
Aggregation Secure Ethernet
Platform Switches
Carrier-Grade Ethernet for Utilities Slide 20
21. Ensuring a Smooth Migration
• Hybrid SDH/SONET and PSN access solutions
Allow utilities to choose the migration path
that best suits their needs
Deterministic QoS for NGN services and
advanced grid applications over TDM or packet
transport
Service continuity for legacy equipment
• Multi-standard ToP synchronization, including 1588
Grandmaster functionality in the same communications
device
• Ethernet switches with cyber security and authentication
protocols
Carrier-Grade Ethernet for Utilities Slide 21
22. Conclusion
• Ethernet has been engineered and standardized and now includes
critical “must haves” for NG utility networks:
Robust clock accuracy
Traffic management and hard/hierarchical QoS
Resiliency
On-going performance monitoring
Support for legacy services and traffic
Security
• A combination of carrier-grade Ethernet in the access/aggregation
with an MPLS core can address the needs of the Smart Grid
• RAD offers a wide selection of utility-grade solutions to ensure high
performance, cost-effective and easy migration
Download comprehensive Carrier-Grade Ethernet for Power Utilities
Solution Paper
Carrier-Grade Ethernet for Utilities Slide 22