The document discusses the requirements and experiences of carrier grade NAT (CGN) technology, including what CGN is, how it relates to IETF standards, managing subscriber sessions, flow analysis of subscriber behavior, logging optimization techniques like port block allocation and deterministic NAT, and the evolution of CGN including port forwarding and the new Port Control Protocol (PCP).

1. CARRIER GRADE NAT
REQUIREMENTS AND EXPERIENCES
Sławek Janukowicz
Emil Gągała
PLNOG, Kraków, 27.09.2011

2. AGENDA
Introducing CGN
What is CGN
CGN and IETF
CGN Requirements
EIM, APP, EIF
Managing Subscriber Sessions
Flow Analysis
2
Flow Analysis
Applications and Subscriber’s behavior
Logging optimization
Port Block Allocation
Deterministic NAT
Evolution of CGN
Port Forwarding and PCP

3. NAT44
rfc1918
private address ISP shared address
CPE EDGE router
v4 v4
NAPT
3
Src 192.168.100.62 port 36980
Dst 217.8.168.244 port 53
Src 66.129.224.36 port 45852
Dst 217.8.168.244 port 53

4. WORLD TODAY (YESTERDAY?)
Leased Line
NAT44
V4 Only
IPv4 Business
Customer
4
V4 Only
V4 Only
Subscriber mgmt
function
NAT44

5. CGN
Leased Line
V4 Only
IPv4 Business
Customer
5
V4 Only
V4 Only
Subscriber mgmt
function
NAT functions
are performed at
the ISP edge or
core

6. CGN AND IPV4 DEPLETION
As of Feb 1st, 2011 IANA has exhausted free IPv4 pools. RIRs
will follow some time after.
Several techniques are available to deal with the problem. A
topic of another presentation
CGN is a cornerstone of at least two of them (NAT444 and DS-
Lite).
6
CGN might end up being used in 6rd and NAT44
with networks and internet moving to IPv6, some services (e.g.
Web, email) will still require native IPv4 reachability for quite
some time. Therefore a NAT64 (another type of NAT) is needed.
In conclusion, CGN becomes a common building block for
different technologies aimed to easier the migration to a IPv6
internet.

7. CGN AND IETF
CGN is the result of several IETF draft and RFCs developed by
the BEHAVE WG. A starting point to understand which are the
relevant drafts and RFCs could be draft-ietf-behave-lsn-
requirements
CGN’s building blocks are in RFC4787 (UDP), RFC5382 (TCP)
and RFC5508 (ICMP)
7
Some deployment scenarios of CGN are covered in draft-
shirasaki-nat444, draft-ietf-v6ops-incremental-cgn
The specific case of NAT64 is addressed by another set of
drafts: draft-ietf-behave-v6v4-framework, draft-ietf-behave-
address-format, draft-ietf-behave-v6v4-xlate, draft-ietf-behave-
v6v4-xlate-stateful, draft-ietf-behave-dns64

8. 8
CGN REQUIREMENTS

9. CGN CHALLENGES
Example of issues
Incoming port negotiation mechanisms may fail
Applications that establish inbound communications
Applications that carry address and/or port information in their
payload
Applications that do not use any port (TCP/UDP transport)
Applications that assume the uniqueness of source addresses
9
Applications that assume the uniqueness of source addresses
(e.g., IP address as identifier)
Service usage monitoring and abuse logging
Spam blacklisting
Geo-location and Geo-proximity
Traceability
draft-ietf-intarea-shared-addressing-issues-05.txt

10. MANAGING SUBSCRIBER’S SESSIONS
Limiting the maximum number of sessions from same subscriber
This allows to provide fairness of public IP and port availability to different
subscribers
Prevents DDOS attacks: a few subscribers starving all available resources
Port Random-Allocation
For each IP address in a pool the initial allocated port is assigned randomly
and then continuing to allocate ports sequentially from there.
It lowers the risk of inbound attacks when EIF is enabled
10
It lowers the risk of inbound attacks when EIF is enabled
Round Robin address allocation
For every different internal source address, a different NAT address is
allocated in a round robin fashion
Instead of using all available ports for a specific public IP address, move to the
next public IP address whenever there is a new internal host requiring a
connection.
Address Pooling behavior unchanged. I.e. new sessions from the same
internal source address will continue to use the same public IPv4

11. HIGHLIGHTS ON BEHAVE RFC REQUIREMENT
APP, Address Pooling ,“Paired” behavior
a CGN must use the same external IP address mapping for all sessions associated
with the same internal IP address
Remote servers or peers for the same application reject connections if not all
originated by the same IP address. Examples are Instant Messengers
EIM, Endpoint Independent Mapping,
A CGN must assign the same external address and port for all connections
originated from a given internal host if they all use the same internal port
11
originated from a given internal host if they all use the same internal port
Enabling EIM allows to have a stable external P address and Port (for a period of
time) that external hosts can use to connect. Very important for p2p, gaming and the
mobile world
EIF, Endpoint Independent Filtering
EIM alone does not influence the inbound filtering behavior. Actually the default
filtering behavior is Address and Port dependant (APM) which means that only
remote Servers or Peers towards which we opened a connection are allowed to
reach the internal host using a specific IP and Port.

12. SUMMARY
APP solves the outbound stability problem (REQ-2 rfc4787)
EIM solves the outbound stability problem (REQ-1 rfc4787)
EIF solves the inbound access rights problem (REQ-8 rfc4787)
12

13. APPLICATION LEVEL GATEWAYS
ALG is used to enable application that otherwise broken by NAT
ALG plus NAT scales better than Cone NAT
Although SPs prefer to not have to deal with ALGs some
protocols require them to be enabled
ICMP, Traceroute, TFTP, RSH, MS-RPC, PPTP, DC, FTP, H.323,
SQLnet, RTSP, SIP
13
SQLnet, RTSP, SIP
CGN should allow the administrator to selectively enable ALGs
on a per protocol basis as there a number of NAT-friendly apps
that ALGs can interfere
Almost endless client implementations

14. CGN SCALING REQUIREMENTS
Basic NAT Scaling
Throughput pps and bits/s (for different packet sizes)
Session table capacity
Throughput at max sessions
Session ramp up rate
Latency
Logging impact
Advanced scaling
14
Advanced scaling
Max Rules per system
NAT Source Pools
NAT Source Address per Pool with/without Port translation
Max session per IP (pool based source NAT)
Persistent NAT binding (EIM/EIF/hairpinning)
Max ALG sessions
Linear scaling for all parameters when adding new hardware
It is layer 4 and above! Good packet throughput is not enough

15. STATEFUL HA OR NOT
Most of flows are UDP and only 0.1% of TCP flows are
longer than 30s.
Most deployments can live without Stateful HA as long
as there is a hot-standby scheme.
HA should be fine tuned for the long lived sessions
(>60s and >1MB)
15
(>60s and >1MB)
Exceptions can be made for special protocols and
applications.
Massive Scaling and Load balancing is probably more
important in production deployments than any type of
stateful HA.

16. OTHER CGN „NICE TO HAVE” FEATURES
Overflow pool
NAT Grace period
Granularity of application timeout configuration
VRF/Routing Instance Awareness
Extended NAT MIB
16

17. TWO APPROACHES FOR BUILDING CGN
Option 1 – Flow Base Approach
Central
Service
Processor
Child
Service
ProcessorChild
Service
Processor2. Packet
Forward to
central SP
3. Packet Forward
to Least Load SP
& the SP that
processes traffi
from the source ip
previously 4. Packet Sent out
5. Program the
Child
Service
Processor
17
Flow
Table
1. Packet from
new flow
central SP
Line Card
5. Program the
flow on Flow
Table for
subsequent
packets within
the same flow
Processor

18. TWO APPROACHES FOR BUILDING CGN
Option 2 – Route base Approach
Option 3 – Combined approach Service
Processor
3. Packet Sent out2. Spread the
Load to existing
service
Service
Processor
Service
Processor
18
PFE to load
balance
traffic based
on SIP
1. Packet from
new flow
Line Card
service
processors
Processor
No NAT v4 and v6
Traffic pass through the
box without going
through the service
processor

19. 19
FLOW ANALYSIS
APPLICATIONS AND SUBSCRIBER’S
BEHAVIOR

20. PURPOSE OF FLOW ANALYSIS
Network planning: oversubscribed or undersubscribed
Definition of Peak vs. average vs. mean
Protocol trends and usage:
New applications
Protocols
Partnerships
20
Partnerships
Optimizations: local and network wide
Forecast required storage capacity for logging
Cost here is a major concern among SPs

21. TOTAL FLOWS PER CATEGORY
8000000
10000000
12000000
Web
P2P
Gaming
22
0
2000000
4000000
6000000
Gaming
IM
TCP-Other
UDP-Other

22. SUBSCRIBER FLOWS FROM REAL DEPLOYMENTS
24
13% of active broadband
Users have less than 1
flow
50% of users have 10 flows
or less

23. NAT REFERENCE PERFORMANCE PROFILE
Mobile smart phones
6 Million users, 4kb/s, 11 sessions average per user, all sessions
refreshed in 5 min
Aggregate: 24 Gb/s, 66 million sessions, 220k session creations/s
Mobile feature phones
6 Million users, 0.1kb/s, 1 session average per user, all sessions
refreshed in 5 min
25
refreshed in 5 min
Aggregate: 600 Mb/s, 6 million sessions, 20k session creation/s
Fixed network
100k users, 100kb/s, 5 session average, one new session every 10s
Aggregate: 10 Gb/s, 500k sessions, 10k session creations/s
NATs for wireless are gated by number of sessions & session creation rate
NATs for wireline are gated by bandwidth

24. 26
LOGGING OPTIMIZATION
PORT BLOCK ALLOCATION
DETERMINISTIC NAT

25. REDUCING THE LOGGING INFORMATION?
Solution: Allocation of contiguous port per subscribers (Just
need to send a range in the Logs messages).
But is it a good solution for all Service Providers?
Questions that need an answer:
How random port allocation should be (security concern)?
27
How random port allocation should be (security concern)?
Do we need to log the destination address?
What is the typical overloading of subscribers per port?
This solution can only be used if:
Low number of subscribers per Public IP
Security issue is understood
Destination IP does not need to be stored

26. WHAT PIECES OF LOG INFORMATION WOULD NEED
TO BE STORED
For each session in case of dynamic source NAT only we currently generate two logs. Normally SP
would need to store at least the following highlighted information:
Jun 28 15:29:20 cypher (FPC Slot 5, PIC Slot 0) {sset2}[FWNAT]:
ASP_SFW_CREATE_ACCEPT_FLOW: proto 6 (TCP) application: any, ge-1/3/5.0:10.0.0.1:8856
-> 128.0.0.2:80, creating forward or watch flow ; source address and port translate to
129.0.0.1:1028
Jun 28 15:29:23 cypher (FPC Slot 5, PIC Slot 0) {sset2}[FWNAT]: ASP_SFW_DELETE_FLOW:
proto 6 (TCP) application: any, (null)(null)10.0.0.1:8856 -> 128.0.0.2:80, deleting forward or
watch flow ; source address and port translate to 129.0.0.1:1028
Let’s assume the syslog server only keeps the essential information of one start log:
28
Let’s assume the syslog server only keeps the essential information of one start log:
(Size per session) =
Private src address (32 bits) +
Private src port (16bit) +
Global src address (32bit) +
Global src port (16 bit) +
Timestamp
(64bit) = 20Bytes

27. DYNAMIC NAT
Random Allocation Ports for each sessions. This is
the default NAPT behavior.
Evaluation:
Good Ratio Users/Public addresses
One log needed per Sessions (Need an important
High
AmountOfLogging
High
Security
Users/PubicIP
29
One log needed per Sessions (Need an important
Logging infrastructure)
No security issue
Public address – Ports allocation (one user per color)
Null
AmountOfLogging
Low
Security
Low
RatioUsers/Pubi

28. NAT WITH PORT BUCKET ALLOCATION (PBA)
When a session is created, the NAT allocate a contiguous bucket
of Ports per user. The port will then be randomly chosen from
this bucket.
New requests for NAT ports will come from this block. Any non-
active block (without any ports in use) will get freed from the NAT
pool.
Logs are only generated for each block allocation and release.
High
AmountOfLogging
High
Security
RatioUsers/PubliIP
30
Evaluation:
Possible to tune the ratio Logging/Security/Users-per-IP
(see next slide)
Reduce dramatically the logs infrastructure needed.
Null
AmountOfLogging
Low
Security
Low
RatioUsers/
Public address – Ports allocation (one user per color)

29. DETERMINISTIC NAT
Algorithmic Allocation of IP addresses and Port bucket per user.
The Public IPv4 address and Port range for a given end user
are fixed. Once the port range is determined, the allocation of a
given port for a new flow is performed dynamically.
Evaluation:
Users keep the same public address all the time (useful for Port
forwarding but could be a privacy issue)
No log messages needed at all.
High
AmountOfLogging
High
Security
Users/PublicIP
32
No log messages needed at all.
Lower ratio of Users/Public address (Even Inactive user get a
public@)
Can’t assign new block if users run out of ports.
Load balancing more complex
Null
AmountOfLogging
Low
Security
Low
RatioUsers/Publi
Public address – Ports allocation (one user per color)

30. 34
EVOLUTION OF CGN
PORT FORWARDING AND PCP

31. PORT FORWARDING IN CG-NAT SCENARIO
Today, users can freely configure port forwarding by web
configuration, either:
direct connection to the CPE (unmanaged CPE)
access to a central configuration server which then remotely
updates CPE (managed CPE)
When moving to a CG-NAT scenario, the requirement is not to
break this model, ie, users should be able to configure port
35
break this model, ie, users should be able to configure port
forwarding
Using a Central NAT Configuration Server which creates
mapping on both the CPE and the CG-NAT looks like the
preferred options to some SPs

32. PORT FORWARIDNG STATISTICS (BB PROVIDER)
100K RGs randomly sampled
1450 failed collection
4512 RGs had at least one entry, thus PF used only by 4.5%
10,300 total port forwarding entries found
Max was one RG with 46 entries
36
Max was one RG with 46 entries
Most had 6 or less
No knowledge if the entries are actively used or if they are
old/stale entries the user never deleted

33. TOP PORTS FORWARDED
Port #RGs Common use of port
3389 622 RDP
80 562 HTTP
5001 503 Slingbox
6112 368 Blizzard games (WoW, Warcraft III, Starcraft, etc.)
443 288 HTTPS
88 272 Xbox
21 269 FTP
3724 202 Blizzard games (WoW)
37
3724 202 Blizzard games (WoW)
5500 200 VNC
22 195 SSH
6881 169 Blizzard and Bit-torrent
47624 154 Games (Microsoft: Age of Kings/Empires, etc)
25 141 SMTP
53 126 DNS and xbox
110 100 POP3
1723 95 PPTP
8080 85 Usually alternative web server port, some web
cams

34. TECHNOLOGY: PCP (NEW DEVELOPMENT)
PCP: Port Control Protocol
PCP objectives are to enable applications to receive incoming
connections in the presence of an ISP NAT/Firewall.
Instead of ‘working around’ NATs like other NAT traversal
techniques like STUN/TURN/ICE, PCP enables an explicit dialog
between applications and the NAT.
38
between applications and the NAT.
PCP can be seen as a ‘carrier-grade’ evolution of UPnP-IGD and
NAT-PMP.
The work on PCP is done at IETF in a new working group co-
chaired by Alain Durand (Juniper) & Dave Thaler (Microsoft).

35. PCP IN A NUTSHELL
ISP network
Applications negotiate ports with the ISP NAT to establish external presence.
Application asks: “I’d like to get port 5000 for 48 hours”, NAT PCP server responds:
“I give you port 6003 for 12 hours”.
No more keep-alive!
Better radio efficiency
Better battery life
39
IPv4
NAT
ISP network

36. 40
SUMMARY

37. SUMMARY
As a matter of fact SPs will need to share limited public IPv4
addresses for many years. CGN serves as a key building block
in any transition strategy to IPv6.
CGN must ensure application level transparency
Storage of Logs could significantly impact on costs. Hence
strategies to reduce the amount of logs should be considered:
PBA, Deterministic NAT
41
PBA, Deterministic NAT
Subscribers who are used to set Port Forwarding on their RG
will ask for the same functionality when NAT is either moved into
the SP network, i.e. DS-Lite, or added in the SP network, i.e.
NAT444 or 6rd+NAT444.
Port Control Protocol as the alternative to Port Forwarding going
forward