This document discusses evolving VPC designs from a single VPC to multiple interconnected VPCs. It begins with a basic single VPC design and evolves it to incorporate multiple subnets, NAT gateways, VPC endpoints and peering. The document explores use cases for separating resources into multiple VPCs and presents a hub-and-spoke design using VPC peering to interconnect VPCs and provide shared services while maintaining isolation and control.

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Robert Alexander, AWS Principal Solutions Architect
October 2015
From One to Many
Evolving VPC Design
ARC 403

2. Disclaimer:
Do Try This at Home!

3. Assuming you’ve heard of…
Route Table
Elastic
Network
Interface
Amazon VPC
Internet
Gateway
Customer
Gateway Virtual
Private
Gateway
VPN
Connection
VPC subnet
Network ACL
Security group
Enhanced
Networking
VPC
Peering
AWS Direct
Connect

4. From one…
Subnet
Availability Zone A
Subnet
Availability Zone B
VPC

5. us-west-2
VPC
us-east-1
sa-east-1
ap-southeast-2
eu-central-1
VPCVPC
VPC
VPC VPC
eu-west-1
ap-southeast-1
VPC
… to many
VPC
ap-northeast-1
VPC

6. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
VPC

7. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
ELB
Web
Back end
VPC CIDR 10.1.0.0/16
ELB
Web
Back end
.1
VPC
.1
.1 .1
.1 .1

8. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
ELB
Web
Back end
ELB
Web
Back end
AWS region
Internet
Public Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 IGW
VPC

9. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
ELB
Web
Back end
ELB
Web
Back end
AWS region
Internet
And what if instances
in a private subnet
need to reach outside
the VPC?
They have no route to
the IGW and no public
IP address.
VPC

10. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
ELB
Web
Back end
ELB
Web
Back end
AWS region
Internet
Why go outside?
VPC
• AWS API endpoints
• Regional services
• Third-party services

11. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
ELB
Web
Back end
ELB
Web
Back end
AWS region
Internet
Deploy an instance
providing:
N etwork
A ddress
T ranslation
NAT
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 NAT instance
VPC

12. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
ELB
Web
Back end
ELB
Web
Back end
AWS region
Internet
NAT
VPC
Deploy an instance
providing:
N etwork
A ddress
T ranslation
Private Route Table
Destination Target
10.1.0.0/16 local
0.0.0.0/0 NAT instance
Private Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0/0 Black Hole

13. Scalable and Available NAT

14. To NAT, or not to NAT…
• Leave NAT for less bandwidth-critical connectivity
• Don’t bottleneck high-bandwidth-out workloads
• Run high-bandwidth components from public subnets
• Goal is full-instance bandwidth out of VPC

15. Evolving design requirements
• Public subnets for high-bandwidth public talkers
• Private subnets with access to public AWS network
• Highly available NAT
• One AWS account
• One VPC
• One region

16. HA NAT Built with:
Amazon EC2 Auto Recovery
Amazon EC2 Auto Reboot
The “Whack-a-Mole” NAT

17. EC2 status checks

18. EC2 status checks
StatusCheckFailed_System
StatusCheckFailed_Instance
CloudWatch
per-instance metrics:

19. Amazon CloudWatch alarm actions
Instance
status check fails?
REBOOT
System
status check fails?
RECOVER
Instance ID
Instance metadata
Private IP addresses
Elastic IP addresses
EBS volume attachments
Instance retains:

20. A few things to remember…
• Recover action only applies to system status checks
• Limited to C3, C4, M3, R3, and T2 instance types
• Cannot use local instance store
• Cannot be dedicated instances
• Use EC2ActionsAccess AWS Identity and
Access Management (IAM) role
Amazon EC2 Auto Recovery

21. Amazon EC2
Auto Recovery
Set your failed check threshold
Choose 1-minute period
and statistic minimum
Choose recover action
Metric = StatusCheckFailed_System
CloudWatch
Console

22. Amazon EC2
Auto Reboot
Choose reboot action
Metric = StatusCheckFailed_Instance
CloudWatch
Console

23. Availability Zone A
Private subnet
Public subnet
Private subnet
Availability Zone B
Private subnet
Public subnet
Private subnet
Web
Back end
Web
Back end
AWS region
Internet
NAT
VPC
NAT
Average tested recovery time:
~ 1 to 4 minutes
Could be shorter or longer
depending on nature of failure
HA NAT
with
EC2 Auto Recovery
+ Auto Reboot

24. Pick a NAT, any NAT
Amazon Linux NAT Amazon Machine Image (AMI)

25. Availability Zone A
Private subnet
Public subnet
Private subnet
AWS region
Internet
NAT
VPC
Availability Zone B
Private subnet
Private subnet
NAT
Private subnet
Private subnet
NAT
Private subnet
Public subnet
Private subnet
NAT
Private subnet
Private subnet
NAT
Private subnet
Private subnet
NAT
Scaling NAT

26. AWS
region
Considering multiple VPCs
Public-facing
web apps
Internal
company
apps
What’s next?
VPN
connection
VPC VPC VPC
Customer
network

27. One VPC, Two VPC

28. Common customer use cases
Application isolation
Scope of audit containment
Risk-level separation
Separate production from nonproduction
Multi-tenant isolation
Business unit alignment

29. Considerations for one or many VPCs
Know your inter-VPC traffic
Separate AWS accounts
IAM/resource permissions and controls
Know your VPC limits:
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html

30. AWS region
Internal application to VPC
Public-facing
web app
Internal
company
app
VPN
connection
VPCVPC
Customer
network

31. Availability Zone A
Private subnet Private subnet
AWS region
Virtual
Private
Gateway
VPN
connection
Intranet
app
Intranet
app
Availability Zone B
Internal customers
Private Route Table
Destination Target
10.1.0.0/16 Local
Corp CIDR VGW
VPC
Internal application to VPC
Customer
network

32. But apps want to leverage…
Amazon S3
…as a primary data store

33. This Is the End(point)

34. Evolving design requirements
• VPN connectivity to private-only VPC
• No egress in the VPC to public networks
• Private IP access to Amazon S3
• Content-specific access controls
• One AWS account
• One VPC
• One region

35. Availability Zone A
Private subnet Private subnet
AWS
region
Virtual
Private
Gateway
VPN
connection
Intranet
app
Intranet
app
Availability Zone B
You really don’t want to do this:
Amazon
S3
Internet
Customer
border router
Customer VPN
Internet
VPC
Customer
network

36. Availability Zone A
Private subnet Private subnet
AWS
region
Virtual
Private
Gateway
Intranet
app
Intranet
app
Availability Zone B
So do this instead:
Amazon
S3
VPC
VPN
connection
VPC Endpoints
• No IGW
• No NAT
• No public IPs
• Free
• Robust access
control
Customer
network

37. “Currently, we support endpoints for connections with
Amazon S3 within the same region only. We'll add support for
other AWS services later.”
From the Amazon VPC User Guide:
VPC endpoints
$ aws ec2 describe-vpc-endpoint-services
SERVICENAMES com.amazonaws.us-west-2.s3

38. Creating S3 VPC endpoint
aws ec2 create-vpc-endpoint
--vpc-id vpc-40f18d25
--service-name com.amazonaws.us-west-2.s3
--route-table-ids rtb-2ae6a24f rtb-61c78704
Private subnet
VPC
Route Table
Destination Target
10.1.0.0/16 Local
Corp CIDR VGW
Prefix List for S3 us-west-2 VPCE

39. Creating S3 VPC endpoint
aws ec2 create-vpc-endpoint
--vpc-id vpc-40f18d25
--service-name com.amazonaws.us-west-2.s3
--route-table-ids rtb-2ae6a24f rtb-61c78704
Public subnet
VPC
Route Table
Destination Target
10.1.0.0/16 Local
0.0.0.0 IGW
Prefix List for S3 us-west-2 VPCE

40. Prefix lists
aws ec2 describe-prefix-lists
PREFIXLISTS pl-68a54001 com.amazonaws.us-west-2.s3
CIDRS 54.231.160.0/19
• Logical route destination target
• Dynamically translates to service IPs
• S3 IP ranges change over time
• S3 prefix lists abstract change

41. Prefix lists
… and use them in security groups!

42. Private subnet
Controlling VPC access to Amazon S3
IAM policy on VPCE:
VPC
{
"Statement": [
{
"Sid": "vpce-restrict-to-backup-bucket",
"Principal": "*",
"Action": [
"s3:GetObject",
"s3:PutObject”
],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::backups-reinvent2015",
"arn:aws:s3:::backups-reinvent2015/*"]
}
]
}
Backups bucket?

43. Private subnet
Controlling VPC access to Amazon S3
S3 bucket policy:
VPC
From
vpce-bc42a4e5?
{
"Statement": [
{
"Sid": "bucket-restrict-to-specific-vpce",
"Principal": "*",
"Action": "s3:*",
"Effect": "Deny",
"Resource": ["arn:aws:s3:::backups-reinvent2015",
"arn:aws:s3:::backups-reinvent2015/*"],
"Condition": {
"StringNotEquals": {
"aws:sourceVpce": "vpce-bc42a4e5”
}
}
}
]
}

44. Controlling VPC access to Amazon S3
Recap on security layers:
1. Route table association
2. VPCE policy
3. Bucket policy
4. Security groups with prefix list
Private subnet
VPC
1.
2.
3.
4.

45. Private subnet Private subnet
AWS
region
Intranet
apps
Compliance
app
Endpoints in action
VPC
Compliance Backups
VPCE1 VPCE2
Private subnet
Intranet
apps

46. Private subnet Private subnet
AWS
region
Intranet
apps
Compliance
app
Endpoints in action
VPC
Compliance Backups
VPCE1 VPCE2
Private subnet
Intranet
apps
Private subnet Private subnet
Private subnet

47. What about Amazon Linux?
Compliance VPCE policy modified:
{
"Statement": [
{
"Sid": "vpce-restrict-to-backup-bucket-and-alinux",
"Principal": "*",
"Action": [
"s3:GetObject",
"s3:PutObject”
],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::compliance-reinvent2015",
"arn:aws:s3:::compliance-reinvent2015/*",
"arn:aws:s3:::repo.us-west-2.amazonaws.com",
"arn:aws:s3:::repo.us-west-2.amazonaws.com/*",
"arn:aws:s3:::packages.us-west-2.amazonaws.com",
"arn:aws:s3:::packages.us-west-2.amazonaws.com/*"]
}
]
}
VPCE1
Compliance

48. What about Amazon Linux?
Backup VPCE policy modified:
{
"Statement": [
{
"Sid": "vpce-restrict-to-backup-bucket-and-alinux",
"Principal": "*",
"Action": [
"s3:GetObject",
"s3:PutObject”
],
"Effect": "Allow",
"Resource": ["arn:aws:s3:::backups-reinvent2015",
"arn:aws:s3:::backups-reinvent2015/*",
"arn:aws:s3:::repo.us-west-2.amazonaws.com",
"arn:aws:s3:::repo.us-west-2.amazonaws.com/*",
"arn:aws:s3:::packages.us-west-2.amazonaws.com",
"arn:aws:s3:::packages.us-west-2.amazonaws.com/*"]
}
]
}
VPCE2
Backups

49. Private subnet Private subnet
AWS
region
Intranet
apps
Compliance
app
Access to Amazon Linux repositories
VPC
Compliance Backups
VPCE1 VPCE2
Private subnet
Intranet
apps
Private subnet Private subnet
Private subnet
Repo Packages

50. A few things to remember…
• Endpoint and bucket must be in same region
• Amazon DNS enabled on VPC
• Source IPs to S3 will be private
• Don’t forget about S3 dependent services
VPC endpoints for Amazon S3

51. AWS region
Public-facing
web apps
Internal-
only
apps
What’s next?
VPN
connection
VPC VPC VPC
Customer
network
Customer Gateway
(CGW)

52. VPC Mass Transit

53. AWS
region
VPC
VPC
VPC
VPC VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPCVPC
Customer
network
Public
apps
Internal
apps

54. AWS region
VPC
HA VPN Pair
Availability Zone A
HA VPN
To
VPC
iBGP
eBGP
Customer CIDRs or Default Route
eBGP
AWS ASN 7224
Re-advertise VPC CIDR via IGP
VGW
VPC CIDR
Customer ASN (Public or Private)
CGW1 CGW2
VPN1
Tun1
VPN1
Tun2
Availability Zone A
VPN2
Tun1
VPN2
Tun2
Reuse your CGW Public IP
to connect to more VPCs
Customer
network

55. AWS
region
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
• DNS
• Directory
• Logging
• Monitoring
• SecurityShared services
Customer
network

56. Evolving design requirements
• Centralize network connectivity to and from cloud
• Centralize management, security, and common services
• Account owners in control of own VPC resources
• Many AWS accounts
• Many VPCs
• One region

57. AWS
region
VPC
VPC
VPC
VPC
VPC
VPC
• DNS
• Directory
• Logging
• Monitoring
• SecurityShared services
Hub and
Spoke
with
Peering
VPC
Shared
services
VPC
VPC
Customer
network
Spoke VPC
Spoke VPC
Spoke VPCSpoke VPC
Spoke VPC
Spoke VPC
VPC

58. Customer
network
AWS region
VPC
Hub VPC
Private subnet
VPC
Spoke VPC
Public subnet
10.2.0.0/1610.1.0.0/16
Private subnet
Private Route Table
Destination Target
10.1.0.0/16 Local
10.2.0.0/16 PCX-1
Private Route Table
Destination Target
10.2.0.0/16 Local
10.1.11.0/24 PCX-1
VPC peering
Shared services
10.2.22.0/24
10.1.11.0/24

59. AWS region
VPC
Hub VPC
Private subnet
VPC
Spoke VPC
Public subnet
10.2.0.0/1610.1.0.0/16
Private subnet
Private Route Table
Destination Target
10.1.0.0/16 Local
10.2.0.0/16 PCX-1
Private Route Table
Destination Target
10.2.0.0/16 Local
10.1.11.0/24 PCX-1
172.16.0.0/16 PCX-1
Edge-to-edge routing
Shared services
10.2.22.0/24
10.1.11.0/24
172.16.0.0/16
Customer
network

60. AWS region
VPC
Hub VPC
Private subnet
VPC
Spoke VPC
Proxy
subnets
10.2.0.0/1610.1.0.0/16
Private Route Table
Destination Target
10.2.0.0/16 Local
10.1.0.0/16 PCX-1
Edge-to-edge via proxy
PCX-1 10.2.22.0/24
Internal
ELB
Proxy
fleet
Internet
Public
services
S3
VPC
Customer
network
Proxy Route Table
Destination Target
10.1.0.0/16 local
10.2.0.0/16 PCX-1
172.16.0.0/16 VGW
Proxy Route Table
Destination Target
10.1.0.0/16 Local
10.2.0.0/16 PCX-1
172.16.0.0/16 VGW
0.0.0.0/0 IGW
S3 Prefix List VPCE

61. Customer
network
Availability Zone A
Private subnet
Public subnet
Private subnet
Elastic
Load
Balancer
Shared
services
AWS region
Internet
VPC
Auto Scaling
proxy
fleet
Public
servicesS3
PCX-1
Availability Zone B
Private subnet
Public subnet
Private subnet
Elastic
Load
Balancer
Shared
services
Auto Scaling
proxy
fleet
Spoke VPC
VPC
Private subnet
Proxy in practice
Hub VPC

62. Availability Zone A
Private subnet
Public subnet
Private subnet
Elastic
Load
Balancer
Shared
services
AWS region
Internet
VPC
Auto Scaling
proxy
Fleet
Public
servicesS3
PCX-1
Availability Zone B
Private subnet
Public subnet
Private subnet
Elastic
Load
Balancer
Bastion
host
Auto Scaling
proxy
fleet
Spoke VPC
VPC
Private subnet
Proxy in practice
Hub VPC
Customer
network

63. A few things to remember…
• Use IAM to restrict spoke AWS accounts
• Create a NetOps IAM role in all accounts
• Enable AWS CloudTrail and AWS Config for all accounts
• Integrate CloudTrail with CloudWatch Logs and create alarms:
Shared Services Hub and Spoke
https://aws.amazon.com/blogs/aws/
cloudtrail-integration-with-cloudwatch-now-available-in-four-more-regions/

64. AWS region
VPC
VPC
VPC
VPC
VPC
• DNS
• Directory
• Logging
• Monitoring
• Security
VPC
VPC
VPC
VPC
VPC VPC
VPC
VPC
VPC
VPC
Shared services
Customer
network
Dev hub
Prod hub
Data
services
hub

65. Go with the Flow

66. Evolving design requirements
• Audit VPC network security configuration
• Analyze network usage
• Automated responses to network security alarms
• Many AWS accounts
• Many VPCs
• Many regions

67. VPC Flow Logs
• Agentless
• Enable per ENI, per subnet, or per VPC
• Logged to AWS CloudWatch Logs
• Create CloudWatch metrics from log data
• Alarm on those metrics
AWS
account
Source IP
Destination IP
Source port
Destination port
Interface Protocol Packets
Bytes Start/end time
Accept
or reject

68. VPC Flow Logs: Automation
Amazon
SNS
CloudWatch
Logs
Private subnet
Compliance
app
AWS
Lambda
If SSH REJECT > 10,
then…
Elastic
Network
Interface
Metric filter
Filter on all
SSH REJECTFlow Log group
CloudWatch
alarm
Source IP

69. VPC Flow Logs

70. VPC Flow Logs

71. https://aws.amazon.com/blogs/aws/new-amazon-elasticsearch-service/
VPC Flow Logs
• Amazon
Elasticsearch
Service (ES)
• Amazon
CloudWatch
Logs
subscriptions
• Kibana

72. AWS region
VPC
VPC
VPC
VPC
VPC
• DNS
• Directory
• Logging
• Monitoring
• Security
VPC
VPC
VPC
VPC
VPC VPC
VPC
VPC
VPC
VPC
Shared services
Customer
network
Dev hub
Prod hub
Data
services
hubs

73. VPC
VPCVPC
Bringing It All Back Home
VPC
VPC
VPC

74. Evolving design requirements
• Many Gbps network connectivity to AWS
• Cost-effective
• Predictable latency
• Leverage existing corporate network
• Many AWS accounts
• Many VPCs
• Many regions

75. Customer
network
AWS Direct Connect
location
AWS Direct Connect Private Virtual
Interface (PVI) connects to VGW on
VPC
• 1 PVI per VPC
• 1 eBGP peer per VPC
• 1 802.1Q VLAN Tags per VPC
Private fiber connection
One or multiple
50 – 500 Mbps,
1 Gbps or 10 Gbps pipes
Simplify with AWS Direct Connect
AWS region
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC
VPC VPC
VPC
VPC
VPC
VPC
Dev hub
Prod hub
Data
services
hub

76. 0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
10 20 30 40 50 60 70 80 90 100
$PERMONTH
TB PER MONTH
AWS Direct Connect vs EC2 Data Out Cost
EC2 Data Out 2 x 1 Gbps DX 2 x 10 Gbps DX
* Calculated from us-west-2 and does not include telco cost to reach a DX location if required

77. AWS Direct Connect (DX) in the United States
SuperNAP
Equinix SE
Coresite LA
N. Virginia
N. California
Oregon
Coresite NY
Equinix DC
Equinix
SV

78. AWS Direct Connect (DX) in Europe and Asia Pacific
Telecity
Eircom Interxion
Sydney
Frankfurt
Ireland
Tokyo
Singapore
Equinix OS
Beijing
Equinix TY
Equinix
FR
Equinix SY
Global Switch
Equinix SG
CIDS
Sinnet

79. Bring it
Headquarters
Branch
Branch
DX Location
Provider Edge (PE)Customer Edge (CE)
eBGP
Provider
MPLS
Network
PECE
PE
CE
AWS region
MPLS / IPVPN
PE DX
eBGP
CE PE

80. Bring It
Headquarters
Branch
Branch
DX Location
L2
Provider
VPLS
Network
PECE
PE
CE
AWS region
VPLS
PE DX
L2
CE PE
eBGP

81. Prod hub
Private Virtual Interface 1
VLAN Tag 101
BGP ASN 7224
BGP Announce 10.1.0.0/16
Interface IP 169.254.251.5/30 10.1.0.0/16
VGW 1
Multiple VPCs over AWS Direct Connect
Customer
Switch + Router
Customer Interface 0/1.101
VLAN Tag 101
BGP ASN 65001
BGP Announce Customer Internal
Interface IP 169.254.251.6/30
VLAN 101
VLAN 102
VLAN 103
Data hub
10.2.0.0/16
VGW 2
Dev hub
10.3.0.0/16
VGW 3
Private Virtual Interface 2
VLAN Tag 102
BGP ASN 7224
BGP Announce 10.2.0.0/16
Interface IP 169.254.251.9/30
Customer Interface 0/1.102
VLAN Tag 102
BGP ASN 65002
BGP Announce Customer Internal
Interface IP 169.254.251.10/30
Customer Interface 0/1.103
VLAN Tag 103
BGP ASN 65003
BGP Announce Customer Internal
Interface IP 169.254.251.14/30
Private Virtual Interface 3
VLAN Tag 103
BGP ASN 7224
BGP Announce 10.3.0.0/16
Interface IP 169.254.251.13/30
Route Table
Destination Target
10.1.0.0/16 PVI 1
10.2.0.0/16 PVI 2
10.3.0.0/16 PVI 3
VPC
Customer
network
VPC
VPC

82. Prod hub
Public Virtual Interface 1
VLAN Tag 501
BGP ASN 7224
BGP Announce AWS Regional
Public CIDRs
Interface IP Public /30 Provided
10.1.0.0/16
VGW 1
Public AWS + VPCs over AWS Direct Connect
Customer
Switch + Router
Customer Interface 0/1.501
VLAN Tag 501
BGP ASN 65501 (or Public)
BGP Announce Customer Public
Interface IP Public /30 Provided
VLAN 101
VLAN 102
VLAN 103
VLAN 501
Data hub
10.2.0.0/16
VGW 2
Dev hub
10.3.0.0/16
VGW 3
Public US AWS
regions
Route Table
Destination Target
10.1.0.0/16 PVI 1
10.2.0.0/16 PVI 2
10.3.0.0/16 PVI 3
Public AWS PVI 5
NAT + Security layer
Customer
network
VPC
VPC
VPC

83. AWS Direct Connect in the United States
Equinix SV
us-west-1
us-west-2
us-east-1
AWS Private Network Disaster recovery
VPN to VGW

84. A few things to remember…
AWS Direct Connect
• Be selective in your public network announcements
• Remember prefix lists
• Authoritative AWS public IP list available:
https://ip-ranges.amazonaws.com/ip-ranges.json
• For notification of IP changes, subscribe to SNS topic:
arn:aws:sns:us-east-1:806199016981:AmazonIpSpaceChanged

85. Headquarters
Branch
Branch
Seattle DX Location
eBGP
Provider
MPLS
Network
PECE
PE
CE
AWS
Oregon
region
Multi-region DX
PE DX
eBGP
CE PE
London DX Location
AWS
Ireland
region
PE DX
eBGP
Going global
AS 7224
AS 7224
100 BGP Route Max
100 BGP Route Max

86. BGP AS override
router bgp <asn>
address-family ipv4 vrf <vrf-id>
neighbor <AWS DX eBGP Peer IP> as-override
neighbor <AWS DX eBGP Peer IP> as-override
PE DX
set protocols bgp group <group-name> neighbor <AWS DX eBGP Peer IP> peer-as 7224
set protocols bgp group <group-name> neighbor <AWS DX eBGP Peer IP> as-override
Cisco IOS:
Junos OS:

87. Provider
MPLS
Network
VPCVPC
VPC
VPC
EU-West-1 region
London DX
US-West-2 region
Seattle DX
AP-Northeast-1
region
Tokyo DX
VPC
VPC
VPC
VPC
VPC
VPC
VPC
Branch
HQ
Branch Branch

88. Evolving design requirements
• Cross-region network between all VPCs
• Scalable, full-mesh IPsec network
• Minimal operational overhead
• Leverage AWS network
• Many AWS accounts
• Many VPCs
• Many regions

89. The Monster Mesh

90. Dynamic Multipoint VPN
DMVPN:
Built with Cisco Cloud Services Router (CSR) 1000V
• Available on the AWS Marketplace
• A virtualized ASR with full IOS-XE software stack
• BYOL or Pay-as-you-Go license models

91. Dynamic Multipoint VPN
Proven, scalable VPN design framework
Key components:
Next Hop Resolution Protocol (NHRP - RFC2332)
Multipoint GRE (mGRE)
IPsec

92. us-west-2
VPC
NHRP hub
10.1.0.0/16
us-east-1
VPC
10.2.0.0/16
Spoke 1
eu-west-1VPC
10.3.0.0/16
Spoke 2
ap-northeast-1VPC
10.5.0.0/16
Spoke 4
eu-central-1
VPC
10.4.0.0/16
Spoke 3
Global AWS
network
DMVPN
Dynamic
Multipoint
Virtual
Private
Network:
Phase 3
DMVPN
network
10.100.0.0/24
NHRP
request

93. us-west-2
VPC
NHRP Hub 1
10.1.0.0/16
us-east-1
VPC
10.2.0.0/16
Spoke 1
eu-west-1VPC
10.3.0.0/16
Spoke 2
ap-northeast-1VPC
10.5.0.0/16
Spoke 4
VPC
10.10.0.0/16
NHRP Hub 2
DMVPN
Dual hub
Single subnet
10.100.0.1
10.100.0.2
10.100.0.3
10.100.0.410.100.0.5
Global AWS
network
DMVPN
network
10.100.0.0/24
VPC
10.4.0.0/16
Spoke 3
10.100.0.6
eu-central-1

94. DMVPN hub configuration
interface Tunnel0
bandwidth 1000000
ip address 192.168.0.1 255.255.255.0
no ip redirects
ip mtu 1400
no ip split-horizon eigrp 192
ip nhrp map multicast dynamic
ip nhrp network-id 1
ip nhrp redirect
ip tcp adjust-mss 1360
tunnel source GigabitEthernet1
tunnel mode gre multipoint mGRE

95. DMVPN spoke configuration
interface Tunnel0
bandwidth 1000000
ip address 192.168.0.2 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp map 192.168.0.1 52.24.102.22
ip nhrp map multicast 52.24.102.22
ip nhrp map 192.168.0.5 52.64.165.176
ip nhrp map multicast 52.64.165.176
ip nhrp network-id 1
ip nhrp nhs 192.168.0.1
ip nhrp nhs 192.168.0.5
ip nhrp shortcut
ip nhrp redirect
ip tcp adjust-mss 1360
tunnel source GigabitEthernet1
tunnel mode gre multipoint
Hub 1
Hub 2

96. us-west-2
VPC
us-east-1
Regional HQ
Remote
workforce
eu-central-1
VPCVPC
eu-west-1
Branches
VPC
From one to many
ap-northeast-1
VPC
Global HQ
Regional HQ

97. And now for something…
completely different…

98. • Many AWS Accounts
• Many VPCs
• Many regions
• Public subnets for high-bandwidth public talkers
• Private subnets with access to public AWS Network
• Highly available NAT
• VPN connectivity to private-only VPC
• No egress in the VPC to public networks
• Private IP access to Amazon S3
• Content-specific access controls
• Centralize network connectivity to and from cloud
• Centralize management, security, and common services
Evolving design requirements

99. Remember to complete
your evaluations!

100. Related Sessions
ARC402 – Double Redundancy with AWS Direct Connect
NET403 – Another Day, Another Billion Packets
NET404 – Making Every Packet Count
NET406 – Deep Dive: AWS Direct Connect and VPNs
NET308 – Consolidating DNS Data in the Cloud with
Amazon Route 53

101. Thank you!