"Amazon Virtual Private Cloud (Amazon VPC) has many obvious benefits. For example, you can use Amazon VPC to define a virtual network in your own logically isolated area within the AWS cloud, and launch your EC2 instances into a VPC. But how can you take advantage of the EC2-VPC platform if your services and infrastructure are already deployed in the EC2-Classic platform? In this deep-dive session, learn how to safely and reliably migrate from EC2-Classic to EC2-VPC with zero downtime. We show you how Twilio approached the problem of a VPC migration, or what we internally called the “Moving Datacenters Project.” We discuss the technologies and tools (both internal and external) we used to complete the migration, the infrastructure we built along the way, and the lessons we learned.  Session sponsored by Twilio."

1. © 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
@Sumbry
Director of Cloud Services
Twilio.com
October, 2015
NET409
Movin' On Up to the VPC
How Twilio Migrated its Infrastructure from
EC2-Classic to EC2-VPC

2. Purpose of this talk
- Learn about Twilio
- Review legacy infrastructure
- Why EC2-VPC?
- How we built the Twilio Cloud
- How we migrated
- Internal tools developed
- Lessons learned

3. What Is a Twilio?
- A global communications company
- A real-time communications API
- Used by over 500,000 developers
- Requires low-latency resilient infrastructure
- Has lots of infrastructure on EC2-Classic

4. Who are Twilio customers?

5. Legacy Twilio

6. What did Twilio look like yesterday?
- Twilio has used AWS since 2008
- Three products
- All infrastructure located in us-east-1
- Hundreds of instances
- 10/8 shared private network
- Non-consecutive EIPs

7. Before global

8. What is going global?
- Launched outside US
- Global provisioning
- Route traffic between regions
- Low-latency communications
- Global service discovery

9. The network after global

10. Problems with going global
- Overlapping 10/8 networks
- Proxies not ideal, point-to-point
- Routing around failovers
- Need low latency connectivity

11. Why EC2-VPC?

12. What is EC2-VPC?
EC2-VPC is the next major revision of the EC2 platform:
- Software Defined Network
- Elastic Network Interfaces
- HVM and SR-IOV

13. What is a software defined network?
- Define your own network
- VPC and subnet routing tables
- Network Access Control Lists
- Provision networks like virtual machines
- Protects data-in-transit

14. What are elastic network interfaces?
- Public and Private EIPs
- Multiple Private EIPs per interface
- Multiple ENIs per instance
- Security groups follow an ENI
- ENI has a MAC address

15. What are HVM instances?
- Hardware Virtualized Machine instances
- PCI Express speeds to network adapter
- Low-latency access to network adapter
- Up to 10 GB network speeds

16. Why move to EC2-VPC?
- SDN solves overlapping 10/8 networks
- Route tables eliminates proxies
- Routing around failovers is an API call
- HVM solves low latency connectivity problem

17. The Twilio Cloud

18. What is the Twilio Cloud?
- Iteration 2.0 of our infrastructure
- Addresses many EC2-Classic limitations
- Connectivity between data centers
- Automatic failover and redundancy
- Provider agnostic

19. What does the Twilio Cloud look like?

20. What about routing?

21. We built it, did they come?
We solved all previous issues but no one used it:
- Twilio Cloud was isolated from EC2-Classic
- Existing services had no migration path

22. Data center migration

23. Why is a migration like moving data centers?
- Separate infrastructure from EC2-Classic
- Need to migrate all your compute
- Zero downtime

24. The networks

25. What problems do we need to solve?
- Move an instance from Classic to VPC
- Network connectivity
- Instance discoverability
- No service interruptions

26. Classic deploy

27. VPC deploy

28. Kill Classic

29. Steps to migrate a service

30. Wait - you just invented a bunch of stuff …
- Bridge EC2-Classic and VPC?
- Global Service Discovery?
- Multiple Service Deployments?
- WTF!

31. Migration tools

32. What are the tools for migrating to EC2-VPC?
We modified existing internal tools:
- IP Tunnel Manager / ClassicLink
- Global Service Discovery
- HAProxy Distributed Load-Balancing
- Config-Renderer

33. What is IP Tunnel Manager ClassicLink?
ClassicLink allows you to link
your EC2-Classic instance to
a VPC in your same account
in the same region.
It provides network
connectivity between EC2-
Classic and EC2-VPC
instances.

34. What is Global Service Discovery?
GSD stores IP addresses for any service in the cluster and
serves them on-demand.

35. What is distributed load balancing?
Every instance in the cluster runs its own instance of
HAProxy. It load balances requests to any downstream
services.

36. What is Config-Renderer?
Config-Renderer renders configuration files filled with data
from Global Service Discovery, like HAProxy Configs!

37. What about deploying services?
Our internal
provisioning tool
called BoxConfig lets
us deploy services
with the click of a
button.

38. How does it all work?

39. Unix philosophy
We use lots of small tools and combine them:
- Twilio Cloud to route
- ClassicLink to bridge
- HAProxy for distributed load-balancing
- Global Service Discovery for IP info
- Config-Renderer to write HAProxy configs
- BoxConfig to deploy

40. In conclusion

41. Where are you today?
- The Twilio Cloud is live today
- Routes traffic through nine virtual data centers
- Over 100 IPSEC Mesh links
- Automatic region failover thanks to EIGRP
- 35% of Twilio infrastructure is in EC2-VPC
- We can complete the migration in 2015

42. What are some lessons learned?
- Properly subnet your VPC. You have one shot.
- No need to do a giant migration all at once.
- Tools need to work both ways in case you screw up.
- Less complexity always wins.

43. Thank you!

44. Remember to complete
your evaluations!

45. Related Sessions