Learn how Comcast addressed their data quality and system resiliency issues by using AWS technologies such as Amazon SQS, Amazon SNS, Amazon DynamoDB, Elasticsearch, and Logstash to replace a legacy messaging system. The new messaging system, Publish Subscribe Platform (PSP), interconnects customer management, billing, and provisioning applications. It is becoming the core platform for Comcast's next generation billing transformation program. Comcast shares the best practices, scaling considerations, and migration strategies they employed.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Building Modern Platforms:
A Practical Way to Migrate Legacy
Systems to Amazon SQS
A P I 3 1 9 - R
Jasbir Rajpal
Senior Manager
Comcast/System
Engineering
James (Jim) Doyle
Principal Software
Architect
Comcast/System
Engineering
Rajeev Chakrabarti
Principal Enterprise
Architect, AWS

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why do you need messaging?
“Loosely coupled systems”
The more loosely they are coupled,
the bigger they will scale,
the more fault tolerant they will be,
the fewer dependencies they will have,
the faster you will innovate.

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
When should I use messaging?
Separate parts of an application
Web tier instances create work; worker instances
complete it
Scale and manage tiers separately
Perform tasks asynchronously
Long-running tasks (such as transcoding,
transactions)
Don’t need to wait for a response (for example, JS
web apps)
Independent and fault tolerant

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
When should I use messaging?
Connect multiple components
Send individual messages or fan out to many recipients
Provide instant or delayed notification
Batch and burst processing
Be resilient to spikes in traffic
Perform work only as fast as necessary to lower costs
Don’t lose data

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS messaging: No infrastructure to manage
Focus on business logic, not infrastructure
Just write your code; Amazon handles:
Capacity
Scaling
Deployment
Fault tolerance
Monitoring
Logging
Security

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Messaging options from AWS
Amazon Simple Queue Service
(Amazon SQS)
Durable and scalable
Comprehensive security
Query-in-place
Amazon Simple Notification
Service (Amazon SNS)
Performance at scale
Fully managed
Enterprise-ready
Amazon MQ
Industry standard APIs
Low-latency
Full featured
For born-in-the-cloud apps
Simple APIs
Unlimited throughput
Pay per request
For application migration
API-compatible
Feature-rich
Low latency
Pay per hour and per GB

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jasbir Rajpal
Senior Manager
Comcast/System
Engineering
James (Jim) Doyle
Principal Software
Architect
Comcast/System
Engineering

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Problem
• What is the biggest problem in IT organizations?
• What hurts the most?
• What do you hate?

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our problem(s)
• Poor resiliency
• Flaky software
• Flaky underlying hardware
• Flaky network
• Flaky inter data center network connectivity
• Data quality issues introduced
• Disconnecting services
• Even senior executives
• Spent too much time in that non-business value add

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Consideration/evaluation of technologies
• What goes into selecting a technology?

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Consideration/evaluation of technologies
• Resilience (availability, response time)
• Hardware
• Software
• Network connectivity
• Faster time to market
• Build development environment
• SDLC
• Production-ready environment

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Fixing is an evolutionary process
• With regard to transaction processing
• Amazon Simple Queue Service (Amazon SQS) FIFO queues solve a difficult
problem in a compelling way
• Exactly-once message delivery across data centers
• Publish-subscribe, order-respecting messaging
• Legacy, DC bound, transactional applications became
active-active and highly-available
• Wrappers for various levels of app sophistication
• Proprietary
• SOAP
• REST

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Unordered PSP Architecture
UPSP
Users Elastic Load
Balancer
Data Center 1
Data Center 2
Data Center 3
Publication Service
queue
encrypted
data
Data Center 1
Data Center 2
Data Center 3
Pull
Subsription Servicetopic
Users
Elastic load
balancer
queue
queuetopic
topic
Pull
Decrypt data
Virginia
Comcast
Comcast
Publication Service
Publication Service
Internet
Internet
Internet
encrypted
data
encrypted
data
Subsription Service
Subsription Service
Decrypt data
Decrypt data
Internet
Internet
Internet
EAAS
Elastic Load
Balancer
Data Center 1
Data Center 2
Data Center 3
Publication Service
queueencrypted
data
Data Center 1
Data Center 2
Data Center 3
Push
topic
Elast
balan
queue
queuetopic
topic
Push
DynamoDb
table
item
Lambda
function
Lambda
function
Lambda
function
Virginia
Internet
Internet
Internet
Publication Service
Publication Service
encrypted
data
encrypted
data
Comcast
Comcast
AWS Direct Connect
Push
Load
Balancer
Load
Balancer

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ordered PSP Architecture
OPSP
Users
Elastic
Load
Balancer
Data Center 1
Data Center 2
Data Center 3
Publication Service
FIFO queueencrypted
data
Data Center 1
Data Center 2
Data Center 3
Subsription Service
Users
Elastic load
balancer
FIFO queue
FIFO queue
DynamoDb
table
item
Decrypt data
Virginia
Comcast Comcast
Publication Service
Publication Service Internet
Internet
Internet
encrypted
data
encrypted
data
Subsription Service
Subsription Service
Decrypt data
Decrypt data
Internet
Internet
Internet
Lambda
function
Fetch SQS Names
Load
Balancer
Load
Balancer
DynamoDB
Table

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
EaaS Architecture
EAAS
Users Elastic Load
Balancer
Data Center 1
Data Center 2
Data Center 3
Publication Service
queueencrypted
data
Data Center 1
Data Center 2
Data Center 3
Push
topic
Users
Elastic load
balancer
queue
queuetopic
topic
Push
DynamoDb
table
item
Lambda
function
Lambda
function
Lambda
function
Virginia
Internet
Internet
Internet
Publication Service
Publication Service
encrypted
data
encrypted
data
Comcast
Comcast
AWS Direct Connect
Push
Load
Balancer
Load
Balancer
DynamoDB
Table

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Challenges of migration
• What are they?

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration
• Minimum functionality production-ready software
• Incremental software build
• Management patterns
• Dedicated team for migration
• Incentives tied to migration
• Celebrations
• Senior executive visibility

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Migration continued
• Software patterns
• Keep API same
• Redirect publish first to new system, but build path to old system
• Strangulate old system with time
• Data quality patterns
• LWE/API
• Refresh
• Self healing

20. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jasbir Rajpal
Jim Doyle
Rajeev Chakrabarti

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What it is
Simple, flexible,
fully managed
message queuing
service for reliably
and continuously
exchanging any
volume of messages
from anywhere
Use case
Build decoupled,
highly scalable
microservices,
distributed systems,
and serverless
applications in the
cloud
Cool capabilities
Nearly infinite
scalability and
ability to increase
message
throughput without
pre-provisioning
capacity
Amazon Simple Queue Service (Amazon SQS)

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SQS capabilities
• Messages are encrypted at rest (AWS Key Management Service (AWS KMS))
and in flight (HTTPS/TLS)
• Integrated with other AWS services (such as AWS Lambda) as a destination
• Instant message delivery with long polling
• Message payloads up to 256 KB (2 GB using Amazon Simple Storage Service
(Amazon S3))
• Simplified troubleshooting with a Dead Letter Queue
• Easy monitoring with Amazon CloudWatch, with alarming
• Message batching delivers even higher throughput and reduced cost
• Support standard and ordered queues

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What it is
Simple, flexible,
fully managed
publish/subscribe
messaging and
mobile push
notification service
for high throughput,
highly reliable
message delivery
Use case
Push messages to a
variety of endpoints
and clients in
distributed systems,
microservices, and
serverless
applications and
enable event-driven
architecture
Cool capabilities
Highly reliable
delivery of any
volume of messages
to any number of
recipients across
multiple protocols
Amazon Simple Notification Service (Amazon SNS)

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon SNS capabilities
• Flexible message delivery across multiple transports
• Simplify development with filtered message delivery
• Support for PrivateLink to use Amazon SNS inside customer VPC
• Messages are encrypted in flight with HTTPS/TLS
• Deep integration with over 30 AWS services (source and destination)
• Retry policy ensures message delivery (customizable for HTTP)
• Easy monitoring with CloudWatch, with alarming
• Failure notifications

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
What it is
Managed message
broker service for
Apache ActiveMQ
that makes it easy to
set up and operate
message brokers in
the cloud and
enable hybrid
architecture
Use case
Migrate to a
managed message
broker to automate
software
administration and
maintenance,
without having to
rewrite existing
applications
Cool capabilities
Compatible with
industry-standard
APIs and protocols,
including JMS, NMS,
AMQP, STOMP,
MQTT, and
WebSocket
Amazon MQ

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon MQ capabilities
• Fully managed implementation of Apache ActiveMQ
• Highly durable (multi-AZ store) and highly available (active and standby)
• Messages encrypted at rest (service managed) and in flight (TLS)
• Message delivery across multiple protocols and transports (JMS, AMQP,
MQTT, NMS, STOMP, WebSockets)
• Supports customer VPC endpoints and public endpoints
• Easy monitoring with CloudWatch
• Unlimited message size and retention
• Rich features (queues and topics, transient/persistent, distribute
transactions, filtering, scheduling, virtual destinations, request/reply)

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Ordered fan-out publishing
• Amazon SNS not available for FIFO
• Lambda, Amazon DynamoDB, FIFO deduplication to the rescue!
• Lambda
• Publish interface
• Uses topic to look up subscription queues
• Fans out one thread per target queue
• Fail at first sign of trouble
• DynamoDB
• Topic name to subscription (queue URL) mapping
• FIFO deduplication
• Solves the lambda early bail problem