Contenu connexe Similaire à BBC Built Massive Media Pipeline Using Microservices and AWS Messaging Similaire à BBC Built Massive Media Pipeline Using Microservices and AWS Messaging (20) Plus de Amazon Web Services Plus de Amazon Web Services (20) BBC Built Massive Media Pipeline Using Microservices and AWS Messaging1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
How the BBC Built a Massive Media
Pipeline Using Microservices
S t e p h e n G o d w i n , L e a d A r c h i t e c t , B r i t i s h B r o a d c a s t i n g C o r p o r a t i o n ( B B C )
R o b e r t C h o w , P r i n c i p a l P r o d u c t M a n a g e r , A W S ( S Q S a n d S N S )
A R C 3 3 0
N o v e m b e r 2 8 , 2 0 1 7
2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Where we’re going
Messaging overview
BBC iPlayer
• Background
• Challenges
• Transformation to microservices
• The role of messaging
• Results
• Pitfalls and best practices
Wrap-up
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
System building blocks for modern
applications
DatabaseCompute Storage Messaging
4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Simple Queue Service (SQS)
• Asynchronous
• Each message processed by
one consumer
Simple Notification Service
(SNS)
• Pub/sub
• Broadcast or filtered
• High fan-out of messages
Amazon SQS & SNS: Decouple applications
5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Stephen Godwin
• Lead technical architect at the BBC
• Designed the systems that provides audio and video
to BBC iPlayer and iPlayer Radio
• Led the migration of the systems that power iPlayer
to a cloud-based microservice architecture
• Worked for nearly 10 years as one of the developers
of IBM WebSphere@SteveGodwin
6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
03
PERCENT
OF THE UK
POPULATION
WATCHED BY
11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
01 THOUSAND HOURS
OF MEDIA PUBLISHED
EVERY WEEK
12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A NEW PROGRAM
IS AVAILABLE
ON iPLAYER 51
EVERY
MINUTES
13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Increasing
demand and
instability
14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A perfect pattern for an elastic
architecture
Fri Oct 18 Sun Oct 20 Tue Oct 22 Thu Oct 24
transcode_requests
15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Broadcast
video feed x
24
Video
chunker
Time
addressable
media store
FFmpeg
Pre-delivered
content
Media
distribution
service
PaaS
Transcode
service
Playout
event feed
Live Ingest
Logic
We are
here
Inside iPlayer: Video factoryInside iPlayer: Live Ingest Logic
16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Playout
event feed
Rights
checker
Inside the Live Ingest Logic service
Live Ingest Logic
Live Ingest
Logic
Broadcast
event notifier
Interactive
TV services
Other
TV services
17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Broadcast
event notifier
Playout
event feed
Rights
checker
Live Ingest Logic
Using messaging to wire the system
Time
addressable
media store
Media
distribution
service
Pre-delivered
content
Broadcast
video feed x
24
Video
chunker
FFmpeg PaaS
Transcode
service
18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Message-oriented microservices pattern
Service
SQS
In Q
SQS
Out Q
19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Message-oriented microservices pattern
EC2 instance
Service
SQS
In Q
SQS
Out Q
20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Scaling to handle peaks
Service
Service
Service
SQS
In Q
Auto Scaling group
21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service
Resiliency and durability
Service
Service
Auto Scaling group
SQS
In Q
SQS
Out Q
22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Service
Resiliency and durability
Service
Service
Auto Scaling group
SQS
In Q
SQS
Out Q
23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Error handling
EC2 instance
Service
SQS
In Q
SQS
Out Q
SQS
dead letter Q
24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Playout
event feed
Broadcast
video feed x
24
Video
chunker
FFmpeg
Pre-delivered
content
PaaS
Transcode
service
Live Ingest
Logic
Media
distribution
service
Time
addressable
media store
We are
here
Inside iPlayer: Time addressable media store
25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Event-driven trace and analysis
Request
received
Found all
chunks
Created
new file
File
complete
Time
addressable
media store
26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Inside iPlayer: Media distribution service
Playout
event feed
Broadcast
video feed x
24
Video
chunker
Time
addressable
media store
FFmpeg
Pre-delivered
content
PaaS
Transcode
service
Live Ingest
Logic
We are all the
way over here
Media
distribution
service
27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Refreshing caches for data consistency
Synchronous call
Asynchronous call
Availability
database
Media distribution
service
Audience
Read Update
28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Where we ended up
Happier audience
Faster time to deployment
Easy extensibility and reuse
Massive scalability
Improved resilience
© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Designing for idempotency
Elastic scaling is great—linear scaling is better
Reliable pipelines of microservices
Think about how you will debug your system
30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
aws.amazon.com/sqs/getting-started
aws.amazon.com/sns/getting-started
L E A R N M O R E
32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Thank you!