Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Serverless 

Media Workflow
Speaker
• Serverless Lover 

• Software Engineer @ Vingle

• Freelancer Software Engineer
@ Artistscard

• Software Engine...
A journey to

Video Service 🎥
Video is becoming 

the most important content

🔥🔥🔥
Let’s add video feature
on Vingle! 😎
…but how?
Common VOD
workflow
Common Video workflow
Common Video workflow
Common Video workflow
Common Video workflow
Building Ingest block
Ingest Step Requirements
• Support File-based workflow only for now

• Simple upload workflow for clients

• Fast upload spe...
Ingest Step Requirements
• Support File-based workflow only for now

• Simple upload workflow for clients

• Fast upload spe...
S3 Upload using pre-signed
URL
• Allows client can upload
directly to S3 using pre-
signed URL

• AWS-SDK supports pre-
si...
S3 Upload using pre-signed
URL
…but we are using 

us-east-1 region
some regional ISPs have
bandwidth issue on
international network transfer 💩
Slow Upload on some ISPs
Upload speed is so
slow!🐢🐢🐢
Submarine cables 

in South Korea
Submarine cables 

in South Korea
Two options to
accelerate S3 transfer 🚀
Two options to accelerate S3 transfer
• Proxy file transfer traffic through CloudFront

• Cloudfront can accelerate dynamic c...
S3 Transfer Acceleration is faster than Cloudfront as
Proxy
S3 Transfer Acceleration makes throughput
faster
Ingest block architecture
Problem solved! ✅
Building Processing
block
Processing Requirements
• use fully-managed service if possible

• scalable, reliable

• adaptive bitrate streaming

• Imp...
Wait! 

What is Adaptive Bitrate
Streaming?
Adaptive Bitrate Streaming
• From Wikipedia:

• Adaptive bitrate streaming is a technique used in
streaming multimedia ove...
Adaptive Bitrate Streaming
Choosing 

Streaming Protocol
Choosing Streaming
Protocol
• Non-HTTP based protocols like RTMP are not in our interest

• Looked for HTTP-based protocol...
Choosing Streaming
Protocol
• Progressive Download Single File via HTTP

• Apple HLS

• MPEG-DASH
Choosing Streaming
Protocol
• Progressive Download Single File via HTTP

• Apple HLS - Winner!
• MPEG-DASH
Why we choose HLS
Progressive Download

Single File via HTTP
• Pros

• Easy to implement

• Cons

• Implementing Adaptive Bitrate Streaming ...
MPEG-DASH
• Pros

• New standard

• Based on Fragmented MPEG4 (similar to MPEG2 Transport Stream on
HLS)

• Broader codec ...
First Video
Specification
First Spec - Protocol
• HTTP Live Stream (HLS)

• Apple invented standard, natively supported on macOS/iOS/tvOS

• Nativel...
First Spec - Input
• No codec limitation of input source



… but expects most input source are H.264/AVC
+ AAC based

• M...
First Spec - Output
• Common

• H.264/AVC1 + AAC LC

• Many graphic chipset supports hardware accelerated decoding

• Most...
First Spec - Output
• 4 Outputs

• HLS HQ

• HLS MQ

• HLS LQ

• HTTP Progressive MP4 MQ
Check requirements
Requirements
• use fully-managed service if possible

• scalable, reliable

• adaptive bitrate streaming

• Important: com...
Requirements
• use fully-managed service if possible

• scalable, reliable

• adaptive bitrate streaming

• Important: com...
Requirements
• use fully-managed service if possible

• scalable, reliable

• Important: Delivery video to users ASAP
AWS provides bunch
of media products 🎉
Video Encoding Products
• AWS Elastic Transcoder

• Fully-managed service, pay only for what you use (per minute of output...
…or build your own
video encoding service
Most time-consuming
task is video encoding
Tested two products
Test conditions
• Input

• f(x) 4 Walls M/V (3m 33s)

• H.264/AVC, 1080p, 23.97fps, 8Mbps, High/4.1

• AAC LC, 192k, 48KHz...
Elastic Transcoder
Elastic Transcoder
Elemental MediaConvert
Elemental MediaConvert
Results
• Overall workflow was stable

• Quality

• The output quality of both was similar

• Fixed GOP => Consistant segme...
Problems
• Pricing is quite expensive

• Needed more faster encoding speed without losing quality

• Missing CRF (Constant...
Make our own
processing workflow 💥
Three ways to run job
• Batch - EC2

• Fargate - ECS

• Step Functions + Lambda
Three ways to run job
• Batch - EC2

• Fargate - ECS

• Step Functions + Lambda - WINNER!
Three ways to run job
• Batch - EC2

• Requires additional instance provisioning time (extremely slow)

• Fargate - ECS

•...
Video Processing
Workflow
Video Processing Workflow
Single Video Processing
Function
Issue #1
How to get encoding
software running on AWS
Lambda?
Lambda-runtime compatible
build
• Linux program

• Lambda execution runtime is based on Amazon Linux AMI (amzn-ami-hvm-201...
Lambda-runtime compatible
build
• Node.js Native Addons

• lambci provides docker images to rebuild
native addons

• docke...
Issue #2
Timeout issues

Hard to organize each
steps
Step functions
• Makes workflow simple to orchestrate AWS Lambda functions

• Supports Parallel state

• built-in error han...
Single Video Processing
Function
Break logics
State Machine - 1st version
Issue #3
VPC makes trouble
VPC Issue
• Assigning Lambda to VPC introduces long cold-start time

• about 10 sec, even used 3008MB memory size

• impac...
VPC Issue
• Scaling Issue when invocation spikes
VPC Issue
• VPC was required for S3 access through S3
Endpoint only (to authenticate s3 access using VPCE
ID)

• Removed V...
Issue #4
Tiny ephemeral
storage (/tmp)
Tiny ephemeral storage (/
tmp)
• Ephemeral storage capacity is 512MB, too small for media processing

• You can’t add bloc...
Tiny ephemeral storage (/
tmp)
• S3 - only available option for now

• stream everything if possible
• use s3 as temporary...
Issue #5
Optimize container
reusability
Optimize container
reusability
• Invoking different functions 

=> lambda container cannot be reused
Optimize container
reusability
• HACK: Group multiple lambda functions to single lambda
function

• Inject task name using...
Processing Pipeline
Architecture - 1st version
Building Delivery block
Delivery block
• Currently we don’t need any DRM or AD
insertion solution

• Simple architecture

• Use S3 as origin, Clou...
Delivery block architecture
Connect blocks to
make pipeline
Media Pipeline Architecture
Media Pipeline Architecture
Media Pipeline Architecture
Media Pipeline Architecture
Launch
Appearing video uploads
Wait! There’s more!
2nd Version
• Improves encoding speed, output quality

• Support Parallel Encoding

• Split video streams to small chunks
...
Parallel Encoding
Break logics
State Machine - 2nd version
Results
• Fast processing without losing quality

• now processing f(x) M/V (1080p, 3m 33s) just takes 60 sec

• if input ...
Further Steps
• Support 1 hour+ video processing using Elastic Transcoder

• Support Live Streaming

• MediaLive? Fargate?...
Postmortem
•


• Elastic Transcode Elemental MediaConvert


• Processing pipeline
VOD 

• Lambda 

• Lambda
How about other media
types?

(Audio, GIF, Image…)
We’re hiring!

https://careers.vingle.net
Thank you

* * *
Serverless Media Workflow
Prochain SlideShare
Chargement dans…5
×

Serverless Media Workflow

14 833 vues

Publié le

4월 26일 AWSKRUG (AWS 한국 사용자 모임) 아키텍쳐 모임에서 발표한 "Serverless Media Workflow" 발표 슬라이드입니다.

Publié dans : Logiciels
  • Very Nice: See Latest Blogs @ https://www.thesisscientist.com/Blog
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Serverless Media Workflow

  1. 1. Serverless 
 Media Workflow
  2. 2. Speaker • Serverless Lover • Software Engineer @ Vingle • Freelancer Software Engineer @ Artistscard • Software Engineer @ Law&Company • https://github.com/mooyoul • https://mooyoul.github.io
  3. 3. A journey to
 Video Service 🎥
  4. 4. Video is becoming 
 the most important content
 🔥🔥🔥
  5. 5. Let’s add video feature on Vingle! 😎
  6. 6. …but how?
  7. 7. Common VOD workflow
  8. 8. Common Video workflow
  9. 9. Common Video workflow
  10. 10. Common Video workflow
  11. 11. Common Video workflow
  12. 12. Building Ingest block
  13. 13. Ingest Step Requirements • Support File-based workflow only for now • Simple upload workflow for clients • Fast upload speeds anywhere in the world • Durable, scalable, fast storage • No additional disk storage required for upload • No maintenance required
  14. 14. Ingest Step Requirements • Support File-based workflow only for now • Simple upload workflow for clients • Fast upload speeds anywhere in the world • Durable, scalable, fast storage • No additional disk storage required for upload • No maintenance required
  15. 15. S3 Upload using pre-signed URL • Allows client can upload directly to S3 using pre- signed URL • AWS-SDK supports pre- signed url generation
  16. 16. S3 Upload using pre-signed URL
  17. 17. …but we are using 
 us-east-1 region
  18. 18. some regional ISPs have bandwidth issue on international network transfer 💩
  19. 19. Slow Upload on some ISPs
  20. 20. Upload speed is so slow!🐢🐢🐢
  21. 21. Submarine cables 
 in South Korea
  22. 22. Submarine cables 
 in South Korea
  23. 23. Two options to accelerate S3 transfer 🚀
  24. 24. Two options to accelerate S3 transfer • Proxy file transfer traffic through CloudFront • Cloudfront can accelerate dynamic content • Uses Cloudfront Edge locations to optimize route path • Enable S3 Transfer Acceleration • Takes advantage of CloudFront’s globally distributed edge locations • More transfer optimizations
  25. 25. S3 Transfer Acceleration is faster than Cloudfront as Proxy
  26. 26. S3 Transfer Acceleration makes throughput faster
  27. 27. Ingest block architecture
  28. 28. Problem solved! ✅
  29. 29. Building Processing block
  30. 30. Processing Requirements • use fully-managed service if possible • scalable, reliable • adaptive bitrate streaming • Important: compatible with various platforms, devices • Desktop Web - IE 10+, Edge, Chrome, Firefox • Mobile Web - Mobile Safari, Chrome Mobie • Android - 4.1 ICS+ • iOS - 9+ • Most Important: Delivery video to users ASAP
  31. 31. Wait! 
 What is Adaptive Bitrate Streaming?
  32. 32. Adaptive Bitrate Streaming • From Wikipedia: • Adaptive bitrate streaming is a technique used in streaming multimedia over computer networks. • It works by detecting a user's bandwidth and CPU capacity in real time and adjusting the quality of a video stream accordingly. • The result: very little buffering, fast start time and a good experience for both high-end and low-end connections.
  33. 33. Adaptive Bitrate Streaming
  34. 34. Choosing Streaming Protocol
  35. 35. Choosing Streaming Protocol • Non-HTTP based protocols like RTMP are not in our interest • Looked for HTTP-based protocols, or methods • Progressive Download Single File via HTTP • 1st request: Range: 0-255 • 2st request: Range: 256-511 • 3rd request: Range: 512-767 • … • Apple HLS • MPEG-DASH
  36. 36. Choosing Streaming Protocol • Progressive Download Single File via HTTP • Apple HLS • MPEG-DASH
  37. 37. Choosing Streaming Protocol • Progressive Download Single File via HTTP • Apple HLS - Winner! • MPEG-DASH
  38. 38. Why we choose HLS
  39. 39. Progressive Download
 Single File via HTTP • Pros • Easy to implement • Cons • Implementing Adaptive Bitrate Streaming is so hard (almost impossible) without getting “choppy” frames, or “pop” gap sound between streams • Slow seeking/starting since container requires metadata reading • Large binary can occur lower cache hit • No Live Stream compatibility
  40. 40. MPEG-DASH • Pros • New standard • Based on Fragmented MPEG4 (similar to MPEG2 Transport Stream on HLS) • Broader codec support • Cons • IMO it is not production ready • Market, Platform, Community support is not good at this moment • Workflow was unstable during experiment
  41. 41. First Video Specification
  42. 42. First Spec - Protocol • HTTP Live Stream (HLS) • Apple invented standard, natively supported on macOS/iOS/tvOS • Natively supports adaptive bitrate streaming, Live Streaming, and DRM (Apple FairPlay) • Based on HTTP Protocol, Play well with transitional HTTP CDN • Trusted by many providers, Many stable implementations • Amazon, Youtube, Facebook, Twitter, Vimeo, Twitch, Akamai, New York Times, FOX TV, TED, VK, POOQ, SK Broadband TV … • Android - ExoPlayer • Web - MediaSource Extension is standard spec, HLS.js, Video.js
  43. 43. First Spec - Input • No codec limitation of input source
 
 … but expects most input source are H.264/AVC + AAC based • Max acceptable resolution: 1080p • Max acceptable bitrate: 10Mbps • Max acceptable duration: 5 minutes
  44. 44. First Spec - Output • Common • H.264/AVC1 + AAC LC • Many graphic chipset supports hardware accelerated decoding • Most popular codec in market • Apple HLS • Three variants for adaptive bitrate streaming • High Quality: Up to 1080p, 60p, High/4.1, 4Mbps • Medium Quality: Up to 720p, 30p, Main/4.0, 2Mbps • Low Quality: Up to 480p, 30p, Baseline/3.1 800kbps • Progressive Download Single File via HTTP for HLS fallback • Medium Quality only
  45. 45. First Spec - Output • 4 Outputs • HLS HQ • HLS MQ • HLS LQ • HTTP Progressive MP4 MQ
  46. 46. Check requirements
  47. 47. Requirements • use fully-managed service if possible • scalable, reliable • adaptive bitrate streaming • Important: compatible with various platforms, devices • Desktop Web - IE 10+, Edge, Chrome, Firefox • Mobile Web - Mobile Safari, Chrome Mobie • Android - 4.1 ICS+ • iOS - 9+ • Important: Delivery video to users ASAP
  48. 48. Requirements • use fully-managed service if possible • scalable, reliable • adaptive bitrate streaming • Important: compatible with various platforms, devices • Desktop Web - IE 10+, Edge, Chrome, Firefox • Mobile Web - Mobile Safari, Chrome Mobie • Android - 4.1 ICS+ • iOS - 9+ • Important: Delivery video to users ASAP
  49. 49. Requirements • use fully-managed service if possible • scalable, reliable • Important: Delivery video to users ASAP
  50. 50. AWS provides bunch of media products 🎉
  51. 51. Video Encoding Products • AWS Elastic Transcoder • Fully-managed service, pay only for what you use (per minute of output video) • HLS Support • Configurable Presets • AWS Elemental MediaConvert • Newly launched, Fully-managed service, pay only for what you use (per minute of output video) • HLS Support • Broadcasting-grade, high quality output, Professional Codec Support (e.g. ProRes) • Configurable Presets • In some scenarios, MediaConvert is cheaper than Elastic Transcoder • New! Basic Tier for Simple Web Videos
  52. 52. …or build your own video encoding service
  53. 53. Most time-consuming task is video encoding
  54. 54. Tested two products
  55. 55. Test conditions • Input • f(x) 4 Walls M/V (3m 33s) • H.264/AVC, 1080p, 23.97fps, 8Mbps, High/4.1 • AAC LC, 192k, 48KHz, 2 Channels • Mr. Robot Clip (52m 54s) • H.264/AVC, 720p, 23.97fps, 1.5Mbps, High/4.1 • AC3, 384k, 48KHz, 6 Channels • Output • Single HLS Output • H.264/AVC, 1080p, 23.97fps, ABR 3Mbps (Max 4.5Mbps), High/4.1 • AAC LC, 128k, 48KHz, 2 Channels • No Thumbnail Generation
  56. 56. Elastic Transcoder
  57. 57. Elastic Transcoder
  58. 58. Elemental MediaConvert
  59. 59. Elemental MediaConvert
  60. 60. Results • Overall workflow was stable • Quality • The output quality of both was similar • Fixed GOP => Consistant segment duration • Processing Speed (sorted by faster) • Elastic Transcoder • MediaConvert Basic • MediaConvert Pro 
 
 

  61. 61. Problems • Pricing is quite expensive • Needed more faster encoding speed without losing quality • Missing CRF (Constant Rate Factor) bitrate mode support • Can’t control encoder options • MediaConvert has better encoder option (e.g . quantize algorithm - CABAC) • Can’t control video filters
 (e.g. lanczos for resizing, yadif for deinterlacing, color filters like instagram) • There’s no way to remux video
  62. 62. Make our own processing workflow 💥
  63. 63. Three ways to run job • Batch - EC2 • Fargate - ECS • Step Functions + Lambda
  64. 64. Three ways to run job • Batch - EC2 • Fargate - ECS • Step Functions + Lambda - WINNER!
  65. 65. Three ways to run job • Batch - EC2 • Requires additional instance provisioning time (extremely slow) • Fargate - ECS • Requires additional container provisioning time
 (about 30sec~60sec) • Setting High CPU power requires High Memory Size • Lambda • Computing Power is quite slower than EC2/Fargate instances • …but now we have 3008MB Memory option, which has faster processing power • No additional provisioning time required
  66. 66. Video Processing Workflow
  67. 67. Video Processing Workflow
  68. 68. Single Video Processing Function
  69. 69. Issue #1
  70. 70. How to get encoding software running on AWS Lambda?
  71. 71. Lambda-runtime compatible build • Linux program • Lambda execution runtime is based on Amazon Linux AMI (amzn-ami-hvm-2017.03.1.20170812- x86_64-gp2) • Instead of using build instance, just make build image which is based on amazon linux docker image • docker pull amazonlinux:2017.03.1.20170812 • vim Dockerfile # Be careful updating openssl. it can make incompatible shared library issue • docker build . -t my-awesome-lambda-build-env • docker run --rm -v $PWD:/workspace my-awesome-lambda-build-env /workspace/build.sh • Test built binaries or libraries using lambci lambda image • docker pull lambci/lambda:nodejs8.10 • docker run --rm -v $PWD:/workspace --entrypoint bash lambci/lambda:nodejs8.10 / workspace/test.sh
  72. 72. Lambda-runtime compatible build • Node.js Native Addons • lambci provides docker images to rebuild native addons • docker pull lambci/lambda:build-nodejs8.10 • docker run --rm -v $PWD:/var/task lambci/ lambda:build-nodejs8.10 # runs “npm rebuild”
  73. 73. Issue #2
  74. 74. Timeout issues Hard to organize each steps
  75. 75. Step functions • Makes workflow simple to orchestrate AWS Lambda functions • Supports Parallel state • built-in error handling, retry, fallback conditions • Useful patterns • Delay execution for X seconds or until specific date/time • Recursive Execution using Choice State • Scan all records from DynamoDB table • dequeue messages from SQS until queue is empty
  76. 76. Single Video Processing Function
  77. 77. Break logics
  78. 78. State Machine - 1st version
  79. 79. Issue #3
  80. 80. VPC makes trouble
  81. 81. VPC Issue • Assigning Lambda to VPC introduces long cold-start time • about 10 sec, even used 3008MB memory size • impacts overall video processing time • Occasionally we had “connection timeout” issues • AWS Support confirmed sometimes ENI may not have internet connectivity • we had to implement our own retry logic
  82. 82. VPC Issue • Scaling Issue when invocation spikes
  83. 83. VPC Issue • VPC was required for S3 access through S3 Endpoint only (to authenticate s3 access using VPCE ID) • Removed VPC configuration • Switched to stream data using pre-signed urls or download object using AWS-SDK • reduced cold-start time (<40ms) • “connection timeout” issue has gone
  84. 84. Issue #4
  85. 85. Tiny ephemeral storage (/tmp)
  86. 86. Tiny ephemeral storage (/ tmp) • Ephemeral storage capacity is 512MB, too small for media processing • You can’t add block device to lambda • NFS / EFS • Lambda does not provide root permission • so you cannot mount disks - you cannot use NFS in Lambda • Ramdisk • Ramdisk like /dev/shm is not available on Lambda • Lambda does not provide root permission • so you cannot create or mount ramdisk
  87. 87. Tiny ephemeral storage (/ tmp) • S3 - only available option for now • stream everything if possible • use s3 as temporary storage • Video Input - S3 supports seeking object using Range request header => can be streamed • Video Output - may vary depending on container • MP4 - requires seekable output => you cannot pipe output => you cannot stream outputs • HLS - does not support writing outputs to pipes, since HLS consists of several files • Watch file system (e.g. chokidar) • Upload segments to s3 if generated • Removed uploaded file • Note: Higher memory size option provides faster Disk I/O and Network I/O • Note: S3 Object keys are stored in UTF-8 binary ordering across multiple partitions in the index
 => UUID key prefix helps to reduce first byte latency
  88. 88. Issue #5
  89. 89. Optimize container reusability
  90. 90. Optimize container reusability • Invoking different functions 
 => lambda container cannot be reused
  91. 91. Optimize container reusability • HACK: Group multiple lambda functions to single lambda function • Inject task name using Pass State
  92. 92. Processing Pipeline Architecture - 1st version
  93. 93. Building Delivery block
  94. 94. Delivery block • Currently we don’t need any DRM or AD insertion solution • Simple architecture • Use S3 as origin, Cloudfront as CDN • Use maximum TTL to maximize cache hit
  95. 95. Delivery block architecture
  96. 96. Connect blocks to make pipeline
  97. 97. Media Pipeline Architecture
  98. 98. Media Pipeline Architecture
  99. 99. Media Pipeline Architecture
  100. 100. Media Pipeline Architecture
  101. 101. Launch
  102. 102. Appearing video uploads
  103. 103. Wait! There’s more!
  104. 104. 2nd Version • Improves encoding speed, output quality • Support Parallel Encoding • Split video streams to small chunks • Split Video/Audio encoding process • Encode streams per each preset, not outputs • HLS HQ / HLS MQ / HLS LQ / MP4 MQ
 => Encode streams to HQ / MQ / LQ and assemble them • Bypass encoding if input is compatible with our spec • Skip HLS encoding process if source quality is not good, or duration is too short • Support GIF input/output
  105. 105. Parallel Encoding
  106. 106. Break logics
  107. 107. State Machine - 2nd version
  108. 108. Results • Fast processing without losing quality • now processing f(x) M/V (1080p, 3m 33s) just takes 60 sec • if input source is compatible with our spec, processing can be done within <10 sec • Great output quality • Saving cost • encoding is not required if input source is compatible with our spec • Stable workflow • There’s no resource to manage • Acceptable max bitrate: Up to 300Mbps (Tested with ProRes source)
 Acceptable max resolution: 4K
 Acceptable codec: HEVC, ProRes
 Acceptable max duration: 1 hour
  109. 109. Further Steps • Support 1 hour+ video processing using Elastic Transcoder • Support Live Streaming • MediaLive? Fargate? • Finding solution for WebRTC gateway to relay live video stream • Video Advertisement • MediaTailor? • Metrics, Management Tools … • NSFW filtering - rekognition video or rekognition
  110. 110. Postmortem
  111. 111. • • Elastic Transcode Elemental MediaConvert • Processing pipeline VOD • Lambda • Lambda
  112. 112. How about other media types?
 (Audio, GIF, Image…)
  113. 113. We’re hiring! https://careers.vingle.net
  114. 114. Thank you
 * * *

×