Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Real-Time Data Processing Using AWS Lambda - September Webinar Series

1 810 vues

Publié le

Serverless architectures can eliminate the need to provision and manage servers required to process files or streaming data in real time.

In this session, we will cover the fundamentals of using AWS Lambda to process data from sources such as Amazon DynamoDB Streams, Amazon Kinesis, and Amazon S3. We will walk through sample use cases for real-time data processing and discuss best practices on using these services together. We will then demonstrate  run a live demonstration on how to set up a real-time stream processing solution using just Amazon Kinesis and AWS Lambda, all without the need to run or manage servers.

Learning Objectives:
• Learn the fundamentals of using AWS Lambda with various AWS data sources
• Understand best practices of using AWS Lambda with Amazon Kinesis

Publié dans : Technologie
  • Soyez le premier à commenter

Real-Time Data Processing Using AWS Lambda - September Webinar Series

  1. 1. © 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cecilia Deng Software Engineer AWS Lambda Stream Processing
  2. 2. Agenda Overview of Lambda Overview of Kinesis to Lambda event source Configuring the event source Lambda side processing
  3. 3. AWS Lambda: Overview Lambda functions: a piece of code with stateless execution Triggered by events: • Direct Sync and Async API calls • AWS Service integrations • 3rd party triggers • And many more … Makes it easy to: • Perform data-driven auditing, analysis, and notification • Build back-end services that perform at scale
  4. 4. Amazon Kinesis: Overview Real-time Ingest Highly Scalable Durable Replay-able Reads Continuous Processing GetShardIterator and GetRecords(ShardIterator) Allows checkpointing/ replay Enables multi concurrent processing KCL, Firehose, Analytics, Lambda • Managed Service - makes it easy to provision, deploy, manage your stream • Low end-to-end (real time) latency • Pay as you go, no upfront cost • Enables data movement to many stores or processing engines
  5. 5. Data Processing/Streaming Architecture & Workflow Smart Devices Click Stream Log Data
  6. 6. AWS Lambda and Amazon Kinesis integration How it Works Pull event source model: ▪ Kinesis mapped as Event Source in Lambda ▪ Lambda polls the stream and batches available records ▪ Batches are passed for invocation to Lambda through function parameters What this means: ▪ Fleet of pollers running a flavor of KCL ▪ No resource policy
  7. 7. AWS Lambda and Amazon Kinesis integration Event structure: ▪ Event received by Lambda function is a collection of records from Kinesis stream
  8. 8. AWS Lambda and Amazon Kinesis integration Synchronous invocation: ▪ Lambda invoked as synchronous RequestResponse type ▪ Lambda honors Kinesis at least once semantics ▪ Each shard blocks on in order synchronous invocation
  9. 9. AWS Lambda and Amazon Kinesis integration • Multiple functions can be mapped to one stream • Multiple streams can be mapped to one Lambda function • Each mapping is a unique key pair Kinesis stream to Lambda function • Each mapping has unique shard iterators Amazon Kinesis 1 AWS Lambda 1 Amazon CloudWatc h Amazon Dynamo DB AWS Lambd a 2 Amazo n S3Amazon Kinesis 2
  10. 10. DEMO
  11. 11. Creating a Kinesis stream Streams ▪ Made up of Shards ▪ Each Shard supports writes up to 1MB/s ▪ Each Shard supports reads up to 2MB/s across maximum 5 transactions/s Data ▪ All data is stored and replayable for 24 hours ▪ A Partition Key is supplied by producer and used to distribute the PUTs across Shards (using MD5 function to hash to 128-bit integers) ▪ A unique Sequence # is returned to the Producer upon a successful PUT call ▪ Make sure partition key distribution is even to optimize parallel throughput ▪ Pick a key with more groups than shards
  12. 12. Creating a Lambda function Memory: ▪ CPU and disk proportional to the memory configured ▪ Increasing memory makes your code execute faster (if CPU bound) ▪ Increasing memory allows for larger record sizes processed Timeout: ▪ Increasing timeout allows for longer functions, but more wait in case of errors Retries: ▪ For Kinesis, Lambda retries until the data expires (default 24 hours) Permission model: • The execution role defined for Lambda must have permission to access the stream
  13. 13. Configuring the Event Source Batch size: ▪ Max number of records that Lambda will send to one invocation ▪ Not equivalent to how many records Lambda will poll ▪ Effective batch size is every 250 ms MIN(records available, batch size, 6MB) ▪ Increasing batch size allows fewer Lambda function invocations with more data processed per function
  14. 14. Configuring the Event Source Starting Position: ▪ The position in the stream where Lambda starts reading ▪ Set to “Trim Horizon” for reading from start of stream (all data) ▪ Set to “Latest” for reading most recent data (LIFO) (latest data)
  15. 15. Processing Per Shard: ▪ Lambda calls GetRecords with max limit from Kinesis (10k or 10MB) ▪ If no record, wait 250 ms ▪ From in memory, sub batches and formats records into Lambda payload ▪ Invoke Lambda with synchronous invoke … … Source Kinesis Destination 1 Lambda Poller Destination 2 Functions Shards Lambda will scale automaticallyScale Kinesis by adding shards Batch sync invokesPolls
  16. 16. Processing ▪ Lambda blocks on ordered processing for each individual shard ▪ Increasing # of shards with even distribution allows increased concurrency ▪ Batch size may impact duration if the Lambda function takes longer to process more records … … Source Kinesis Destination 1 Lambda Poller Destination 2 Functions Shards Lambda will scale automaticallyScale Kinesis by adding shards Batch sync invokesPolls
  17. 17. Processing ▪ Maximum theoretical throughput : # shards * 2MB / (s) ▪ Effective theoretical throughput : # shards * batch size (MB) / Lambda function duration (s) ▪ If put / ingestion rate is greater than the theoretical throughput, your processing is at risk of falling behind Common observations: ▪ Effective batch size may be less than configured during low throughput ▪ Effective batch size will increase during higher throughput ▪ Number of invokes and GetRecord calls will decrease with increased Lambda duration
  18. 18. Processing ▪ Retry execution failures until the record is expired ▪ Retry with exponential backoff up to 60s ▪ Throttles and errors impacts duration and directly impacts throughput ▪ Effective theoretical throughput : ( # shards * batch size (MB) ) / ( function duration (s) * retries until expiry) Kinesis Destination 1 Destination 2 … … Source Functions Shards Lambda will scale automaticallyScale Kinesis by adding shards Receives errorPolls Receives error Receives success Lambda Poller
  19. 19. Monitoring Kinesis •GetRecords (effective throughput) : bytes, latency, records etc •PutRecord : bytes, latency, records, etc •GetRecords.IteratorAgeMilliseconds: how old your last processed records were. If high, processing is falling behind. If close to 24 hours, records are close to being dropped.
  20. 20. Monitoring Lambda •Monitoring: available in Amazon CloudWatch Metrics • Invocation count • Duration • Error count • Throttle count •Debugging: available in Amazon CloudWatch Logs • All Metrics • Custom logs • RAM consumed • Search for log events
  21. 21. Best Practices ▪ Create enough shards for parallel processing ▪ Distribute load evenly across shards ▪ Monitor and address Lambda errors and throttles ▪ Monitor Kinesis limits and throttles
  22. 22. Best Practices Create different Lambda functions for each task, associate to same Kinesis stream Log to CloudWatch Logs Push to SNS
  23. 23. Get Started: Data Processing with AWS Next Steps 1. Create your first Kinesis stream. Configure hundreds of thousands of data producers to put data into an Amazon Kinesis stream. Ex. data from Social media feeds. 2. Create and test your first Lambda function. Use any third party library, even native ones. First 1M requests each month are on us! 3. Read the Developer Guide, AWS Lambda and Kinesis Tutorial, and resources on GitHub at AWS Labs • http://docs.aws.amazon.com/lambda/latest/dg/with-kinesis.html • https://github.com/awslabs/lambda-streams-to-firehose
  24. 24. Thank you!