This document discusses using AWS Lambda and Step Functions to orchestrate serverless functions. It provides an overview of Lambda and Step Functions, demonstrates how to define a Step Function state machine using the Serverless Framework, and discusses monitoring Lambda and Step Functions using Datadog.

1. © 2018, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Alex Poe
Solutions Engineer Team Lead, Datadog
Lambda, Step Functions and
Datadog: A Symphony

2. Who am I
Alex Poe
Lead Solutions Engineer, APAC
Datadog
@alextpoe
alex.poe@datadoghq.com

3. Who is Datadog
SaaS-based Monitoring & Analytics
Infrastructure, APM, Logs
Open Source Agent
Trillions of data
points per day

4. What is Lambda
• Serverless
• As far as you are concerned
• Don’t pay for idle
• Scale with ease
• FaaS
• Node 6.10 (java, go, python, c#)

6. …but AWS Lambda != ec2

7. …but AWS Lambda != application

8. …applications are complicated

9. …lambdas should respect the
single responsibility principle…

12. we can create many serverless functions

13. we can create many serverless functions
but how do we orchestrate them?

14. Amazon Kinesis Amazon SQS Amazon DynamoDB
Event/Message/Database as Traffic Cop

15. https://github.com/Nordstrom/hello-retail

17. Enter AWS Step Functions

18. a step function is

19. a step function is
a state machine

20. but what about SWF?
(Amazon Simple Workflow Service)

21. AWS Step Function Concepts
• State Machine
• Transitions
• Executions
• States
• Task
• Choice
• Succeed/Fail
• Pass
• Wait
• Parallel
• You define the State Machine with the Amazon
States Language
• A diagram will be generated based on that
structure

22. State Example: Task

23. State Example: Choice

24. State Example: Succeed / Fail

25. State Example: Pass

26. State Example: Wait

27. State Example: Parallel

29. Many ways to build Step Functions
Serverless Framework

30. Benefits of Serverless Framework
• Define everything in a single file:
• IAM Role Statements
• Lambda Function Handlers
• Lambda Function Triggers
• Step Function States
• CloudFormation for other things

31. serverless.yml
service:
name: mattw-reinvent2017-build-website
plugins:
- serverless-pseudo-parameters
- serverless-step-functions
- serverless-webpack
custom:
repoName: reinvent2017-website

32. serverless.yml cont’d
provider:
name: aws
runtime: nodejs6.10
stage: ${opt:stage, 'dev'}
region: ${opt:region, 'us-east-1'}
iamRoleStatements:
- Effect: "Allow"
Action:
- "states:StartExecution"
Resource:
- ${self:resources.Outputs.MyStateMachine.Value}
- Effect: "Allow"
Action:
- "codecommit:GetCommit"
- "codecommit:GetDifferences"
Resource:
- arn:aws:codecommit:${self:provider.region}:#{AWS::AccountId}:${…custom.repoName}

33. serverless.yml cont’d
functions:
buildSite:
handler: handler.build
whatChanged:
handler: handler.whatChanged
startStepFunction:
handler: handler.startStepFunction
events:
- sns: arn:aws:sns:${…provider.region}:#{AWS::AccountId}:mattw-reinvent-websitebuild
environment:
statemachine_arn: ${self:resources.Outputs.MyStateMachine.Value}

34. serverless.yml cont’d
stepFunctions:
stateMachines:
FirstStateMachine:
name: myStateMachine
definition:
StartAt: WhatChanged
States:
WhatChanged:
Type: Task
Resource: arn:aws:lambda:#{AWS::Region}:#{AWS::AccountId}
:function:${self:service}-${self:provider.stage}-
whatChanged
Next: BuildDockerOrNot

35. AWS
CodeCommit
Amazon
SNS
AWS
Lambda

36. startStepFunction:
handler: handler.startStepFunction
events:
- sns: arn:aws:sns…mattw-reinvent-websitebuild
environment:
statemachine_arn: ${self:reso…puts.MyStateMachine.Value}

37. StartStepFunction
Get Step Function ARN
Get SNS details
Start SF Execution

38. 1. What changed in the repo?
2. If docker-related, build docker
1. Tell CodeBuild to rebuild docker
image
2. Wait for it
3. Push to ECS repo
3. Use the docker image to build the
website
4. Wait for it
5. Send Matt an SMS

39. WhatChanged

40. WhatChanged:
Type: Task
Resource: arn:aws:lambda:...
Next: BuildDockerOrNot
Retry :
- ErrorEquals:
- Lambda.Unknown
States.ALL
IntervalSeconds: 3
MaxAttempts: 4

41. WhatChanged
Get the commit that triggered the sf
Get the previous commit
Whats different
If it was docker related
return hasDocker = true
else
return hasDocker = false

42. BuildDockerOrNot

43. BuildDockerOrNot:
Type: Choice
Choices:
- Variable: "$.hasDocker"
BooleanEquals: true
Next: BuildDocker
- Variable: "$.hasDocker"
BooleanEquals: false
Next: BuildSite

44. BuildDocker:
Type: Task
Resource: arn:aws:lambda…buildDocker
Next: WaitForDockerBuild

45. BuildDocker
Start the CodeBuild for docker image

46. WaitForDockerBuild:
Type: Wait
Seconds: 20
Next: IsTheContainerBuilt

47. IsTheContainerBuilt:
Type: Task
Resource: arn:aws:lambda...
Next: ContainerBuiltOrNot

48. IsTheContainerBuilt
Get build info from CodeBuild
Check if its done

49. IsTheContainerBuild

50. ContainerBuiltOrNot

51. ContainerBuiltOrNot:
Type: Choice
Choices:
- Variable: "$.containerBuildDone"
BooleanEquals: true
Next: BuildSite
- Variable: "$.containerBuildDone"
BooleanEquals: false
Next: WaitForDockerBuild

52. 1. What changed in the repo?
2. If docker-related, build docker
1. Tell CodeBuild to rebuild docker
image
2. Wait for it
3. Push to ECS repo
3. Use the docker image to build the
website
4. Wait for it
5. Send Matt an SMS

53. How about monitoring?
We monitor Lambda via Logs
MONITORING|unix_epoch_timestamp|value|count|
my.metric.name|#tag1:value,tag2
console.log(monitoringstring);

54. What are the Lambda Metrics
aws.lambda.duration, min, max, sum
aws.lambda.errors
aws.lambda.invocations
aws.lambda.throttles
aws.lambda.iterator_age

55. What are the Step Function Metrics
aws.states.execution_time (.maximum, .minimum)
aws.states.executions_aborted, failed, started, succeeded, timed_out
aws.states.lambda_functions_scheduled, started, succeeded, timed_out
aws.states.execution_throttled
aws.states.lambda_function_time (.maximum, .minimum)
aws.states.lambda_function_run_time (.maximum, .minimum)
aws.states.lambda_function_schedule_time (.maximum, .minimum)
aws.states.lambda_functions_started

56. Conclusion
• Lambda Basics
• It’s Serverless!
• Step-Functions
• Orchestrate
• Demo
• Monitoring

57. Questions?
Alex Poe
Lead Solutions Engineer, APAC
Datadog
@alextpoe
alex.poe@datadoghq.com