Even if a Serverless Function is small by design, testing it business logic and integration with other cloud components is kind of hard. Good to have a "Survival Guide".

1. The utlimate
SERVERLESS
TESTING GUIDE
#WISSENTEILEN
THE PRACTICAL SKILLS YOU‘LL
NEED TO TEST SERVERLESS
by Lars Röwekamp
a.k.a @mobileLarson

2. ABOUT OPEN KNOWLEDGE
Software development and IT consulting

3. ABOUT ME
Who am i?
• CIO New Technologies
• Enterprise & Mobile
• Author, Speaker, Coach & Mentor
• Snowboard & MTB Enthusiast
• Traveller between the worlds
Lars Röwekamp (a.k.a. @mobileLarson)
LR

4. Testing
Serverless

5. Shouldn‘t be to
complicated!- famous last words -

6. Testing is about …
gaining confidence in your
code running as expected on
a specific target platform.

7. Testing is about …
minimizing the risk that
something unexpected will
happen in production.

8. „What does this mean in the
context of Serverless?"

9. „Run Code,
not Servers!“

10. #WISSENTEILEN
“Run your business code
highly-available
in the cloud in response
to events and scale
without any servers to
manage.“*
*(AWS Lambda product description)

11. #WISSENTEILEN
“Run your business code
highly distributed
and event driven in a non
transparent environment
with no single
point of control.“*
*(my personal interpretation)

13. The Road to the Cloud ...
The Serverless Showcase

14. AWS Cloud
Use-Case: Upload Image
upload image
with additional
information

15. AWS Cloud
Store raw Image
1
Use-Case: Upload Image
upload image
with additional
information

16. AWS Cloud
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

17. AWS Cloud
AWS Step Functions workflow: Store Image
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

18. AWS Cloud
AWS Step Functions workflow: Store Image
Create ThumbnailStore raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

19. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

20. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

21. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

22. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

23. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2
Use-Case: Upload Image
upload image
with additional
information

24. „What can go
wrong?“

25. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

26. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

27. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

28. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

29. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

30. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

31. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

32. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

33. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

34. AWS Cloud
AWS Step Functions workflow: Store Image
Create Thumbnail
Inform Subscribers
Store raw Image
Store Image Information
1
2

35. What could possibly go wrong?
• internal programming /business logic errors
• abnormalities (e.g. unexpected or invalid calls)
• lost or duplicated events
• high latency / timeouts
• security attacks (e.g. DoS/DDoS)
• personalized SLAs / usage plans / unexpected peaks of workload
• customer location changes
• …

36. „Run Code, not Servers“, they said!
„It will be fun“, they said

37. What, how and where should we test to …
• gain confidence in our code
• minimize risk of failures in production

38. Testing, the traditional way

39. Testing, the serverless way
„The biggest complexity is not within
the function itself, but in how it interacts
with other functions and services
(a.k.a. cloud components).“

40. Testing, the serverless way
Remember goals of testing: „minimize risk“
• configuration risk
• technical workflow risk
• business logic risk
• integration risk

41. Testing, the serverless way
Remember goals of testing: „minimize risk“
• configuration risk
• technical workflow risk
• business logic risk
• integration risk

42. Testing, the serverless way

43. Unit Testing
… for Serverless Functions
„Make sure that your business logic is
working as expected.“
• business logic errors
• coding errors
• …

44. Unit Testing
… for Serverless Functions
• seperation of logic and infrastructure
• mock dependencies
• jUnit, testNG, Mockito
• Jasmin, MochaJS, JEST

45. Integration Testing
… for Cloud Components
„Make sure that the integration of your
serverless function with all the other
cloud components works as expected.“
• wrong DynamoDB query
• unexpected response format
• ….

46. Integration Testing
… for Cloud Components
• dev/test components in the cloud*
• mock/fake cloud components**
• S3rver
• dynamodb-local
• kinesalite
• node-lambda
• fake-sns
• elasticmq* could become kind of expensive
** potential problem of self fulfilling prophecy

47. End-to-End Testing
… for User-Stories
„Make sure that chains of serverless
functions and cloud components work as
expected - inside the target environment.“
• wrong IAM permissions
• timeouts not set correctly
• not enough memory allocated

48. End-to-End Testing
… for User-Stories
• cloud test stages*
• local cloud emulation**
• SAM local (AWS)
• LocalStack (localstack.org)
• Serverless Offline (Serverless Framework)* could become kind of expensive
** emulate only part of the real world restrictions

49. Best Practices

50. #1 seperate business logic from infrastructure inside your functions
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
integration test candidate
end-to-end test candidate
u

51. Infrastructure Business Logic

52. Infrastructure Business Logic

53. Business Logic Tests

54. #2 mock cloud infrastructure components of your functions
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
integration test candidate
end-to-end test candidate
u
um

55. Infrastructure

56. AWS agnostic logger (log4j2)

57. fake infrastructure component (Context)

58. mock infrastructure component (Context)

59. #3 use local environment for functional testing (e.g. SAM local)
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
integration test candidate
end-to-end test candidate
uvia SAM local
via SAM local
SAM
yaml
TEST
u
u

61. $ sam local invoke "Greetings" -e event-greeting.json --env-vars env.json
function name payload for function

62. $ sam local invoke "Greetings" -e event-greeting.json --env-vars env.json
function name payload for function

63. #WISSENTEILEN
#4 use local environment for trigger integration testing
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
integration test candidate
end-to-end test candidate
uvia SAM local
via SAM local
SAM
yaml
TEST
u
u
i
i

64. $ sam local start-api –p 8080

65. $ sam local start-api –p 8080

66. $ sam local start-api –p 8080

67. #WISSENTEILEN
#5 use local „cloud“ components for basic integration testing
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
integration test candidate
end-to-end test candidate
u
via DynamoDB local
via FakeS3 via SAM local
via SAM local
SAM
yaml
TEST
u
u
i
i
i
i
WARNING: local cloud
components can only proof
functional correctness but
not infrastructural
correctness like DLQs,
timeouts, throttling, SLAs, …

68. $ sam local generate-event [SERVICE] [OPTION]
Simulate Component Event to trigger Lambda

69. $ sam local genarte-event [SERVICE] [OPTION]
Simulate Component Event to trigger Lambda

70. Simulate Component triggered by Lambda
$ aws –endpoint-url=http://localhost:8000 dynamodb list-tables
$ java –jar DynamoDBLocal.-jar

71. Simulate Component triggered by Lambda
$ aws –endpoint-url=http://localhost:8000 dynamodb list-tables

72. Simulate Component triggered by Lambda
$ aws –endpoint-url=http://localhost:8000 dynamodb list-tables

73. #WISSENTEILEN
#6 use temporary integration cloud for partial integration testing
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
integration test candidate
end-to-end test candidate
uvia SAM local
via SAM local
SAM
yaml
TEST
u
u
via DynamoDB local
via FakeS3
i
i
Temorary Intregration #Dev1
ii
INT
i
i

74. #WISSENTEILEN
#7 use permanent integration cloud for end-to-end testing
Testing Best Practices
AWS CloudOn-Premise
handler
logic
unit test candidate
e
i
u
integration test candidate
end-to-end test candidate
uvia SAM local
via SAM local
SAM
yaml
TEST
u
u
via DynamoDB local
via FakeS3
i
i
Permament IntregrationINT
e
e
e
e
i
i

75. „Are we
done?“

76. #WISSENTEILEN

77. Testing in Production
Remember goals of testing: „gain confidence“
• outages of cloud & cloud components
• outages of 3rd party apps you depend on
• bugs / problems related to scale

78. Testing in Production
Robust monitoring and error reporting
• logging
• tracing
• metrics
• alerting
Anticipation of disruption
and automated recovery!

79. Testing in Production
Chaos Engineering
• Inject small doses of failure into your system!

80. seperate business logic from infrastructure inside your functions
mock cloud infrastructure components of your functions
use local environment for functional testing (e.g. SAM local)
use local environment for trigger integration testing
use local „cloud“ components for basic integration testing
use temporary integration cloud for partial integration testing
use permanent integration cloud for end-to-end testing
REMEBER: Testing does not end in production!
Testing Summary
unit test
e
i
u
integration test
end-to-end test
e
i
i
i
u
u
i

81. Testing does not end
in production!