We’d like to introduce you to one of the most devastating ways to cause service instability in modern micro-service architectures: application DDoS. A specially crafted application DDoS attack can cause cascading system failures often for a fraction of the resources needed to conduct a more traditional DDoS attack. By Scott Behrens and Jeremy Heffner

1. Starting the Avalanche:
Application DoS In Microservice Architectures
Scott Behrens
Jeremy Heffner

2. $1.71

3. Photo of a battering ram by Flickr user Patrick Denker; License: https://creativecommons.org/licenses/by/2.0/

4. Jami al-Tawarikh (Compendium of Chronicles) by Rashid ad-Din. Edinburgh University Library via Wikipedia Commons.

5. Outline
What is Application DDoS (3 minutes)
Intro to Microservices (4 minutes)
Application DDoS in Microservices (7 mins)
Framework Discussion (9 minutes)
Case Study/Introduction to Repulsive Grizzly and Cloudy Kraken (15 minutes)
Demo (2 minutes)
Mitigation Strategies (3 minutes)
Future Work/Call to Action (2 minutes)

6. DDoS focused on
application layer logic

7. How Novel is Application DDoS?
Akamai’s State of the Internet Security Report." https://www.akamai.com. N.p., 19 Feb. 2017. Web.

8. Microservices are a
collection of small loosely
coupled but collaborative
services

9. Who Uses Microservices?

10. Microservice Primer: API Gateways and Client Libraries
Interface for middle tier
services
Services provide client
libraries to API Gateway
Diagrams provided with permission by microservices.io

11. Microservice Primer: Circuit Breaker
Helps with handling service failures
How do you know what timeout to
choose?
How long should the breaker be triggered?
Diagrams provided with permission by microservices.io

12. Microservice Primer: Cache
Speeds up response time
Reduces load on services
fronted by cache
Reduces the number of servers
needed to handle requests
https://github.com/netflix/evcache

13. Common Application DDoS
CPU
Mem
Cache
Disk
Network

14. Less Common But Interesting Application DDoS
Queueing
Client Library Timeouts
Healthchecks
Connection Pool
Things That Don’t Autoscale

15. How Logical Work Per Request Differs
Most Monolithic Application DDoS
Often 1 to 1
Some Microservice Application DDoS
Often 1 to Many

16. PROXY
PROXY
PROXIES
API
GATEWAY
WEBSITE
Middle Tier Service
Middle Tier Service
Middle Tier Service
Middle Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Edge Middle Backend
New School Microservice API DDoS
> python grizzly.py
…

17. PROXY
PROXY
PROXIES
API
GATEWAY
WEBSITE
Middle Tier Service
Middle Tier Service
Middle Tier Service
Middle Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Edge Middle Backend
New School Microservice API DDoS
> python grizzly.py
…
Fallback or
Site Error
API Gateway making
many client requests
Middle tier services
making many calls to
backend services
Backend service
queues filling up with
expensive requests
Client Timeouts, circuit
breakers triggered,
fallback experience
triggered

18. API GATEWAY
Middle Tier Service
Middle Tier Service
Middle Tier Service
Middle Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Edge Middle Backend
Microservice API DDoS - Simplified
Single
request
asking for
1,000
objects not
in cache

19. Single Request
Asking for 1,000
Objects not in
Cache*

20. Cache Miss Attack
Figure out what is in the cache
Make calls that require lookups outside of cache

21. API GATEWAY
Middle Tier Service
Middle Tier Service
Middle Tier Service
Middle Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Edge Middle Backend
Microservice API DDoS - Simplified
Middle tier has to call 3
backend services, or
6,000 calls.
Single
request
asking for
1,000
objects not
in cache
Middle tier client
library doesn’t
support batching
Results in 2 RPC calls
per object, or 2,000
RPC calls

22. Workflow for Identifying Application DDoS - Part 1
Identify the most latent service calls
Investigate if latent calls allow for manipulation
Determine API Gateway error conditions, thresholds, and library timeouts
Tune payload to fly under WAF/Rate Limiting
Test hypothesis
Scale your test using Cloudy Kraken (orchestrator) and Repulsive Grizzly (attack
framework)

23. Identifying Latent Service Calls

24. Example Workflow - Ordinary API Call
POST data seems to be encrypted.
Modifying POST body seems to return
fast error code
Changing I/O didn’t result in increased
latency of API calls

25. Example Workflow - Interesting API Call
Adding more items to list results in
longer response time
Adding too many items results in a
special error code (exp. 502)
Changing I/O resulted in increased
latency of API calls

26. Identifying Latent Service Calls

27. Microservice Application DDoS: Attack Patterns
Range
Object Out per Object in
Request Size
Combination

28. Application DDoS Technique: Range

29. Application DDoS Technique: Object Out Per Object In

30. Application DDoS Technique: Request Size

31. Application DDoS Technique: Combination
<--What about N languages?
<--What about more object fields?

32. Build a List of Indicators on API Health
Healthy Response
API Gateway Timeout
Client Library Timeout
WAF
Rate Limiter
Framework Exceptions

33. Other Indicators
HTTP 200 + Latency
Empty Response
Look for Correlations

34. Rate Limited*
Service May Scale Up/
Service Healthy
Logical Work Per Request
#
Req
Service
Healthy
Service Impact
ServiceImpact

35. New School Application DDoS Attack: Case Study

36. Making the Call More Expensive

37. Workflow for Identifying Application DDoS - Part 2
Identify the most latent service calls
Investigate if latent calls allow for manipulation
Determine API Gateway error conditions, thresholds, and library timeouts
Tune payload to fly under WAF/Rate Limiting
Test hypothesis
Scale your test using Cloudy Kraken (orchestrator) and Repulsive Grizzly (attack
framework)

38. Repulsive Grizzly
Skunkworks* application DDoS
framework
Written in Python3
Eventlet for high concurrency
Uses AWS SNS for logging analysis
Easily configurable

39. Repulsive Grizzly: Bypass WAF with Sessions
http://nerdprint.com/wp-content/uploads/2014/09/Dumle-Mountain-Cookies-Advertising2.png

40. Repulsive Grizzly: Single Node

41. Cloudy Kraken

42. Cloudy Kraken - What is it?

43. Cloudy Kraken - Key Features
Fresh global fleet of test instances each run
Lots of fresh global IP addresses
Easily push new attack code and configs
Coordinates the attack timing
Safety kill-switch

44. Cloudy Kraken Overview

45. Update Code
Push the configuration file
and attack scripts to S3.
Reset the DynamoDB state.
Build
Environment
Configure the VPCs in each
region
Start up
Attack
Nodes
Launch instances
Collect
Data
Wait for data to come
through SNS
Tear-Down
Tear down and reset the
environment in each region
Cloudy Kraken Workflow

46. We scaled up, time to run the test!
Tested against prod
Multi-region and
multi-agent
Conducted two 5
minute attacks
Monitored for success

47. Results of Test
80%

48. $1.71

49. So What Failed?
Expensive API calls could be invoked with non-member cookies
Expensive traffic resulted in many RPCs per request (1:7200)
WAF/Rate Limiter was unable to monitor middle tier RPCs

50. API
GATEWAY
Middle Tier Service
Middle Tier Service
Middle Tier Service
Middle Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Backend Tier Service
Edge Middle Backend
Case Study API DDoS
5. Queue continues to
fill up, takes longer to
return objects
6. Middle Tier
services
returning
5XX and 200
status codes
1. Agents cycling
through multiple
session cookies and IP
addresses
7. API gateway starts to
trigger breakers,
returning some 403’s,
503’s, 200’s, 504’s.
3. Objects not in
Cache, Cache Miss
4. About 15
seconds to
return huge
object store8. Automatic scaling of
services in progress,
but not fast enough
9. CPU Spiking in API
Gateway, Gateway
Timeouts occurring
2. Each request asking
for 7,200 expensive
calculations from
multiple backends

51. Demo

52. Microservice
Application DDoS:
Mitigations
Understand which
microservices impact
customer experience

53. Microservice
Application DDoS:
Mitigations
Limit batch and object
size in both client and
server

54. Microservice
Application DDoS:
Mitigations
Rate limiter (WAF)
should monitor
middle tier signals or
cost of request*

55. Microservice
Application DDoS:
Mitigations
Rate limiter (WAF)
should monitor
volume of cache
misses*

56. Microservice
Application DDoS:
Mitigations
Prioritize
authenticated traffic
over unauthenticated

57. Microservice
Application DDoS:
Mitigations
Configure reasonable
client library timeouts

58. Microservice
Application DDoS:
Mitigations
Trigger fallback
experiences when
cache or lookups fail

59. Future Work
Automated identification of potential vulnerable endpoints
Autotuning attack parameters during exercise based up on error
condition criteria
Testing common open source microservice
frameworks/libraries/gateways

60. Thanks!
https://github.com/netflix-skunkworks/repulsive-grizzly
https://github.com/netflix-skunkworks/cloudy-kraken
Questions? Come find us in the chill room after the session!
Scott Behrens Jeremy Heffner
@helloarbit

61. Extra Slides

62. Repulsive Grizzly: Payload and Header Files
Provide payloads in any format you want
Headers are provided as a JSON key/value hash
Use $$AUTH$$ placeholder to tell grizzly where to place tokens

63. Microservice Primer: Architecture
Scale
Service independence
Fault isolation
Eliminates stack debt
Distributed system complexity
Deployment complexity
Cascading service failures if
things aren’t set up right
GOOD BAD