Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2Dv9VKr. Nora Jones talks about designing microservices for enabling resiliency testing and the moving parts we need to consider when designing them from the get go, and along their lifetime. She shares tips and tricks on how to design microservices for resiliency tests, examples of poorly designed services, and how to ensure pertinent design decisions are in place on a continuous basis. Filmed at qconsf.com. Nora Jones is a Senior Chaos Engineer at Netflix. She is passionate about delivering high-quality software, improving processes, and promoting efficiency within architecture. Occasionally, she pokes holes in distributed systems to make them more resilient.

1. Designing Services for
Resilience Experiments:
Lessons from Netflix
Nora Jones, Senior Chaos Engineer
@nora_js

2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-microservices-resiliency

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

9. Designing Services for
Resilience Experiments:
Lessons from Netflix
Nora Jones, Senior Chaos Engineer
@nora_js

10. So, how can teams design services
for resilience testing?
● Failure Injection Enabled

11. So, how can teams design services
for resilience testing?
● Failure Injection Enabled
● RPC enabled

12. So, how can teams design services
for resilience testing?
● Failure Injection Enabled
● RPC enabled
● Fallback Paths
○ And ways to discover them

13. So, how can teams design services
for resilience testing?
● Failure Injection Enabled
● RPC enabled
● Fallback Paths
○ And ways to discover them
● Proper monitoring
○ Key business metrics to look for

14. So, how can teams design services
for resilience testing?
● Failure Injection Enabled
● RPC enabled
● Fallback Paths
○ And ways to discover them
● Proper monitoring
○ Key business metrics to look for
● Proper timeouts
○ And ways to discover them

15. Known Ways to Increase
Confidence in Resilience

16. Known Ways to Increase
Confidence in Resilience
● Unit Tests

18. Known Ways to Increase
Confidence in Resilience
● Integration Tests

20. New Ways to Increase Confidence
in Resilience
● Chaos Experiments

22. SPS: Key Business Metric

24. Chaos Engineering: Netflix’s ChAP
API Personalization
100%

25. Chaos Engineering: Netflix’s ChAP
APIGateway Personalization
API Control
1%
98%

26. Chaos Engineering: Netflix’s ChAP
APIGateway Personalization
API Control
1%
98%

27. Chaos Engineering: Netflix’s ChAP
APIGateway Personalization
API Control
API Exp
1%
1%
98%

28. Chaos Engineering: Netflix’s ChAP
APIGateway Personalization
API Control
API Exp
1%
1%
98%

29. Monitoring

30. Monitoring
SHORTED

31. 1. Have Failure Injection
Testing Enabled.

32. Sample Failure Injection
Library
https://github.com/norajones/FailureInjectionLibrary

39. Types of Chaos Failures

40. Types of Chaos Failures

41. Criteria&API

44. Automating Creation of Chaos
Experiments

45. 2. Have Good Monitoring in
Place for Configuration
Changes.

46. Have Good Monitoring in Place
● RPC Enabled

47. Have Good Monitoring in Place
● RPC Enabled
○ Associated Hystrix Commands

48. Have Good Monitoring in Place
● RPC Enabled
○ Associated Hystrix Commands
■ Associated Fallbacks

49. Have Good Monitoring in Place
● RPC Enabled
○ Associated Hystrix Commands
■ Associated Fallbacks
● Timeouts

50. Have Good Monitoring in Place
● RPC Enabled
○ Associated Hystrix Commands
■ Associated Fallbacks
● Timeouts
● Retries

51. Have Good Monitoring in Place
● RPC Enabled
○ Associated Hystrix Commands
■ Associated Fallbacks
● Timeouts
● Retries
● All in One Place!

53. ● Java library managing REST clients to/from
different services
● Fast failing/fallback capability
RPC/Ribbon

54. RPC/Ribbon Timeouts

55. RPC Timeouts
At what point does the service give up?

56. Retries
Immediately retrying a failure after an operation
is not usually a great idea.

57. Retries
Understand the logic between your timeouts and
your retries.

58. Circuit Breakers/Fallback Paths

59. Hystrix Commands/Fallback Paths
If your service is non-critical, ensure that there
are fallback paths in place.

60. Fallback Strategies
Static Content Cache Fallback
Service

61. Fallback Strategies
Know what your fallback strategy is and how to
get that information.

64. 3.Ensure Synergy
between Hystrix
Timeouts, RPC timeouts,
and retry logic.

67. ChAP’s Monocle

68. ChAP’s Monocle

69. ChAP’s Monocle

71. There isn’t always money in
microservices

72. Criticality Score

73. Criticality Score
RPS Stats Range bucket * number of retries * number of Hystrix Commands = Criticality
Score

74. Criticality Score
RPS Stats Range bucket * number of retries * number of Hystrix Commands = Criticality
Score

75. Criticality Score
RPS Stats Range bucket * number of retries * number of Hystrix Commands = Criticality
Score

76. Criticality Score
RPS Stats Range bucket * number of retries * number of Hystrix Commands = Criticality
Score

77. Chaos Success Stories

78. “We ran a chaos experiment which
verifies that our fallback path works
and it successfully caught a issue in
the fallback path and the issue was
resolved before it resulted in any
availability incident!”

79. “While [failing calls] we discovered an increase in
license requests for the experiment cluster even
though fallbacks were all successful...

80. “While [failing calls] we discovered an increase in
license requests for the experiment cluster even
though fallbacks were all successful. ...This likely
means that whoever was consuming the fallback
was retrying the call, causing an increase in
license requests.”

81. Don’t lose sight of your
company’s customers.

82. Takeaways
● Designing for resiliency testability is a shared
responsibility.
● Configuration changes can cause outages.
● Have explicit monitoring in place on
antipatterns in configuration changes.
@nora_js

83. Questions?
@nora_js

84. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-microservices-resiliency