Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2hkPH7v. Josh Evans talks about the chaotic and vibrant world of microservices at Netflix. He starts with the basics, - the anatomy of a microservice, the challenges around distributed systems, and the benefits. Then he builds on that foundation exploring the cultural, architectural, and operational methods that lead to microservice mastery. Filmed at qconsf.com. Josh Evans is Director of Operations Engineering at Netflix, with experience in e-commerce, tools, testing, and operations. For the past three years he has led an organization that creates, integrates, and evangelizes proven technical solutions and practices like continuous delivery, real-time ​operational insight, and chaos engineering to achieve operational excellence at scale.

1. Josh Evans – Engineering Leader
November 8, 2016
Mastering Chaos
A Netflix Guide to Microservices

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-chaos-microservices

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. Illness in the Family

5. Myelin Sheathe
Autoimmune disorder
Externally trigger
Treatable
Guillain-Barré Syndrome

6. Breathing is a miraculous act of
bravery

7. ELB
and so is taking traffic

8. Introductions
Microservice Basics
Challenges & Solutions
Organization & Architecture
Our Talk Today

9. Introductions
Microservice Basics
Challenges & Solutions
Organization & Architecture
Our Talk Today

10. 1999 – 2009
Engineer & Engineering Manager
Ecommerce (DVD à Streaming)
2009 – 2013
Director of Engineering - Playback Services
2013 – 2016
Director of Operations Engineering
Josh Evans

11. Taking time off
Spending time with family
Thinking about what’s next
Today

12. Leader in subscription internet tv service
Hollywood, indy, local
Growing slate of original content
86 million members
~190 countries, 10s of languages
1000s of device types
Microservices on AWS

13. Introductions
Microservice Basics
Challenges & Solutions
Organization & Architecture
Our Talk Today

14. Netflix DVD Data Center - 2000
Linux Host
What microservices are not
Apache
Tomcat
Javaweb
STORE
Load Balancer
BILLING
HTTP
JDBC
DB Link
HTTP/S
Monolithic code base
Monolithic database
Tightly coupled architecture

15. What is a microservice?

16. …the microservice architectural style is an
approach to developing a single application as a
suite of small services, each running in its own
process and communicating with lightweight
mechanisms, often an HTTP resource API.
- Martin Fowler

17. Separation of concerns
Modularity, encapsulation
Scalability
Horizontally scaling
Workload partitioning
Virtualization & elasticity
Automated operations
On demand provisioning
An Evolutionary Response

18. Organ Systems
Each organ has a purpose
Organs form systems
Systems form an organism

19. Edge
ELB
Zuul
NCCP
API
Middle Tier & Platform
Product
• Bucket testing
• Subscriber
• Recommendations
Platform
• Routing
• Configuration
• Crypto
Persistence
• Cache
• Database

20. Client Application
Client Library
EVCache Client Service Client
S S S S. . .
DB DB DB DB. . .
. . .
Microservices are an abstraction
. . .
Microservice

21. Introductions
Microservice Basics
Challenges & Solutions
Organization & Architecture
Our Talk Today

22. Dependency
Scale
Variance
Change
Challenges & Solutions

23. Dependency
Scale
Variance
Change
Challenges & Solutions

24. Intra-service requests
Client libraries
Data Persistence
Infrastructure
Use Cases

25. Intra-service Requests

26. Crossing the Chasm

27. Linux Host
Linux Host
Linux Host
Linux Host
Crossing the Chasm
Linux Host
Apache Tomcat
Linux Host
Apache Tomcat
Network latency, congestion, failure
Logical or scaling failure
Service A Service B

28. Cascading Failure

30. How do you know if it works?

31. Inoculation

32. Device Service B
Service C
Internet EdgeZuul
Service A
ELB
FITSynthetic transactions
Override by device or account
% of live traffic up to 100%
Fault Injection Testing (FIT)

33. Device Service B
Service C
Internet EdgeZuul
Service A
ELB
FIT
Fault Injection Testing (FIT)
Enforced throughout the call path

34. ELB
API

35. How do we constrain testing scope?

36. API Gateway
App 1
App 2
App 4
App 5
App 6
App 3
App 7
App 8
99.99
99.99
99.99
99.99
99.99
99.99
99.99
99.99
Proxy
99.99 99.99
Combinatorial Math
99.9910 = 99.9

37. Critical Microservices

38. Client Libraries

39. • Many clients
• Common business logic
• Common access patterns

40. Return of the Monolith

41. Heap consumption
Logical defects
Transitive dependencies
Parasitic Infestation

42. Client Application
Client Library
EVCache Client Service Client
S S S S. . .
DB DB DB DB. . .
. . .
Simple Logic, Common Patterns
. . .

43. Persistence

44. In the presence of a network partition, you must choose
between consistency and availability
CAP Theorem
DB
DB
DB
Network B
Network C
Network D
Service
Network A
X

45. Zone A
Zone B
Zone C
Zone B
Zone C
Client
Zone A
Local Quorum
(Typical)
100ms
Eventual Consistency

46. Infrastructure

47. December 24th, 2012

48. US-East-1
Canada
No place to go
US
Latin America

49. US-East-1US-West-2 EU-West-1

50. #NetflixEverywhere Global Architecture
QCon London, 2016
https://www.infoq.com/presentations/netflix-failure-multiple-regions

51. Dependency
Scale
Variance
Change
Challenges & Solutions

52. Stateless services
Stateful services
Hybrid services
Use Cases

53. Stateless Services

54. Not a cache or a database
Frequently accessed metadata
No instance affinity
Loss a node is a non-event
What is a stateless service?

56. Minimum size
Desired capacity
Maximum size
Scale out as needed
S3AMI retrieved on demand
Compute efficiency
Node failure
Traffic spikes
Performance bugs
Auto Scaling Groups

57. Cluster A Cluster D
Edge Cluster
Cluster B
Cluster C
Surviving Instance Failure

58. Stateful Services

59. Databases & caches
Custom apps which hold large amounts of data
Loss of a node is a notable event
What is a stateful service?

60. Dedicated Shards – An Antipattern
Squid 1 Squid 2 Squid 3
Client Application
Subscriber Client Library
Cache Client Service Client
S S S S. . .
DB DB DB DB. . .
Squid n
HA Proxy
Set 1 Set 2 Set 3 Set n
X

61. Redundancy is fundamental

62. Zone A Zone B Zone C
. . .. . .. . .
EVCache Writes
Client Application
Client Library
EVCache Client
Client Application
Client Library
EVCache Client
Client Application
Client Library
EVCache Client

63. Zone A
Client Application
Client Library
EVCache Client
Zone B
Client Application
Client Library
EVCache Client
Zone C
Client Application
Client Library
EVCache Client
. . .. . .. . .
EVCache Reads

64. Hybrid Services

65. Client Application
Client Library
EVCache Client Service Client
S S S S. . .
DB DB DB DB. . .
. . .
Hybrid Microservice
. . .

66. It’s easy to take EVCache for granted
30 million requests/sec
2 trillion requests per day globally
Hundreds of billions of objects
Tens of thousands of memcached instances
Milliseconds of latency per request

67. Batch
S S S S. . .
DB DB DB DB. . .
. . . . . .
Member Path
Member Path
Member Path
Batch
Batch
Called by many services
Online & offline clients
Called many times / request
800k – 1M RPS
Fallback to service/db
Excessive Load

68. Batch
S S S S. . .
DB DB DB DB. . .
. . . . . .
Member Path
Member Path
Member Path
Batch
Batch
Excessive Load
X X

69. Batch
S S S S. . .
DB DB DB DB. . .
. . . . . .
Member Path
Member Path
Member Path
Batch
Batch
Workload partitioning
Request-level caching
Secure token fallback
Chaos under load
Solutions
Online Offline

70. Dependency
Scale
Variance
Change
Challenges & Solutions

71. Operational drift
Polyglot & containers
Use Cases

72. Operational Drift
(Unintentional Variance)

73. Over time
Alert thresholds
Timeouts, retries, fallbacks
Throughput (RPS)
Across microservices
Reliability best practices
Operational Drift

74. Autonomic Nervous
System
You don’t have to think about
digestion or breathing

75. Incident
Resolution
Review
Remediation
Analysis
Best Practice
Automation
Adoption
Continuous Learning & Automation

76. Alerts
Apache & Tomcat
Automated canary analysis
Autoscaling
Chaos
Consistent naming
ELB config
Healthcheck
Immutable machine images
Squeeze testing
Staged, red/black deployments
Timeouts, retries, fallbacks
Production Ready

77. Polyglot & Containers
(Intentional Variance)

78. The Paved Road
Stash
Nebula/Gradle
BaseAMI/Ubuntu
Jenkins
Spinnaker
Runtime Platform

80. In the Critical Path

81. In the Critical Path

82. Productivity tooling
Insight & triage capabilities
Base image fragmentation
Node management
Library/platform duplication
Learning curve - production expertise
Cost of Variance

83. Raise awareness of costs
Constrain centralized support
Prioritize by impact
Seek reusable solutions
Strategic Stance

84. Dependency
Scale
Variance
Change
Challenges & Solutions

87. How do we achieve velocity with confidence?

88. Global Cloud Management & Delivery

89. Integrated, Automated Practices
Conformity checks
Red/black pipelines
Automated canaries
Staged deployments
Squeeze tests

90. Alerts
Apache & Tomcat
Automated canary analysis
Autoscaling
Chaos
Consistent naming
ELB config
Healthcheck
Immutable machine images
Squeeze testing
Staged, red/black deployments
Timeouts, retries, fallbacks
Production Ready

91. https://www.youtube.com/watch?v=IkPb15FfuQU

92. Introductions
Microservice Basics
Challenges & Solutions
Organization & Architecture
Our Talk Today

93. Customer Device Netflix Data Center - 2009
NCCP
Electronic Delivery - NRDP 1.x
Load Balancer
Netflix App
Security
Activation
Playback
Platform (NRDP)
UI
Collaborative design
XML payloads
Custom responses
Versioned firmware releases
Long cycles
Simple UI – “Queue Reader”
ED

94. Netflix API - let a 1000 flowers bloom!

95. Netflix Data Center - 2009
API
Netflix API – from public to private
Load Balancer
General REST API
JSON schema
HTTP response codes
Oauth security model
Content Metadata
Content
Metadata
Application

96. Customer Device
Netflix Data Center – 2010
API
Hybrid Architecture
LB
Netflix App
Security
Activation
Playback
Platform (NRDP)
UI
Content
Metadata
NCCP
ED
LB
Distinct
• Services
• Protocols
• Schemas
• Security

97. Josh: what is the right long term architecture?
Peter: do you care about the organizational
implications?

98. Conway’s Law
Organizations which design systems are constrained to
produce designs which are copies of the
communication structures of these organizations.
Any piece of software reflects the organizational
structure that produced it.

99. Conway’s Law
If you have four teams working on a compiler you will
end up with a four pass compiler

100. NCCP
API

101. Blade Runner

102. Outcomes
Productivity & new capabilities
Refactored organization
Lessons
Solutions first, team second
Reconfigure teams to best support your architecture
Outcomes & Lessons

103. Introductions
Microservice Basics
Challenges & Solutions
Organization & Architecture
Recap
Our Talk Today

104. Microservice architectures are
complex and organic
Health depends on discipline and
chaos

105. Dependency
Circuit breakers, fallbacks, chaos
Simple clients
Eventual consistency
Multi-region failover
Scale
Auto-scaling
Redundancy – avoid SPoF
Partitioned workloads
Failure-driven design
Chaos under load
Variance
Engineered operations
Understood cost of variance
Prioritized support by impact
Change
Automated delivery
Integrated practices
Organization & Architecture
Solutions first, team second

106. netflix.github.io

107. techblog.netflix.com

108. Questions?

109. Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-chaos-microservices