Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2vwBTO2. Josh Evans uses Netflix as a case study to illustrate how specific strategies, framed as technical analogs, have been employed to maximize engineering agility, velocity, and impact. These strategies and solutions are meant to provide a blueprint for organizational success. Filmed at qconnewyork.com. Josh Evans is a 17-year Netflix veteran with experience spanning e-commerce, video streaming services, infrastructure, tools, testing, support, and operations. He most recently led Operations Engineering at Netflix and now is currently taking time off, seeking new opportunities in the new future.

1. Josh Evans – (Former Netflix) Engineering Leader at Large
June 26, 2017
Refactoring Organizations
A Netflix Study

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-refactor-organizations

3. Presented at QCon New York
www.qconnewyork.com
Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide

4. 2009

5. Devices
…

6. Queue Reader UX

7. DVD Foundation

8. Customer Device Netflix Data Center
NCCP
Electronic Delivery
LoadBalancer
Netflix App
Security
Activation
Playback
Platform (NRDP)
UI
XML/RPC
Ticket-based security
Custom responses
DB
HTTP/S
DVD Legacy
DVD Legacy

10. Netflix API

11. Let 1000 flowers bloom!

12. Netflix Data Center
API
Netflix API
LoadBalancer
REST API
JSON schema
HTTP response codes
Oauth
Content
Metadata
Application
HTTP/S

13. Customer Device
Netflix Data Center
API
Proposal
LB
Netflix App
Security
Activation
Playback
Platform (NRDP)
UI
Content
Metadata
NCCP
ED
LB

14. Pros
Separation of concerns
Bandwidth
Increased innovation
Cons
API reliability
Heterogeneous architecture
Lack of domain knowledge
Assessment

15. Neil Hunt, CPO

16. What was the reaction?

17. Concern
Anger

18. Tribalism

19. Conway’s Law

20. If you have four teams working on a compiler you will end
up with a four pass compiler

21. Today’s Premise
Conway’s Law describes dysfunction
We must embrace architecture before organization
Technical analogs drive better organizational solutions

22. Selfless Leadership
Company
Team
You
In that order!

23. Introductions
Framework
Scaling Teams
IQ v EQ
Conway’s Revenge
Today’s Program

24. Introductions
Framework
Scaling Teams
IQ v EQ
Conway’s Revenge
Today’s Program

25. 1999 – 2009
Ecommerce (DVD  Streaming)
2009 – 2013
Streaming Infrastructure
2013 - 2016
Operations Engineering
2017
Time off, exploring options
Josh Evans – Engineering Leader at Large
@ops_engineering
@

26. Global leader in subscription internet TV
Growing slate of original content
100 million members
190 countries, 10s of languages
1000s of device types
Microservices on AWS
Unique company culture

27. Introductions
Framework
Scaling Teams
IQ v EQ
Conway’s Revenge
Today’s Program

28. Why do we refactor?

29. Functionality
Engineering velocity
Functional and operational quality
As we scale!
We refactor to improve or sustain

30. The ability to enhance a system by adding new functionality at
minimal effort
Functional Scalability
The ease with which a system or component can be modified,
added, or removed, to accommodate changing load
Load Scalability

31. The ability for an organization to easily add people and domain
responsibilities in response to increased work and complexity
The ease with which an organization or team can adapt to shifts
in business strategy
Organizational Scalability

32. Common tasks are difficult
Strategic efforts are impractical or impossible
When do we refactor?

33. How do we refactor?

34. Technical Patterns
Object-oriented design
Micro-service architecture
Systems engineering

35. Example: Organizational Polymorphism

36. With the right people
Instead of a culture process adherence
We have a culture of creativity and self discipline,
freedom and responsibility
Netflix Culture

37. You build it
You run it

38. Introductions
Framework
Scaling Teams
IQ v EQ
Conway’s Revenge
Today’s Program

39. 2009

40. Devices in Production
…

41. Key Platforms in Progress

42. Anthony Park

43. Surprise!
John Funge

44. Big Picture
Device Ubiquity
Product Innovation
Cloud Migration
Internationalization
Service Reliability

45. How many engineers?
6

47. What would you do?

48. Prioritize & Queue
…
…
Task Queue
Completed Tasks
Thread Pool
… …

49. Prioritize
Service availability
Game consoles
Downloadable apps
CE expansion
Mobile
…
Queue
Audio & subtitles
International support
New codecs
…
Prioritize & Queue

50. Scale Up
Work profile
Roles & throughput
Team structure

51. Manager
Engineer Engineer Engineer Engineer Engineer Test/Ops
Monolithic Team
One leader
Undifferentiated roles
Ad hoc responsibilities

52. Monolithic Decomposition
Distinct modules & services
Workload partitioning
Dependency awareness
Loose coupling

53. Server
NRDP Features
Protocols
Security
Bootstrap
Key Platforms
Device integration
Device launch
Streaming Infrastructure
Device & partner-oriented load balancing

54. On Call Overload

55. Vicious Cycle
Philip was great at
Development
Test infrastructure
Project management
Troubleshooting
Philip Fisher-Ogden

56. You build it
You run it

57. Risks
Burnout
Slow progress on key initiatives
Philip Fisher-Ogden

58. Thread Starvation
… …
Shared
exclusive
resource
High priority/frequency
Other - blocked
Tasks

59. Context Switching
Process 1 Process 2
OS
Interrupt or system call
Save state - pcb1
..
Get state – pcb2
Interrupt or system call
Save state – pcb2
Get state – pcb1
..
Executing
Executing
Idle Executing
Idle
Idle

60. Thread Pool Isolation
Partition pools & locks
Distribute problematic workloads
… …
… …
… …
…
…

61. Organizational Solution
Deepen troubleshooting skills
Distribute escalations
Engineer operations

62. Key Platforms
Device integration
Device launch
Server
NRDP
Protocols
Security
Bootstrap
Insight/Tools
Delivery
Dashboards
Performance
Operational tools
Consolidating Operations Engineering

63. Cloud Migration

64. Rapid iteration v. systematic, long-cycle execution
Cloud v. Product

65. S S S S. . .
DB DB DB DB. . .
. . . . . .
Member Traffic Batch Processes
Heterogeneous
Workloads

66. Batch
S S S S. . .
DB DB DB DB. . .
. . . . . .
Member Path
Member Path
Member Path
Batch
Batch
Interference

67. Batch
S S S S. . .
DB DB DB DB. . .
. . . . . .
Member Path
Member Path
Member Path
Batch
Batch
Interference

68. Batch
S S S S. . .
DB DB DB DB. . .
. . . . . .
Member Path
Member Path
Member Path
Batch
Batch
Interference

69. . . .
Batch
S S S S. . .
DB DB DB DB. . .
. . .
Member Path
Member Path
Member Path
Batch
Batch
Interference
X

70. Batch
S S S S. . .
DB DB DB DB. . .
. . .
Member Path
Member Path
Member Path
Batch
Batch
Partitioning
Online Offline
. . .

71. Silverlight Migration

72. Partitioning & Domain Portability
Streaming Infrastructure Platform Engineering
Engineer as a Library
Ranjit Ranjit

73. Partitioning & Domain Portability
Platform Engineering
Engineer as a Library
Streaming Infrastructure

74. Systems
Cloud migration
Key Platforms
Device integration
Device launch
Server
NRDP
Protocols
Security
Bootstrap
Insight/Tools
Delivery
Dashboards
Performance
Ops tools
Streaming Infrastructure

75. Staffing

76. 6  24

77. Bottleneck

78. Systems
Cloud migration
Viewing history
Viewing sessions
Key Platforms
Device integration
Device launch
Server
NRDP
Protocols
Security
Bootstrap
Insight/Tools
Delivery
Dashboards
Performance
Ops tools
Cloning & Parallel Processing

79. By 2012
cloud migration
Canada, Latin America, UK
massive device expansion
major product improvements
Netflix CDN

80. Introductions
Framework
Scaling Teams
IQ v EQ
Conway’s Revenge
Today’s Program

81. IQ
Task-oriented
Logical
Literal
Detached
Autocratic
EQ
Feeling-oriented
Emotional
Social
Empathetic
Democratic
Bimodal Thinking

82. IQ
Design
Evaluation
Implementation
EQ
Inception
Socialization
Overcoming Tribalism

83. Flawed Inception

84. Introductions
Framework
Scaling Teams
IQ v EQ
Conway’s Revenge
Today’s Program

85. 2012

86. Customer Device
Netflix Data Center
API
This…
LB
Netflix App
Security
Activation
Playback
Platform (NRDP)
UI
Content
Metadata
NCCP
ED
LB

87. ELB
NCCP
API
…has become this
Zuul

88. ELB
…and this

89. Growing complexity
Duplication of effort
Engineering tax

90. Raising the Stakes
Playback start in 500ms
More UI/Playback scenarios
Faster rate of innovation
Better service reliability

91. Common tasks are difficult
Strategic efforts are impractical or impossible
When do we refactor?

92. If you have four teams working on a compiler you will end
up with a four pass compiler
Conway’s Revenge!
We had two teams and a two-service edge architecture

93. Mature API team
Robust API platform
Strong operational focus
Trust & respect
A Better Foundation
Daniel Jacobson

94. Josh: what’s the right architectural solution?
Peter: do you care about the organizational implications?
Moment of Truth

95. Selfless Leadership

96. Josh: what’s the right architectural solution?
Peter: do you care about the organizational implications?
Moment of Truth
Josh: no – we’ll figure that out later

98. ELB
NCCP
API
Before
Zuul

99. After
Integrated architecture
Distributed functionality
Shared services
Common practices

100. Edge Services
Zuul API Server
Playback
Services
Features Security Data Systems
Platform Insight/Tools
Edge Services
Shared services
Organized around microservices, functionality, shared services

101. Takeaways
Put architecture first
Leverage technical analogs
Know when to use IQ v EQ
Be selfless

102. www.linkedin.com/in/jevansnflx
Where to find me

103. ?
Refactoring Organizations

104. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/netflix-
refactor-organizations