Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1SUVxKa. Roy Rapoport demonstrates the power of alignment (or lack thereof) using real-world examples from previous and current employer, with specific emphasis on his experience introducing Python to production use within Netflix, the organizational structures he interacted with through that process, and the way they tie into Netflix's formal culture. Filmed at qconsf.com. Roy Rapoport manages the Insight Engineering group at Netflix, responsible for building Netflix's Operational Insight platforms, including cloud telemetry, alerting, and real-time analytics". Roy has been in tech for about 20 years with positions in IT engineering and operations, software development, and software quality engineering, but his passion remains with operations and automation.

1. Culture and the Games
People Play
Roy Rapoport
rsr@netflix.com @royrapoport
November 18, 2015

2. InfoQ.com: News & Community Site
• 750,000 unique visitors/month
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• News 15-20 / week
• Articles 3-4 / week
• Presentations (videos) 12-15 / week
• Interviews 2-3 / week
• Books 1 / month
Watch the video with slide
synchronization on InfoQ.com!
http://www.infoq.com/presentations
/culture-games

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. SHALL WE PLAY A GAME?

5. What We Want
(And How We Get It)
Outcomes
ActionsDecisions
What
environment
says
What
environment
does

6. What We Want
(And How We Get It)
Outcomes
ActionsDecisions
What
environment
says
What
environment
does

7. What We Want
(And How We Get It)
Decisions
What
environment
says
What
environment
does

8. What We Want
(And How We Don’t Get It)

9. What We Want
(And How We Don’t Get It)

10. Test #1
Attendance Award

11. A Word About Netflix …
• Clear Priorities
1. Innovation
2. Availability
3. Cost
• Hire smart, experienced, people
• Get out of the way
• Anti-process bias
Culture

12. In Practice …

13. Dozens of SSL Certificates
Decentralized
Kept Expiring
Hilarity would ensue
Amazon Resources
“No Preset Limit”
You know when you hit it
Hilarity would ensue
The Before Time

14. Well-developed Developer Ecosystem
Service Discovery
DB Client
Credentials Management
Memory Object Cache
Server Infrastructure
Telemetry
You wanted that for Java, right?
The Before Time

15. Just moved from IT/Ops
Formally tasked with SSL cert
issue as quarterly goal
Limits issue “tacked” on
“Effective” in Python
Didn’t know Java
Presenter Selfie
The Before Time

16. Ported necessary libraries to Python
Boss was dubious. Really dubious.
Ran into security problem
Introducing Jay
No Problem!

17. Democratized Innovation
What would you say you do around here?
Story Time: Shark Tank

18. Conceived by Reliability Engineer
Remote Telemetry Network
Teams involved:
Reliability Engineering
Insight Engineering
Performance Engineering
Some others …
Surprise!
“Proof-of-concept work
on Ansible
configuration
management for Gulo
and Hammerhead.”

19. Avoid Zero-Sum Games
Stack ranking
Fixed bonus / raise pools
No ranking/quantifying
Reviews != raises
Decentralize collaboration
Align goals
I want:
Collaboration and Selflessness

20. Act In Netflix’s Best Interests

21. Test #2
Early Birds, Late Worms

22. I want:
Decentralized Innovation
Autonomy and Independence
Bets and Risk Tolerance:
a Story of Failures

23. Losing Bets
18 month report card (estimated)
Security Monkey Success
Howler Monkey Success
Exploit Monkey Failure
Python Success
Service SLA Dashboard Failure
Alert Outsourcing Success
Alert Response Analytics Failure
Alert Gateway Success
Alerting GUI Success
Latency Monkey Adoption Fizzle
Stateful Alerting Failure
Open Application Alerting Failure
50% Failure Rate

24. I want:
Decentralized Innovation
Autonomy and Independence
An Engineering Manager Walks Into
an Override Bar …

25. The Override Bar
Asgard: Full-fledged cloud
orchestration
GUI-driven
Region-and-account
specific

26. The Override Bar
Four regions
Eight accounts
Hundreds of clusters

27. The Override Bar
A Bold Proposal
Totally duplicates functionality
Customized fit
Failed the override bar:
Am I sure this is the wrong thing?
If I’m right, will this be very expensive for us?

28. The Override Bar
Accomplished predicted results
Massively simplified operational processes
Improved resiliency and velocity
Unpredictable results
Used by other teams
Inspiration
Will retire

29. I want:
Decentralized Innovation
Autonomy and Independence
Spheres of Autonomy:
Staying DRI

30. Yury’s SoAYury’s SoAYury’s SoAJosh’s SoA
Roy’s Sphere of
autonomy
Concentric Spheres of Autonomy
Fang’s
Sphere of
autonomy

31. Reed’s
Sphere of
Autonomy
Neil’s
Sphere of
Autonomy
Yury’s
Sphere of
Autonomy
Josh’s
Sphere of
Autonomy
Roy’s
Sphere of
autonomy
Spheres of Autonomy:
A New Model
Fang’s
sphere of
autonomy

32. Spheres of Autonomy:
A New Model
Set context. Not control.

33. Spheres of Autonomy:
A New Model
Keeping Peers DRI

34. Test #3
Lucy and the Ball

35. Literally* no downsides!
* For very non-literal definitions of the word “literally”
Predictability tradeoffs
Locality optimization
Duplication
Duplication

36. Agility vs Predictability
Neither is bad
Probably need some of both
Do you know how much you want?
Do you have it?
Agility Predictability

37. Agility vs Predictability
Optimize for agility
Constrain predictability
Some things are important to predict
Public KPIs
Big product plans
Fewer are important than you may think
Agility Predictability

38. If a Thing can be built anywhere
Not always in the best place
Extra work
Locality Optimization
Or lack thereof

39. Locality Optimization
Or lack thereof
Story Time: Scryer

40. Scryer: Start State
Real-Time Telemetry System
2 weeks of data

41. Scryer: Goal
Real-Time Telemetry System
2 weeks of data
Predictor
Signal Predictions
Today
Product
Value-add
Process

42. Scryer Architecture, v1
Real-Time Telemetry System
2 weeks of data
Telemetry
Extractor
Telemetry Persistence
4 weeks of data
Predictor
Signal Predictions
Today
Product
Value-add
Process
Waste of
Time
Pain the
[REDACTED]

43. The Thing Is …
Real-Time Telemetry System
2 weeks of data
Cloud Storage
All telemetry, forever
ETL

44. Scryer Architecture, v2
Real-Time Telemetry System
2 weeks of data
Predicted Signal Today
Predictor
Product
Value-add
Process
Cloud Storage
All telemetry, forever
ETL

45. Test #4
Making Friends $100 At a Time

47. "I only want to ride the wind
and walk the waves, slay the
big whales of the Eastern sea,
clean up frontiers, and save
the people from drowning.
Why should I imitate others,
bow my head, stoop over and
be a slave?” - Lady Triệu

48. rsr@netflix.com
@royrapoport
Attributions:
https://www.flickr.com/photos/cseeman/
http://www.flickr.com/photos/watchsmart
http://www.flickr.com/photos/yaketyyakyak/
https://www.flickr.com/photos/gfreeman23/
https://www.flickr.com/photos/dotcode
https://www.flickr.com/photos/tlindfors
And the Rands Leadership Slack

49. Watch the video with slide synchronization on
InfoQ.com!
http://www.infoq.com/presentations/culture-
games