Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2mArA9C. Kolton Andrus and Peter Alvaro present how a “big idea” -- lineage-driven fault injection -- evolved from a theoretical model into an automated failure testing service at Netflix. They describe the challenges (expected as well as unexpected, technical as well as ideological) that arose, and how they overcame them. Filmed at qconsf.com. Kolton Andrus is the founder of Gremlin Inc. He is passionate about building resilient systems, primarily as it lets him break things for fun and profit. Peter Alvaro is an Assistant Professor of Computer Science at the University of California Santa Cruz. He is the creator of the Dedalus language and co-creator of the Bloom language.

1. Monkeys in Lab Coats
Automating Failure Testing Research at

2. InfoQ.com: News & Community Site
• Over 1,000,000 software developers, architects and CTOs read the site world-
wide every month
• 250,000 senior developers subscribe to our weekly newsletter
• Published in 4 languages (English, Chinese, Japanese and Brazilian
Portuguese)
• Post content from our QCon conferences
• 2 dedicated podcast channels: The InfoQ Podcast, with a focus on
Architecture and The Engineering Culture Podcast, with a focus on building
• 96 deep dives on innovative topics packed as downloadable emags and
minibooks
• Over 40 new content items per week
Watch the video with slide
synchronization on InfoQ.com!
https://www.infoq.com/presentations/
netflix-testing-research

3. Purpose of QCon
- to empower software development by facilitating the spread of
knowledge and innovation
Strategy
- practitioner-driven conference designed for YOU: influencers of
change and innovation in your teams
- speakers and topics driving the evolution and innovation
- connecting and catalyzing the influencers and innovators
Highlights
- attended by more than 12,000 delegates since 2007
- held in 9 cities worldwide
Presented at QCon San Francisco
www.qconsf.com

4. The whole is greater than the sum of its parts.
- Aristotle
[Metaphysics]

5. The Professor vs The Practitioner
Peter Alvaro
Ex-Berkeley, Ex-Industry
Assistant Prof @ Santa Cruz
Misses the calm of PhD life
Likes prototyping stuff
Kolton Andrus
Ex-Netflix, Ex-Amazon
‘Chaos’ Engineer
Misses his actual pager
Likes breaking stuff

6. Measures of Success
Academic
H-Index
Grant warchest
Department ranking
Industry
Availability (i.e. 99.99% uptime)
Number of Incidents
Reduce Operational Burden

7. An Unlikely Team?

9. but ... it’s manual
Works Great!

10. Surely there is a better way ...

12. Free lunch?

13. The End?
(Academia + Industry)

14. Let’s build it
“Can we, pretty please?”

15. Freedom and Responsibility
Core Value

16. Responsibility
Academic Industry
Prove that it works
Show that it scales
Find real bugs

17. The Big Idea Lineage Driven
Fault Injection

18. What could possibly go wrong?
Consider computation
involving 100 services
Search Space:
2100
executions

19. “Depth” of bugs
Single Faults Search Space:
100 executions

20. “Depth” of bugs
Combination of 4 faults Search Space:
3M executions

21. “Depth” of bugs
Combination of 7 faults Search Space:
16B executions

22. Random Search
Search Space:
2100
executions

23. Engineer-guided Search
Search Space:
???

24. Fault-tolerance “is just” redundancy

25. How do we find the redundancy?
Could a bad ‘thing’ ever happen?
Why did a good ‘thing’ happen?

26. Lineage-driven fault injection
Why did a good thing happen?
Consider its lineage.
The write
is stable
Stored on
RepA
Stored on
RepB
Bcast1 Bcast2
Client Client

27. Lineage-driven fault injection
Why did a good thing happen?
Consider its lineage.
What could have gone wrong?
Faults are cuts in the lineage graph.
Is there a cut that breaks all supports?
The write
is stable
Stored on
RepA
Stored on
RepB
Bcast1 Bcast2
Client Client

28. Lineage-driven fault injection
Why did a good thing happen?
Consider its lineage.
What could have gone wrong?
Faults are cuts in the lineage graph.
Is there a cut that breaks all supports?
The write
is stable
Stored on
RepA
Stored on
RepB
Bcast1 Bcast2
Client Client

29. What would have to go wrong?
(RepA OR Bcast1)
The write
is stable
Stored on
RepA
Stored on
RepB
Bcast2
Client Client
Bcast1

30. What would have to go wrong?
(RepA OR Bcast1)
AND (RepA OR Bcast2)
The write
is stable
Stored on
RepA
Stored on
RepB
Bcast1 Bcast2
Client Client

31. What would have to go wrong?
(RepA OR Bcast1)
AND (RepA OR Bcast2)
AND (RepB OR Bcast2)
The write
is stable
Stored on
RepA
Stored on
RepB
Bcast1
Client Client
Bcast2

32. What would have to go wrong?
(RepA OR Bcast1)
AND (RepA OR Bcast2)
AND (RepB OR Bcast2)
AND (RepB OR Bcast1)
The write
is stable
Stored on
RepA
Stored on
RepB
Bcast1 Bcast2
Client Client

33. Lineage-driven fault injection The write
is stable
Stored on
RepA
Stored on
RepB
Bcast1 Bcast2
Client Client
Hypothesis: {Bcast1, Bcast2}

34. Search Space Reduction
Each Experiment finds
a bug, OR
Reduces the
Search space

35. The prototype system “Molly”
Recipe:
1. Start with a successful
outcome. Work backwards.
2. Ask why it happened: Lineage
3. Convert lineage to a boolean
formula and solve
4. Lather, rinse, repeat
2. Lineage 3. CNF
Fail1. Success
Why?
Encode
Solve
4. REPEAT

36. The Big Idea Meets Production

37. 1. Start with a successful outcome
2.
Lineage
3.
CNF
Fail1. Success
Why?
Encode
Solve
4. REPEAT

38. What is success?

40. “Start with the customer and work
backwards”
Leadership Principle

43. Lesson 1
Work backwards from what you know

44. 2. Ask why it happened
2.
Lineage
3.
CNF
Fail1. Success
Why?
Encode
Solve
4. REPEAT

45. Request Tracing

47. Request Tracing

48. Alternate Execution

49. Evolution over time

50. Redundancy through History

51. Lesson 2
Meet in the middle

52. 3. Solve
2.
Lineage
3.
CNF
Fail1. Success
Why?
Encode
Solve
4. REPEAT

53. A “small” matter of code

54. 4. Lather, Rinse, Repeat
2.
Lineage
3.
CNF
Fail1. Success
Why?
Encode
Solve
4. REPEAT

55. Turn the crank, right?

56. Idempotence

59. Bins and Balls
Request
Class 1
Class 2
Class 3
Class n
[...]
r’ r

60. Class n
Predicting Request Graphs
Request

61. Class n
Predicting Request Graphs
Request
Some function f:
Requests → Classes

62. F( ) =
Class n
Request
Predicting Request Graphs

64. Solve the Machine Learning problem?
or the Failure Testing one?

65. Simplest thing that will work?

66. ["bookmarks”, “recent”]
["playlist", 0, “name”]
["ratings"]
Falcor Path Mapping
=>
“bookmarks,playlist,ratings”

67. Lesson 3
Adapt the theory to the reality

68. Many moons passed...

69. Does it work? YES!

70. Case study: “Netflix AppBoot”
Services ~100
Search space (executions) 2100
(1,000,000,000,000,000,000,000,000,000,000)
Experiments performed 200
Critical bugs found 11

71. Future Work
Richer device metrics
Request class creation
Better experiment selection
Search prioritization
Richer lineage collection
Exploring temporal
interleavings

81. Lessons
Work backwards from what you know
Meet in the middle
Adapt the theory to the reality

82. Academia + Industry

83. Academia + Industry
Academia Industry

84. Thank You!
Peter Alvaro
@palvaro
palvaro@ucsc.edu
Kolton Andrus
@KoltonAndrus
kolton@gremlininc.com

85. References
● Netflix Blog on ‘Automated Failure Testing’
http://techblog.netflix.com/2016/01/automated-failure-testing.html
● Netflix Blog on ‘Failure Injection Testing’
techblog.netflix.com/2014/10/fit-failure-injection-testing.html
● ‘Lineage Driven Fault Injection’
http://people.ucsc.edu/~palvaro/molly.pdf
● ‘Automating Failure Testing Research at Scale’
https://people.ucsc.edu/~palvaro/socc16.pdf

86. Photo Credits
● http://etc.usf.edu/clipart/4000/4048/children_7_lg.gif
● http://cdn.c.photoshelter.com/img-get2/I0000MIN8fL0q8AA/fit=1000x750/taiw
an-hiking-river-tracing-walking.jpg
● http://i.imgur.com/iWKad22.jpg
● https://blogs.endjin.com/2014/05/event-stream-manipulation-using-rx-part-2/
● http://youpivot.com/category/features/
● https://www.cloudave.com/33427/boards-need-evolve-time/
● https://www.linkedin.com/pulse/amelia-packager-missing-data-imputation-ram
prakash-veluchamy

87. Watch the video with slide synchronization on
InfoQ.com!
https://www.infoq.com/presentations/netflix-
testing-research