Modern systems in production rely on decades of computer science research. Over time, new architectural patterns emerge that enable more resilient and robust systems. In this talk, we'll discuss some of these patterns from systems I've worked on at Google and the related work that provide insights into the motivations behind them.

1. Confidential + ProprietaryConfidential + Proprietary
Distributed Systems in Practice, in Theory
Aysylu Greenberg
March 8th, 2016

2. Confidential + Proprietary

3. Confidential + Proprietary

4. Confidential + Proprietary
Aysylu Greenberg
@aysylu22

5. Confidential + Proprietary
Towards Distributed Build System

6. Confidential + Proprietary
Towards Distributed Build System

7. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

8. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

9. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

10. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

11. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

12. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

13. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

14. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

15. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

16. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

17. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

18. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

19. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

20. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

21. Confidential + Proprietary
http://opencv.org/
Towards Distributed Build System

22. Confidential + Proprietary

23. Confidential + Proprietary
Papers We Love SF

24. Confidential + Proprietary
Aysylu Greenberg
@aysylu22

25. Confidential + Proprietary
meetup.com/Papers-We-Love-London

26. Confidential + Proprietary
Today
● Staged Event-Driven Architecture

27. Confidential + Proprietary
Today
● Staged Event-Driven Architecture
● Leases

28. Confidential + Proprietary
Today
● Staged Event-Driven Architecture
● Leases
● Inaccurate Computations

29. Confidential + ProprietaryConfidential + Proprietary
Computer Science Research
In
Distributed Systems Industry

30. Confidential + Proprietary
Operating systems research

31. Confidential + Proprietary
Operating systems research

32. Confidential + Proprietary
Operating systems research
Concurrency

33. Confidential + Proprietary
Operating systems research
Concurrency
Concurrency primitives:
mutex & semaphore

34. Confidential + Proprietary
Operating systems research
Concurrency
Concurrency primitives:
mutex & semaphore
Processes execute at
different speeds

35. Confidential + Proprietary
Time in distributed systems

36. Confidential + Proprietary
Time in distributed systems

37. Confidential + Proprietary
Time in distributed systems
Pipelining

38. Confidential + Proprietary
Time in distributed systems
Pipelining

39. Confidential + Proprietary
Internet

40. Confidential + Proprietary
Internet
Distributed consensus

41. Confidential + Proprietary
Internet
Distributed consensus

42. Confidential + Proprietary
Internet
Distributed consensus

43. Confidential + Proprietary
Internet
Distributed consensus
Paxos

44. Confidential + Proprietary
Reconsider large systems

45. Confidential + Proprietary
Reconsider large systems
Platform as a service

46. Confidential + Proprietary
CS Research is Timeless
Inform decisions
Mitigate technical risk

47. Confidential + ProprietaryConfidential + Proprietary
Staged Event
Driven
Architecture
&
Deep Pipelines
2001

48. Confidential + Proprietary
Hardware Pipelines

49. Confidential + Proprietary
Data Pipelines
https://en.wikipedia.org/wiki/Graphics_pipeline

50. Confidential + Proprietary
Search Indexing Pipelines

51. Confidential + Proprietary
Search Indexing Pipelines

52. Confidential + Proprietary
Search Indexing Pipelines

53. Confidential + Proprietary
Search Indexing Pipelines

54. Confidential + Proprietary
Search Indexing Pipelines

55. Confidential + Proprietary
Search Indexing Pipelines

56. Confidential + Proprietary
Search Indexing Pipelines

57. Confidential + Proprietary
Search Indexing Pipelines

58. Confidential + Proprietary
Search Indexing Pipelines

59. Confidential + Proprietary
Search Indexing Pipelines

60. Confidential + Proprietary
Search Indexing Pipelines

61. Confidential + Proprietary

62. Confidential + Proprietary
Staged Event Driven Architecture

63. Confidential + Proprietary
Single-machine pipeline
generalizes to distributed pipelines
Staged Event Driven Architecture

64. Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers

65. Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers
+
-

66. Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers
○ automatic tuning
+
-

67. Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers
○ automatic tuning
■ thread pool sizing
+
-

68. Confidential + Proprietary
Staged Event Driven Architecture
● Dynamic resource controllers
○ automatic tuning
■ thread pool sizing
■ event batching +
-

69. Confidential + Proprietary
+
-
Staged Event Driven Architecture
● Dynamic resource controllers
○ automatic tuning
■ thread pool sizing
■ event batching
○ load shedding via queue

70. Confidential + Proprietary
+
-
Staged Event Driven Architecture
● Dynamic resource controllers
○ automatic tuning
■ thread pool sizing
■ event batching
○ load shedding via queue
■ backpressure

71. Confidential + Proprietary
+
-
Staged Event Driven Architecture
● Dynamic resource controllers
○ automatic tuning
■ thread pool sizing
■ event batching
○ load shedding via queue
■ backpressure
■ threshold, filter, reorder, aggregate

72. Confidential + ProprietaryConfidential + Proprietary
Leases
as Heart Beat in
Distributed
Systems 1989

73. Confidential + Proprietary

74. Confidential + Proprietary
Leases
● Distributed locking

75. Confidential + Proprietary
Leases
● Distributed locking
● Lease term tradeoffs
○ short

76. Confidential + Proprietary
Leases
● Distributed locking
● Lease term tradeoffs
○ short
■ delay from client and server failures
minimized
■ reduced false write-sharing

77. Confidential + Proprietary
Leases
● Distributed locking
● Lease term tradeoffs
○ short vs long

78. Confidential + Proprietary
Leases
● Distributed locking
● Lease term tradeoffs
○ short vs long
■ more efficient for frequently accessed
data
■ minimized lease extension overhead on
server and client

79. Confidential + Proprietary
Leases
● Distributed locking
● Lease term tradeoffs
○ short vs long
● Use of leases in modern applications
○ Leader election TTL (in etcd)

80. Confidential + Proprietary
Leases
● Distributed locking
● Lease term tradeoffs
○ short vs long
● Use of leases in modern applications
○ Leader election TTL (in etcd)
○ Liveness detection

81. Confidential + Proprietary

82. Confidential + Proprietary
Do the build, s'il vous plaît!
Build
System
Dita Roque-Gourary

83. Confidential + Proprietary
Do the build, s'il vous plaît!
Build
System
OK/D'Accord!
Dita Roque-Gourary

84. Confidential + Proprietary
Do the build, s'il vous plaît!
Build
System
OK/D'Accord!
Waiting for the results, merci.
Dita Roque-Gourary

85. Confidential + Proprietary
Do the build, s'il vous plaît!
Build
System
OK/D'Accord!
Waiting for the results, merci.
Build is in progress
Dita Roque-Gourary

86. Confidential + Proprietary
Do the build, s'il vous plaît!
Build
System
OK/D'Accord!
Waiting for the results, merci.
Build is in progress
Dita Roque-Gourary
Waiting for the results, merci.

87. Confidential + Proprietary
Do the build, s'il vous plaît!
Build
System
OK/D'Accord!
Waiting for the results, merci.
Build is in progress
Build is finished/fini.
Dita Roque-Gourary
Waiting for the results, merci.

88. Confidential + Proprietary
Leases in Build System

89. Confidential + ProprietaryConfidential + Proprietary
Inaccurate
Computations
&
Serving Search
Results

90. Confidential + Proprietary
From Accurate to
"Good Enough"

91. Confidential + Proprietary

92. Confidential + Proprietary

93. Confidential + Proprietary

94. Confidential + Proprietary

95. Confidential + Proprietary

96. Confidential + Proprietary
Inaccuracy for Resilience
1. Task decomposition

97. Confidential + Proprietary
Inaccuracy for Resilience
1. Task decomposition
2. Baseline for correctness

98. Confidential + Proprietary
Inaccuracy for Resilience
1. Task decomposition
2. Baseline for correctness
3. Criticality Testing

99. Confidential + Proprietary
Inaccuracy for Resilience
1. Task decomposition
2. Baseline for correctness
3. Criticality Testing
4. Distortion and timing models

100. Confidential + Proprietary
Jeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010

101. Confidential + Proprietary
Jeff Dean "Building Software Systems at Google and Lessons Learned", Stanford, 2010

102. Confidential + Proprietary

103. Confidential + Proprietary

104. Confidential + Proprietary
Robust & scaleable pipelines

105. Confidential + Proprietary
Robust & scaleable pipelines
Leases for sharing & heartbeat

106. Confidential + Proprietary
Robust & scaleable pipelines
Leases for sharing & heartbeat
Trade off inaccuracy for
resilience & performance

107. Confidential + Proprietary
Robust, scaleable pipelines
Leases for sharing & heartbeat
Trade off inaccurate for
resilient
CS research is
timeless:
use it to mitigate risk

108. Confidential + Proprietary
Thanks
Ines Sombra
David Greenberg
Alex Hutcheson
Scott Zawalski
Karan Parikh

109. Confidential + Proprietary
References
● T. Wurthinger, C. Wimmer et al. "One VM to Rule Them All"
● M. Rinard "Probabilistic Accuracy Bounds for Fault-Tolerant
Computations that Discard Tasks"
● F. Corbato, M. Daggett, R. Daley "An Experimental Time-
Sharing System"
● E. Dijkstra "Cooperating Sequential Processes"
● L. Lamport "Time, Clocks, and the Ordering of Events in a
Distributed System"

110. Confidential + Proprietary
References
● B. Oki, B. Liskov "Viewstamped Replication: A New Primary Copy
Method to Support Highly-Available Distributed Systems"
● L. Lamport "The Part-Time Parliament"
● M. Welsh, D. Culler, E. Brewer "SEDA: An Architecture for Well-
Conditioned, Scalable Internet Services"
● C. Gray, D. Cheriton "Leases: An Efficient Fault-Tolerant
Mechanism for Distributed File Cache Consistency"
● S. Agarwal, B. Mozafari et al. "BlinkDB: Queries with Bounded
Errors and Bounded Response Times on Very Large Data"

111. Confidential + Proprietary
Should I read papers?
YES