Netflix runs hundreds of multivariate A/B tests a year, many of which help personalize the experience in the UI. This causes an exponential growth in the number of user experiences served to members, with each unique experience resulting in a unique JS/CSS bundle. Pre-publishing millions of permutations to the CDN for each build of each UI simply does not work at Netflix scale. In this session, we discuss how we built, designed, and scaled a brand new Node.js service, Codex. Its sole responsibility is to build personalized JS/CSS bundles on the fly for members as they move through the Netflix user experience. We’ve learned a ton about building a horizontally scalable Node.js microservice using core AWS services. Codex depends on Amazon S3 and Amazon DynamoDB to meet the streaming needs of our 100 million customers.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
A N E T F L I X E N G I N E E R I N G O R I G I N A L
A story of Netflix and AB Testing
in the User Interface Using
DynamoDB
Alex Liu @stinkydofualiu@netflix.com
A W S r e : I N V E N T

2. Alex Liu
Node.js Platform
aliu@netflix.com
@stinkydofu

4. #netflixeverywhere

6. Contents
I. The problem
II. Codex: Bundling as a service
III. Scaling for Netflix
IV. Looking to the future

7. I. The problem

8. It's all about building
JS bundles.

9. // home.js
require('foo');
require('bar');
home.js

10. home.js profile.js
feed.js signup.jsalbum.js
var bundles = [
'home.js',
'profile.js',
'feed.js',
'album.js',
'signup.js',
];

12. var bundles = [
'home.js',
'homeIE.js',
'profile',
'profileIE',
'feed.js',
'feedIE.js'
'album.js',
'albumIE.js',
...
];
/home
home.js homeIE.js
YESNO

13. Netflix
AB testing!

14. AB test w/ multiple cells
Cells Control (Cell 1) Cell 2 Cell 3
Movie
Cover
Art
14% 6%

15. Netflix
AB testing!

16. home.js
newSearch.jsoldSearch.js

17. home.js
newSearch.jsoldSearch.js

18. home.js
newSearch.jsoldSearch.js
jQuery React~80KB ~130KB

19. Larger bundle size:
• File sizes
• Time to download
• Memory usage
• Time to interactive (TTI)

20. Old-school Lego bricks
were generic.

21. New Lego is about
specialization.

22. Hard to reuse
specialized bricks.

23. home.js
newSearch.jsoldSearch.js

24. home.js
oldSearch.js

25. home.js
newSearch.js

26. // starting to look like a
// lot of bundles...
var bundles = [
'homeNewSearch.js',
'homeNewSearchIE.js',
'homeOldSearch.js',
'homeOldSearchIE.js',
...
];
4x variations
already!

27. Netflix runs hundreds of
AB tests.
But we personalize on many
other dimensions too.

30. |S1| ⋅ |S2| ⋅ ⋅ ⋅ |Sn| = |S1 × S2 × ⋅ ⋅ ⋅ × Sn|
3100
= 5.1537752e47

31. 40,000,000,000 bricks to reach the
7,600,000,000,000,000 bricks to reach
Enough bricks to reach 6.7812832e32 times.

32. That’s a ton of bundles!

33. https://xkcd.com/303/

34. Website's full
bundle is 8 MB.

35. How do we deal with
conditional
dependencies?

36. II. Codex: Bundling as a service

37. What if we generate
on-demand?

38. Step 1
Build the conditional
dependency graph.

39. git

40. git artifact

41. artifact

42. artifact

43. web/v1 web/v2 web/v3 web/v4

44. Step 2
Evaluate the dynamic inputs.

45. AB Tests

46. AB Tests

47. noun, plural [trooth z, trooths]
A bucket of boolean
flags used to build a
personalized Netflix
experience

48. {
"webfonts": false,
"instantSearch": true,
"socialFeatures": false,
"motionBanner": true,
"html5video": true,
"customScrollbar": true
}

50. Step 3
Apply the truths against the
conditional dependency graph.

51. <script/>
http://codex.nflxext.com/web/v1/home.js/oldSearch,foo

52. <script/> <script/>
codex
? I got
this!
http://codex.nflxext.com/web/v1/home.js/oldSearch,foo

53. codex {
oldSearch: true,
foo: true
}
web/v1
http://codex.nflxext.com/web/v1/home.js/oldSearch,foo
http://codex.nflxext.com/{team}/{version}/{entryPoint}/{truths}

54. oldSearch
foo
bar
{
oldSearch: true,
foo: true
} newSearch

55. response times <= 70ms

56. <script/> <script/>
codex
Cached! Here
you go!
http://codex.nflxext.com/web/v1/home.js/oldSearch,foo

57. III. Scaling for Netflix

58. web/v1 web/v2 tv/v5 tv/v7

59. Build
management

60. 1. Where are the
artifacts stored?
2. How do we find
stored artifacts?

61. Storage Metadata

62. Codex is mission
critical.

63. Amazon S3
Storage
Amazon
DynamoDB
Metadata

64. • Durable
• Reliable
• Scalable
• Available *
• Node.js client
* Except on 2/28/17
Amazon S3
Storage

65. • NoSQL
• Node.js client
• DynamoDB Streams *
• Available…?
* Didn’t quite work out
DynamoDB
Metadata

66. Saved!
web/v1
Management service
Saved!

67. codex
Runtime service

68. activate
web/v1 web/v1
OK!
Management service
Activate!

69. Runtime service
codex
Here are all
the active
artifacts!

70. codex
Runtime service

71. DynamoDB can’t
keep up!

72. • Polling works because we
are behind CDN
• Understand partition keys
and partition behavior
• Read the limits and best
practices guidesDynamoDB
Caveats

73. What about
operational resiliency?

74. February 28, 2017, was just another day.
Until it wasn’t.

76. eu-west
us-west
us-east

77. eu-west
us-west
us-east

78. Replication has a
lot of caveats.
Amazon S3 can only replicate to one destination.
Amazon S3 can’t replicate transitive objects.
DynamoDB replication isn’t real time.

79. Every AWS Region is wholly independent!
eu-west
us-west
us-east
codex-sdk

81. us-east
???
eu-west
us-west

82. ???
???
???
eu-west
us-west

84. What about
Codex itself?

85. prod prod-new canary

86. 16 GB ought to be
enough for us! 🙂

87. codex

88. 400+ artifacts!
…and we ran out of memory.

89. 32 GB ought to be
enough for us!
🤔

90. 800+ artifacts!
…and we ran out of memory.

91. 64 GB ought to be
enough for us…? 😭

92. 1,600+ artifacts!
Our teams will use as
much as we give them.
…and we ran out of memory. Again.

93. Q: What's cheap,
plentiful, and fast
enough?
A: Disk.

94. codex
LevelDB

95. 100% CPU usage.

96. What's the problem?

97. v8::internal::Runtime_ParseJson~68%

98. codex
LevelDB
JSON.parse()

99. JSON.parse is slow.
And blocks the CPU!

100. codex
LevelDB
JSON.parse()
LRU cache: Saving the CPU

104. JSON wasn’t done
with us.

105. RangeError: Invalid string length

106. It dates back all the way to 2010. Back
then, 512 MB was the overall heap limit,
and I guess it was a no-brainer to decide
that no single string could/should be
bigger than the heap ;-) (512 MB =
memory consumption of a UTF-16 string
with length 2^28.) – jmrk Jun 14 at 15:13

107. IV. Looking to the future

108. How do we clean up
old artifacts?

109. Why not {bundler}?

110. Don't be afraid to
challenge common
convention.

111. Don't make
assumptions about
the upper limits.

112. Don't optimize before you
understand the system.

113. Use the scientific method:
1. Gather data
2. Formulate hypothesis
3. Test hypothesis
4. Repeat

114. Engineer for fault tolerance

115. Netflix scale is
challenging.

116. https://www.flickr.com/clement127/
https://www.flickr.com/jose_antonio_hidalgo_jimenez/
https://www.flickr.com/reiterlied/
Lego photo credits

117. Image credits

118. Image credits

119. Image credits
Artist: alecive (Alessandro Roncone)
Iconset Homepage: https://github.com/alecive/FlatWoken

120. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
fin
Alex Liu @stinkydofualiu@netflix.com