Slides from my talk at Open Source Days 2011 on Event-based webservers.

1. Scaling with event–
based webservers
Morten Siebuhr
Open Source Days
March 5th 2011
1/66

2. whoami(1)
Defining “web–server work”
Killing Apache
Event–based servers
Using Node.js
Questions
2/66

3. whoami(1)
3/66

4. Computer Scientist
Distributed Systems / Scientific Computing
(Web–)developer @ One.com
4/66

5. Web–development @ one.com:
Rich Internet Applications
e–mail, calendar, galleries, . . .
+ 900.000 customers
5/66

6. 6/66

7. Defining
“Web–server work”
7/66

8. Typical web–server work
8/66

9. Typical web–server work
Serving static files
8/66

10. Typical web–server work
Serving static files
Talking to databases
8/66

11. Typical web–server work
Serving static files
Talking to databases
Not doing lots of computations
8/66

12. Typical web–server work
Serving static files
Talking to databases
Not doing lots of computations
≈ 100 → 1000 connections served ASAP
8/66

13. (A)typical web–server work
9/66

14. (A)typical web–server work
Persistent connections
9/66

15. (A)typical web–server work
Persistent connections
= ∞ simultaneous connections
9/66

16. (A)typical web–server work
Persistent connections
= ∞ simultaneous connections
≈ 10000 simultaneous connections
9/66

17. (A)typical web–server work
Persistent connections
= ∞ simultaneous connections
≈ 10000 simultaneous connections
(But we don’t care as much about latency. . . )
9/66

18. waiting...
10/66

19. sleep(1.0)
11/66

20. 12/66

21. Response time w. sleep(1).
25 s
WSGI
20 s
Average response time
15 s
10 s
5s
0s
0 100 200 300 400 500 600 700 800 900 1000
Concurrent requests
13/66

22. Response time w. sleep(1).
25 s
WSGI
20 s
Average response time
15 s
10 s
5s
0s
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Concurrent requests
14/66

23. Threading
15/66

24. Use threads
16/66

25. Wasted resources!
17/66

26. Add threads
18/66

27. Response time w. sleep(1).
25 s
WSGI
Apache
20 s
Average response time
15 s
10 s
5s
0s
0 100 200 300 400 500 600 700 800 900 1000
Concurrent requests
19/66

28. Response time w. sleep(1).
25 s
WSGI
Apache
20 s
Average response time
15 s
10 s
5s
0s
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Concurrent requests
20/66

29. 21/66

30. 22/66

31. 23/66

33. Memory usage w. sleep(1).
1500 MB
WSGI
Apache
1250 MB
1000 MB
Max Virtual Memory
750 MB
500 MB
250 MB
0 MB
0 100 200 300 400 500 600 700 800 900 1000
Concurrent requests
25/66

34. Memory usage w. sleep(1).
1500 MB
WSGI
Apache
1250 MB
1000 MB
Max Virtual Memory
750 MB
500 MB
250 MB
0 MB
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Concurrent requests
26/66

35. A single connection
27/66

36. A single connection
=
1 thread
27/66

37. A single connection
=
1 thread
=
1 C–stack & kernel data structures
27/66

38. A single connection
=
1 thread
=
1 C–stack & kernel data structures
+
Locking & switching overhead &c. . .
27/66

39. 28/66

40. Add memory!
29/66

41. Add memory!
Memory is cheap
29/66

42. Add memory!
Memory is cheap
Except, actually using it isn’t cheap. . .
29/66

43. Memory Wall
1986 → 2000:
30/66

44. Memory Wall
1986 → 2000:
+ 55% CPU speed P/A
30/66

45. Memory Wall
1986 → 2000:
+ 55% CPU speed P/A
+ 10% RAM speed P/A
(Source: http://www.cs.virginia.edu/papers/Hitting_Memory_Wall-wulf94.pdf)
30/66

46. 31/66

47. Threads and connections mapped 1:1
32/66

48. Threads and connections mapped 1:1
We use a lot of threads
32/66

49. Threads and connections mapped 1:1
We use a lot of threads
Which we don’t have the memory bandwidth to move around.
32/66

50. Events!
33/66

51. waiting...
34/66

52. 35/66

53. 36/66

54. Event-basics
37/66

55. Event-basics
Event queue
37/66

56. Event-basics
Event queue
Event loop
37/66

57. Event-basics
Event queue
Event loop
“The Event Loop”
37/66

58. A single connection
38/66

59. A single connection
=
1 “Event”
38/66

60. A single connection
=
1 “Event”
=
Some variables & function
38/66

61. A single connection
=
1 “Event”
=
Some variables & function
=
Hash table & some pointers
38/66

62. A single connection
=
1 “Event”
=
Some variables & function
=
Hash table & some pointers
+
Queue of events waiting to be processed
38/66

63. 39/66

64. 39/66

65. 39/66

66. 39/66

67. Why not earlier
40/66

68. Why not earlier
Simply not a problem!
40/66

69. Why not earlier
Simply not a problem!
Traditional back–end programmers not used to thinking this way
40/66

70. What we gain
41/66

71. What we gain
Relatively low memory use
41/66

72. What we gain
Relatively low memory use
No threading issues
41/66

73. What we gain
Relatively low memory use
No threading issues
Client-side programmers have no preconditions.
41/66

74. Node.js
42/66

75. Node.js is a set of bindings to the V8 JavaScript engine
for scripting network programs.
— Ryan Dahl
43/66

76. Node.js is a set of bindings to the V8 JavaScript engine
for scripting network programs.
— Ryan Dahl
43/66

77. JavaScript
44/66

78. JavaScript
Built for event-based programming.
44/66

79. JavaScript
Built for event-based programming.
44/66

80. JavaScript
Built for event-based programming.
Netscape needed something fast. . .
44/66

81. JavaScript
Built for event-based programming.
Netscape needed something fast. . .
. . . JavaScript “designed” and implemented in 14 days
44/66

82. JavaScript
Built for event-based programming.
Netscape needed something fast. . .
. . . JavaScript “designed” and implemented in 14 days
44/66

83. Google V8
JIT’ing JavaScript to native machine code
45/66

84. Google V8
JIT’ing JavaScript to native machine code
= Compiling server code
45/66

85. Node.js
Non–browser V8–bindings for various libraries
Everything non–blocking
46/66

86. A minimal webserver in Node.js. . .
// Import HTTP package
var http = require ( ’ http ’) ;
// Set up basic server
var server = http . createServer ( function ( req , res ) {
res . writeHead (200 , { ’ Content - Type ’: ’ text / plain ’ }) ;
res . end ( ’ Hello World n ’) ;
}) ;
// Start the server
server . listen (8124 , " 127.0.0.1 " ) ;
console . log ( ’ Server running at http : / / 1 2 7 . 0 . 0 . 1 : 8 1 2 4 / ’) ;
47/66

87. . . . with “database”
// Import HTTP package
var http = require ( ’ http ’) ;
// Set up basic server
var server = http . createServer ( function ( req , res ) {
setTimeout ( function () {
res . writeHead (200 , { ’ Content - Type ’: ’ text / plain ’ }) ;
res . end ( ’ Hello World n ’) ;
} , 1000) ;
}) ;
// Start the server
server . listen (8124 , " 127.0.0.1 " ) ;
console . log ( ’ Server running at http : / / 1 2 7 . 0 . 0 . 1 : 8 1 2 4 / ’) ;
48/66

88. Response time w. sleep(1).
25 s
WSGI
Apache
Node.js
20 s
Average response time
15 s
10 s
5s
0s
0 100 200 300 400 500 600 700 800 900 1000
Concurrent requests
49/66

89. Response time w. sleep(1).
25 s
WSGI
Apache
Node.js
20 s
Average response time
15 s
10 s
5s
0s
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Concurrent requests
50/66

90. Memory usage w. sleep(1).
1500 MB
WSGI
Apache
Node.js
1250 MB
1000 MB
Max Virtual Memory
750 MB
500 MB
250 MB
0 MB
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
Concurrent requests
51/66

91. Connections in Node.js
52/66

92. Connections in Node.js
=
1 “Event”
52/66

93. Connections in Node.js
=
1 “Event”
=
1 Closure + callback function
52/66

94. Connections in Node.js
=
1 “Event”
=
1 Closure + callback function
=
1 struct + some pointers
52/66

95. Connections in Node.js
=
1 “Event”
=
1 Closure + callback function
=
1 struct + some pointers
+
Event queue operations + function calls
52/66

96. Alternatives
53/66

97. Alternatives
Well–written thread program!
53/66

98. Alternatives
Well–written thread program!
Erlang
http://www.erlang.org/
53/66

99. Alternatives
Well–written thread program!
Erlang
http://www.erlang.org/
Google GO
http://golang.org/
53/66

100. Alternatives
Well–written thread program!
Erlang
http://www.erlang.org/
Google GO
http://golang.org/
Lua Event Machine
https://github.com/esmil/lem
53/66

101. Alternatives
Well–written thread program!
Erlang
http://www.erlang.org/
Google GO
http://golang.org/
Lua Event Machine
https://github.com/esmil/lem
Nginx
http://wiki.nginx.org/
53/66

102. Alternatives
Well–written thread program!
Erlang
http://www.erlang.org/
Google GO
http://golang.org/
Lua Event Machine
https://github.com/esmil/lem
Nginx
http://wiki.nginx.org/
Varnish
http://www.varnish-cache.org/
53/66

103. Working with Node.js
54/66

104. I’m new to JavaScript!
55/66

105. 56/66

106. JavaScript WTFs
Always add ;!
Function arguments. . .
this scoped to callee object!
Stacktraces + events sometimes look strange
57/66

107. JSLint
58/66

108. JSLint
www.crockfordfacts.com
58/66

109. Nice code!
def process ( user ) :
user = getAuth ( user )
if not user :
return ’ Fail ’
db = getC onnnecti on ( params )
data = db . getUserData ( user )
if not data :
return ’ Fail ’
return data
59/66

110. Boomerang code
function process ( user , callback ) {
getAuth ( req . username , function ( err , user ) {
if ( err ) return callback ( err ) ;
g etDB Co n ne ct io n ( params , function ( err , db ) {
if ( err ) return callback ( err ) ;
db . getUserData ( user , function ( err , data ) {
if ( err ) return callback ( err ) ;
return callback ( null , data ) ;
}) ;
}) ;
}) ;
}
60/66

111. NPM
61/66

112. Express.js
var app = require ( ’ express ’) . createServer () ;
app . get ( ’/ ’ , function ( req , res ) {
setTimeout ( function () {
res . send ( ’ Hello World ’) ;
} , 1000) ;
}) ;
app . listen (3000) ;
62/66

113. Express.js - Chat server
var app = require ( ’ express ’) . createServer () ,
chat = [];
app . get ( ’/ ’ , function ( req , res ) {
chat . push ( res ) ;
}) ;
app . get ( ’ /: msg ’ , function ( req , res ) {
chat . forEach ( function ( conn ) {
conn . write ( req . params . msg ) ;
}) ;
}) ;
app . listen (3000) ;
63/66

114. Questions
(The end)
64/66

115. http://nodejs.org/
https://github.com/joyent/node
65/66

116. function process ( user , callback ) {
var user = null , db = null ;
function done_ () {
if ( user && db ) {
db . getUserData ( user , function ( err , data ) {
if ( err ) return callback ( err ) ;
return callback ( null , data ) ;
}) ;
}
}
getAuth ( req . username , function ( err , retuser ) {
if ( err ) return callback ( err ) ;
user = retuser ;
done_ () ;
}) ;
g et DBCo n ne ct i on ( params , function ( err , retdb ) {
if ( err ) return callback ( err ) ;
db = retdb ;
done_ () ;
}) ;
}
66/66