An introduction to Twisted, an event-driven networking engine. Presentation I gave at a mind share session at Rackspace.

1. Twisted
“An event-driven networking engine
written in Python”
Ying Li (ying.li@rackspace.com)

2. Twisted
“An event-driven networking engine
written in Python”

4. OS / System
Transport
Protocol
Application (e.g. server)
Runner / Daemonization

5. OS / System
Transport
Protocol
Application (e.g. server)
Runner / Daemonization

6. Transport
•manages socket or pipe

7. Transport
•manages socket or pipe
•writes data

8. Transport
•manages socket or pipe
•writes data
•reads data
•notifies protocol

9. Transport
•TCP •AF_UNIX
•UDP •Pipes

10. Transport
•TCP •AF_UNIX
•UDP •Pipes
•TLS •memory

11. OS / System
Transport
Protocol
Application (e.g. server)
Runner / Daemonization

12. Incoming data
Transport
Protocol
•handle data received

13. Outgoing data Incoming data
Transport
Protocol
•handle data received
•write data to transport

14. Outgoing data Incoming data
Transport
Protocol
•handle data received
•write data to transport
•handle connections

15. Transport
Protocol

16. TCP
TLS

17. Process
TLS

18. Bluetooth?
TLS

19. OS / System
Transport
Protocol
Application (e.g. server)
Runner / Daemonization

20. Outgoing data / System
OS Incoming data
Transport
Protocol
Transport
...
Protocol
Transport
Protocol
Application (e.g. server)

21. Outgoing data Incoming data
TCP
TLS (protocol)
TLS (transport)
HTTP

22. Outgoing data Incoming data
UDP
TCP (protocol)
TCP (transport)
TLS (protocol)
TLS (transport)
HTTP

23. Protocol
•TLS •IMAP •SOCKS
•HTTP •SMTP •TELNET
•IRC •POP •SIP (voip)
•XMPP •DNS •NMEA (gps)
•OSCAR •SSH •...more

24. OS / System
Transport
Protocol
Application (e.g. server)
Runner / Daemonization

25. wait... that’s it?

26. Protocol
Application (e.g. server)

27. Application / Server
•IRC •NNTP
•SMTP •POP
•SSH •Generic
•SFTP •...more
•HTTP

28. Application / Clients
•IRC •NNTP
•SMTP •POP
•SSH •...more
•SFTP
•HTTP

29. server and/or client

30. Protocol
Application
•builds protocol
•hooks up protocol

31. Protocol
Application
•builds multiple protocols
•database
•authentication
•hooks up protocol

32. Authentication
•unix user/password
•ssh key
•user/password list file
•in-memory user/password

33. OS / System
Transport
Protocol
Application (e.g. server)
Runner / Daemonization

34. python myapplication.py
Application (e.g. server)
Runner / Daemonization

35. twistd [options] <myplugin>

36. twistd
•daemonization
•logging
•privilege dropping
•chroot
•non-default reactor
•profiling

37. twistd [options] <myplugin>
[my_plugin_options]

38. twistd
twistd web --port 80 --path /srv/web

39. twistd
twistd web --port 8080 --path /srv/web
twistd telnet --port 4040

40. twistd
•twistd web •twistd ftp
•twistd telnet •twistd mail
•twistd dns •twistd conch (ssh)
•...more (see `twistd --help`)

41. Twisted
“An event-driven networking engine written in
Python”

42. Concurrency Models
(A story of 3 HTTP requests)

43. Time
HTTP Request

44. Time
Await response

45. Time
Process response

46. Time
Request 2 Request 3

47. Time
Interpolate
Responses

48. Time Multi-Process
Process 1
Process 2
Process 3
Blocking I/O Request 1
Interpolation Request 2
Request 3

49. Time Multi-Threaded
Thread 1
Thread 2
Thread 3
Blocking I/O Request 1
Interpolation Request 2
Request 3

50. Time Multi-Threaded
Thread 1
Thread 2
Thread 3
Blocking I/O Request 1
Interpolation Request 2
Request 3

51. Time Multi-Threaded
Thread 1
Thread 2
Thread 3
Blocking I/O Request 1
Interpolation Request 2
Request 3

52. Time Multi-Threaded
Thread 1
Thread 2
Thread 3
Blocking I/O Request 1
Interpolation Request 2
Request 3

53. Time Multi-Threaded
Thread 1
Thread 2
Thread 3
Blocking I/O Request 1
Interpolation Request 2
Request 3

54. Threads
• Locking
• Re-entrancy
• Debugging

55. Event-Driven
(not to scale)

56. Time
Callbacks: CB

57. Time
First HTTP Request
Callbacks: CB CB

58. Time
Second HTTP Request
Callbacks: CB CB CB

59. Time
Event!
Callbacks: CB CB CB

60. Time
Event!
Callbacks: CB CB CB

61. Time
CB
Callbacks: CB CB

62. Time
CB
Event!
Callbacks: CB CB CB

63. Time
CB
Event!
Callbacks: CB CB CB

64. Time
CB CB
Event!
Callbacks: CB CB

65. Time
CB CB CB
Callbacks: CB

66. Time
CB CB CB CB
Callbacks:

68. Events Handlers

69. Events Handlers
• file descriptors
• timed events

70. • select • CoreFoundation
• poll • IOCP
• epoll • ...more
• kqueue

71. Deferreds

72. Deferreds
(promises of future results)

73. D
R1
CB
R2

74. D
R1
CB
R2

75. D
F1
CB EB
R2

76. D
R1
CB1 EB
CB2
CB3
CB3 (CB 2 (CB 1 (R1)))

77. D
CB EB A
EB B
EBC

78. D try:
...
F1
CB1 EB except:
return R2
CB2 R2

79. D try:
...
F1=C
CB1 EB A except A:
...
EB B except B:
...
EBC except C:
return R2

80. D try:
...
F1=C
CB1 EB A except A:
...
EB B except B:
...
EBC except C:
return R2
CB2 R2

81. D
D D D D
CB
EB
CB EB
EB
CB EB
CB

82. courtesy of http://callbackhell.com
fs.readdir(source, function(err, files) {
if (err) {
console.log('Error finding files: ' + err)
} else {
files.forEach(function(filename, fileIndex) {
console.log(filename)
gm(source + filename).size(function(err, values) {
if (err) {
console.log('Error identifying file size: ' + err)
} else {
console.log(filename + ' : ' + values)
aspect = (values.width / values.height)
widths.forEach(function(width, widthIndex) {
height = Math.round(width / aspect)
console.log('resizing ' + filename + 'to ' + height + 'x' + height)
this.resize(width, height).write(destination + 'w' + width + '_' + filename, function(err) {
if (err) console.log('Error writing file: ' + err)
})
}.bind(this))
}
})
})
}
})

83. • jQuery Deferreds
• MochiKit Deferreds
• Dojo Deferreds
• Google Closure Deferreds

84. Twisted
•easier to reason about concurrency
•lots of components
•useful abstractions