1. Twisted
Michal Sedlak
michal6103@gmail.com

2. Based on
Dave Peticolas: Twisted Introduction
http://krondo.com/?page_id=1327
Orestis Markou: Asynchronous programming with Twisted
http://ep2011.europython.eu/conference/talks/asynchronous-pro
https://github.com/orestis/twisted-tutorial
Twisted documentation
http://twistedmatrix.com/documents
2

3. Outline
● What is Twisted?
● Why Twisted?
● Async vs. Sync
● Blocking vs. Non Blocking
● Parts:
● Reactor
● Factory
● Protocol
● Transfer
3

4. What is Twisted
Twisted is a networking engine written in Python,
supporting numerous protocols.
It contains a web server, numerous chat clients,
chat servers, mail servers, and more.
http://twistedmatrix.com/trac/wiki/TwistedProjects
4

5. Why use Twisted
● Python... 2 :(
● Asynchronous and event-based
● Full-featured
● Mail, web, news, chat, DNS, SSH, Telnet, RPC,
database access, and more
● Flexible
● Open source
5
http://www.eecho.info/Echo/python/why-use-twisted/

6. Async? Eventbased?
● Single thread
● Synchronous
● Asynchronous
● Multi-thread
6

7. Synchronous model
● 3 tasks
● One after another
● Simple flow
7

8. Threaded model
● Parallel execution
● Complex flow coordination
● IPC, critical section, race condition, synchronisation
8

9. Asynchronous model
● Interleaved tasks
● Sequence of small steps
● More complex than sync
No parallelism!
Q: Same execution time
as in sync model?
9

10. Typical network server blocks
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
while 1:
conn, addr = s.accept()
while 1:
data = conn.recv(1024)
if not data: break
conn.send(data)
conn.close()
10

11. Blocking
● Some function calls blocks
and sync programs waits
● I/O
● CPU much faster then:
● Disk
● Network
11

12. When and why async?
● Many tasks ● Non-blocking
● Interactive ● Responsive
● Independent tasks ● Less interprocess
communication means
less waiting for
processes
Network services
12

13. How?
● Low level I/O - Platform dependent
● Socket – setblocking(0)
● Async Disk I/O
– Python 3 - http://www.python.org/dev/peps/pep-3116/
– Twisted
http://twistedmatrix.com/documents/8.1.0/api/twisted.internet.f
● Reactor pattern and async libs - Twisted
13

14. Twisted basic parts
● Reactor
● Endpoint
● Factory
● Protocol
● Transfer
● Deferred
14

15. 15

16. Reactor
● Wait for I/O
● Handle Event
● Q: What if event
handler blocks?
16

17. Blocking event handler
stops application
17

18. ● potentially blocking operations
● reading or writing from a non-socket file descriptor
(pipe)
● waiting for a subprocess to finish
● Switch from blocking to non-blocking?
● Many standard Python functions are blocking only
● Example: os.system – always block
● Use Twisted API :)
18

19. First server
from twisted.internet import reactor
reactor.run()
● The reactor isn’t created explicitly, just imported.
● reactor.run(), reactor.stop()
● The reactor loop runs in the same thread it was started in.
● Once the loop starts up, it just keeps going.
● If it doesn’t have anything to do, the reactor loop does not
consume CPU.
● Default reactor is twisted.internet.selectreactor
19

20. Reactor
Status T S U Threading Processes Sched Platforms
C S D uling
P L P
select() Stable Y Y Y Y Y Y Unix,
Win32
poll Stable Y Y Y Y Y Y Unix
WaitForMultipleO Experimental Y Y Y Y Y Y Win32
bjects
Input/Output Experimental Y Y N N N Y Win32
Completion Port
CoreFoundation Unmaintained Y Y Y Y Y Y Mac OS X
epoll Stable Y Y Y Y Y Y Linux 2.6
GTK+ Stable Y Y Y Y Y Y Unix,
Win32
wx Experimental Y Y Y Y Y Y Unix,
Win32
kqueue Experimental Y Y Y Y Y Y FreeBSD
20
http://twistedmatrix.com/documents/current/core/howto/choosing-reactor.html

21. Change reactor
from twisted.internet import pollreactor
pollreactor.install()
from twisted.internet import reactor
reactor.run()
21

22. 22

23. Factory
● persistent configuration common to all
connections
● instantiates “protocol”
● access persistence: protocol_instance.factory
● factory does not listen to connections
● same service on multiple ports or network
addresses
23

24. 24

25. Protocol
● Per connection instance
● Protocol handling
● DataReceived(): Called whenever data is received.
● ConnectionLost(): Called when the connection is
shut down.
● MakeConnection(): Make a connection to a transport
and a server.
● ConnectionMade(): Called when a connection is
made.
25

26. 26

27. Transport
● Represents physical connection
● write(): send data over connection
● GetPeer()
● GetHost()
● loseConnection()
27

28. 28

29. Endpoint
● Listener or connector abstraction
cep = clientFromString(reactor
"tcp:host=www.example.com:port=80")
cep.connect(factory)
cep = clientFromString(reactor,
"ssl:host=web.example.com:port=443:"
"privateKey=foo.pem:certKey=foo.pem")
cep.connect(factory)
29

30. sep = serverFromString(reactor,
"tcp:80:interface=127.0.0.1")
sep.listen(factory)
sep = serverFromString(reactor,
"unix:/var/run/finger:mode=660")
sep.listen(factory)
30

31. Summary
● Reactor is most important
● reactor.run()
● Appropriate endpoint.
● Protocol factory to create protocol instances
● Protocol instances only deal with a single connection
● Shared state in factory
● Data arrive in chunks
● Implementing protocols is hard. Reuse as much code as
possible
31

32. What's next?
● Server example
● Deferred - promise
● Database handling – Async?
● Error handling – reverse error flow
● Trial
32

33. First server
● Proxy at localhost:8000
● Input: URL
● Output: Fetched data
33

34. First server
import time
import urllib2
from twisted.protocols import basic
class ProxyProtocol(basic.LineReceiver):
def lineReceived(self, url):
if not url.startswith('http://'):
return
start = time.time()
print 'fetching', url
connection = urllib2.urlopen(url)
data = connection.read()
print 'fetched', url,
self.transport.write(data)
self.transport.loseConnection()
34
print 'in', time.time() ­ start

35. from twisted.internet import protocol
class ProxyFactory(protocol.ServerFactory):
protocol = ProxyProtocol
from twisted.internet import reactor, endpoints
endpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)
factory = ProxyFactory()
endpoint.listen(factory)
reactor.run()
35

36. Run
Site Server Client Why same download
===== ====== ====== times?
A 0.771 3.817
B 0.567 2.026
C 1.457 2.026 c = urllib2.urlopen(url)
D 1.019 3.815 data=c.read()
­­­­­ ­­­­­­ ­­­­­­
Total: 3.813 3.817
36

37. We need async url data fetcher
● twisted.web.client.getPage
● http://launchpad.net/tx
37

38. import time
from twisted.web import client
from twisted.protocols import basic
class ProxyProtocol(basic.LineReceiver):
def lineReceived(self, url):
if not url.startswith('http://'):
return
start = time.time()
print 'fetching', url
deferredData = client.getPage(url)
def urlFetched(data):
self.transport.write(data)
self.transport.loseConnection()
print 'fetched', url,
print 'in', time.time() ­ start
deferredData.addCallback(urlFetched) 38

39. from twisted.internet import protocol
class ProxyFactory(protocol.ServerFactory):
protocol = ProxyProtocol
from twisted.internet import reactor, endpoints
endpoint = endpoints.TCP4ServerEndpoint(reactor, 8000)
factory = ProxyFactory()
endpoint.listen(factory)
reactor.run()
39

40. Site Server Client Better :)
===== ====== ======
A 0.850 0.853
B 0.486 0.488
C 1.582 1.584
D 0.999 1.000
­­­­­ ­­­­­­ ­­­­­­
Total: 3.918 1.585
40