Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generators:
The Final Frontier
David Beazley (@dabeaz)
...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Previously on Generators
2
• GeneratorTricks for System...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Previously on Generators
3
• A Curious Course on Corout...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Today's Installment
4
• Everything else you ever wanted...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Requirements
5
• You need Python 3.4 or newer
• No thir...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Disclaimer
6
• This is an advanced tutorial
• Assumes g...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Will I Be Lost?
7
• Although this is the third part of ...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Focus
8
• Material in this tutorial is probably not
imm...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part I
9
Preliminaries - Generators and Coroutines
(roc...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generators 101
• yield statement defines a generator fun...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Under the Covers
11
• Generator object runs in response...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Interlude
12
• Generators as "iterators" misses the big...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generators as Pipelines
• Stacked generators result in ...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Coroutines 101
• yield can receive a value instead
14
d...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Coroutines and Dataflow
15
• Coroutines enable dataflow s...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Fundamentals
• The yield statement defines a generator f...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advancing a Generator
• next(gen) - Advances to the nex...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Sending to a Generator
• gen.send(item) - Send an item ...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Closing a Generator
• gen.close() - Terminate a generat...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Raising Exceptions
• gen.throw(typ [, val [,tb]]) - Thr...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator ReturnValues
• StopIteration raised on genera...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Delegation
• yield from gen - Delegate to a s...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Delegation Example
• Chain iterables together
23
def ch...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Mini-Reference
• Generator instance operations
24
gen =...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 2
25
and now for something completely different
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Common Motif
• Consider the following
26
f = open()
....
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Managers
• The 'with' statement
27
with open(fi...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
• It's easy to make your own (@conte...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
29
• Usage
with timethis('counting')...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
• Another example: temporary directo...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Whoa,Whoa, Stop!
• Another example: temporary directori...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Management
• Under the covers
32
with obj:
stat...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Manager
• Implementation template
33
class Mana...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Context Manager Example
• Automatically deleted temp di...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Alternate Formulation
• @contextmanager is just a refor...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• How does it work?
36
@contextmanager
d...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• Each half maps to context manager meth...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• There is a wrapper class (Context Mana...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• enter - Run the generator to the yield...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Deconstruction
• exit - Resumes the generator
40
class ...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Full Disclosure
• Actual implementation is more complic...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Discussion
• Why start with this example?
• A completel...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 3
43
Call me, maybe
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 3
44
Call me, maybe
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Async Processing
• Consider the following execution mod...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example:Thread Pool
46
from concurrent.futures import T...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Futures
47
>>> fut = pool.submit(func, 2, 3)
>>> fut
<F...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Futures
48
def run():
fut = pool.submit(func, 2, 3)
fut...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exceptions
49
>>> fut = pool.submit(func, 2, 'Hello')
>...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Futures w/Errors
50
def run():
fut = pool.submit(func, ...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Interlude
51
def run():
fut = pool.submit(func, 2, 3)
f...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Callback Hell?
52
• No, no, no.... keep focusing.
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Interlude
53
def entry():
fut = pool.submit(func, 2, 3)...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Inlined Futures
54
@inlined_future
def do_func(x, y):
r...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
DéjàVu
55
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
DéjàVu
56
• This twisted idea has been used before...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Preview
57
t = Task(gen)
• There are two separate parts...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Commentary
58
• Will continue to use threads for exampl...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
59
def do_func(x, y):
result = yi...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
60
class Task:
def __init__(self,...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
61
class Task:
def __init__(self,...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
62
class Task:
def __init__(self,...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
63
class Task:
def __init__(self,...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running the Generator
64
class Task:
def __init__(self,...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Does it Work?
65
pool = ThreadPoolExecutor(max_workers=...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Does it Work?
66
pool = ThreadPoolExecutor(max_workers=...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exception Handling
67
class Task:
def __init__(self, ge...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exception Handling
68
class Task:
def __init__(self, ge...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Exception Handling
69
class Task:
def __init__(self, ge...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Error Example
70
def do_func(x, y):
try:
result = yield...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Commentary
71
• This whole thing is rather bizarre
• Ex...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Consider
72
def recursive(n):
yield pool.submit(time.sl...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 4
73
yield from yield from yield from yield from f...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Singular Focus
74
• Focus on the future
• Not the pas...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Singular Focus
75
class Task:
def __init__(self, gen)...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler
76
def after(delay, gen):
'''
Run an inlined fu...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler
77
def after(delay, gen):
'''
Run an inlined fu...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler
78
def after(delay, gen):
'''
Run an inlined fu...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (2nd Attempt)
79
def after(delay, gen):
'''
Run...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (2nd Attempt)
80
def after(delay, gen):
'''
Run...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (3rd Attempt)
81
def after(delay, gen):
yield p...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (3rd Attempt)
82
def after(delay, gen):
yield p...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (4th Attempt)
83
def after(delay, gen):
yield p...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
PEP 380
• yield from gen - Delegate to a subgenerator
8...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Conundrum
85
def after(delay, gen):
yield pool.submit(t...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (5th Attempt)
86
def after(delay, gen):
yield f...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Puzzler (5th Attempt)
87
def after(delay, gen):
yield f...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
??????
88
(Can it be made to work?)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Iterable Futures
89
def patch_future(cls):
def __iter__...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
All Roads Lead to Future
90
yield self
• Future is the ...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
The Decorator
91
import inspect
def inlined_future(func...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Task Wrangling
92
t = Task(gen)
t.step()
• The "Task" o...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tasks as Futures
93
class Task(Future):
def __init__(se...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tasks as Futures
94
class Task(Future):
def __init__(se...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
95
@inlined_future
def do_func(x, y):
result = ...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
96
@inlined_future
def do_func(x, y):
result = ...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Task Runners
97
def start_inline_future(fut):
t = Task(...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Step Back Slowly
98
• Built a generator-based task syst...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
asyncio
99
• Ideas are the foundation asyncio coroutine...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Simple Example
100
import asyncio
def func(x, y):
retur...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
101
import asyncio
@asyncio.coroutine
...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Be on the Lookout!
102
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 103
(source: globalpost.com)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 5
104
"Gil"
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Python Threads
105
• Threads, what are they good for?
•...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Actually...
106
• Threads are great at doing nothing!
t...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
CPU-Bound Work
107
• Threads are weak for computation
•...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Solution
108
• Naturally, we must reinvent the one th...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
CPU-Bound Work
109
• Event-loops have their own issues
...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Standard Solution
110
• Delegate the work out to a proc...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
111
• Didn't we just do this with in...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
112
• Can you launch tasks in parall...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
113
• Can you launch tasks in parall...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thought Experiment
114
• Can you launch tasks in parall...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Execution Model
115
• The way in which it works is a li...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Process Pools
116
• Process pools involve a hidden resu...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
The Issue
117
• Our inlined future switches execution t...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Important Lesson
118
• If you're going to play with con...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Insight
119
• The yield is not implementation
@inlined_...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Inlined Thread Execution
120
• Variant: Run generator e...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
New Execution
121
• Try it again with a thread pool (be...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
New Execution
122
• Output: (2 Threads)
<Thread(Thread-...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Big Idea
123
• You can mold and adapt generator executi...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 6
124
Fake it until you make it
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Actors
125
• There is a striking similarity between cor...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
126
• A very simple example
@actor
def printer(...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Attempt 1
127
_registry = { }
def send(name, msg):
_reg...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Example
128
• A very simple example
@actor
def printer(...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
129
• Recursive ping-pong (inspired by...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
130
ping 0
pong 1
Traceback (most rece...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Problems
131
• Important differences between actors/cor...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Solution 1
132
class Actor(threading.Thread):
def __ini...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Solution 2
133
_registry = { }
_msg_queue = deque()
def...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
134
• Recursive ping-pong (reprise)
@a...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Advanced Example
135
ping 0
pong 1
ping 2
pong 3
ping 4...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Comments
136
• It's still kind of a fake actor
• Lackin...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Part 7
137
A TerrifyingVisitor
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Let's Write a Compiler
138
• Well, an extremely simple ...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Compilers 101
139
• Lexing : Make tokens
2 + 3 * 4 - 5 ...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Compilers 101
140
• Evaluation :Walk the parse tree
PLU...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tokenizing
141
import re
from collections import namedt...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tokenizing
142
text = '2 + 3 * 4 - 5'
for tok in tokeni...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Parsing
143
• Must match the token stream against a gra...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Recursive Descent Parse
144
expr ::= term { +|- term }*...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Recursive Descent Parse
145
def parse(toks):
lookahead,...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tree Building
146
class Node:
_fields = []
def __init__...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Tree Building
147
def parse(toks):
...
def expr():
left...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Our Little Parser
148
text = '2 + 3*4 - 5'
toks = token...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
149
class NodeVisitor:
def visit(self, node)...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
150
class Evaluator(NodeVisitor):
def visit_...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Digression
151
• Last 12 slides a whole graduate CS cou...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Death Spiral
152
# Make '0+1+2+3+4+...+999'
text = '+'....
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
153
class Evaluator(NodeVisitor):
def visit_...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
154
+
0 + 1 + 2 + 3 + 4 ... + 999
0
+
+
+
+
...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 155
• The visitor pattern is bad idea
• Better: Functio...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
QUESTION
156
How do you NOT do something?
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
QUESTION
157
How do you NOT do something?
(yield?)
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Evaluation
158
class Evaluator(NodeVisitor):
def visit_...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
159
class NodeVisitor:
def genvisit(...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
160
>>> v = Evaluator()
>>> n = Numb...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
161
>>> v = Evaluator()
>>> n = BinO...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Generator Wrapping
162
>>> v = Evaluator()
>>> n = BinO...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
163
class NodeVisitor:
def visit(self...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Transcendence
164
# Make '0+1+2+3+4+...+999'
text = '+'...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
165
class Evaluator(NodeVisitor):
def...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
166
>>> v = Evaluator()
>>> n = BinOp...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Running Recursion
167
>>> v = Evaluator()
>>> n = BinOp...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 168
Final Words
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Historical Perspective
169
• Generators seem to have st...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Control Flow Bending
170
• yield statement allows you t...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
asyncio
171
• Inclusion of asyncio in standard library ...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Is it Proper?
172
• Are coroutines/generators a good id...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Two Libraries?
173
Python
Standard Library
Standard
cor...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Personal Use
174
• My own code is dreadfully boring
• G...
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
A Bit More Information
175
Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com
Thanks!
176
• I hope you got some new ideas
• Please fe...
Prochain SlideShare
Chargement dans…5
×

Generators: The Final Frontier

10 235 vues

Publié le

PyCon'2014 tutorial presentation.

Publié dans : Logiciels, Business, Technologie
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y6a5rkg5 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • DOWNLOAD FULL BOOKS, INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. PDF EBOOK here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. EPUB Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... 1.DOWNLOAD FULL. doc Ebook here { https://tinyurl.com/y3nhqquc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Generators: The Final Frontier

  1. 1. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generators: The Final Frontier David Beazley (@dabeaz) http://www.dabeaz.com Presented at PyCon'2014, Montreal 1
  2. 2. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Previously on Generators 2 • GeneratorTricks for Systems Programmers (2008) http://www.dabeaz.com/generators/ • A flying leap into generator awesomeness
  3. 3. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Previously on Generators 3 • A Curious Course on Coroutines and Concurrency (2009) http://www.dabeaz.com/coroutines/ • Wait, wait? There's more than iteration?
  4. 4. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Today's Installment 4 • Everything else you ever wanted to know about generators, but were afraid to try • Part 3 of a trilogy
  5. 5. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Requirements 5 • You need Python 3.4 or newer • No third party extensions • Code samples and notes http://www.dabeaz.com/finalgenerator/ • Follow along if you dare!
  6. 6. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Disclaimer 6 • This is an advanced tutorial • Assumes general awareness of • Core Python language features • Iterators/generators • Decorators • Common programming patterns • I learned a LOT preparing this
  7. 7. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Will I Be Lost? 7 • Although this is the third part of a series, it's mostly a stand-alone tutorial • If you've seen prior tutorials, that's great • If not, don't sweat it • Be aware that we're focused on a specific use of generators (you just won't get complete picture)
  8. 8. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Focus 8 • Material in this tutorial is probably not immediately applicable to your day job • More thought provoking and mind expanding • from __future__ import future practical utility bleeding edge
  9. 9. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part I 9 Preliminaries - Generators and Coroutines (rock)
  10. 10. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generators 101 • yield statement defines a generator function 10 def countdown(n): while n > 0: yield n n -= 1 • You typically use it to feed iteration for x in countdown(10): print('T-minus', x) • A simple, yet elegant idea
  11. 11. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Under the Covers 11 • Generator object runs in response to next() >>> c = countdown(3) >>> c <generator object countdown at 0x10064f900> >>> next(c) 3 >>> next(c) 2 >>> next(c) 1 >>> next(c) Traceback (most recent call last): File "<stdin>", line 1, in ? StopIteration >>> • StopIteration raised when function returns
  12. 12. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Interlude 12 • Generators as "iterators" misses the big picture • There is so much more to yield
  13. 13. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generators as Pipelines • Stacked generators result in processing pipelines • Similar to shell pipes in Unix 13 generator input sequence for x in s:generator generator def process(sequence): for s in sequence: ... do something ... yield item • Incredibly useful (see prior tutorial)
  14. 14. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Coroutines 101 • yield can receive a value instead 14 def receiver(): while True: item = yield print('Got', item) • It defines a generator that you send things to recv = receiver() next(recv) # Advance to first yield recv.send('Hello') recv.send('World')
  15. 15. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Coroutines and Dataflow 15 • Coroutines enable dataflow style processing source coroutine coroutine send() send() • Publish/subscribe, event simulation, etc. coroutine coroutine send() send() coroutine send()
  16. 16. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Fundamentals • The yield statement defines a generator function 16 def generator(): ... ... yield ... ... • The mere presence of yield anywhere is enough • Calling the function creates a generator instance >>> g = generator() >>> g <generator object generator at 0x10064f120> >>>
  17. 17. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advancing a Generator • next(gen) - Advances to the next yield 17 def generator(): ... ... yield item ... • Returns the yielded item (if any) • It's the only allowed operation on a newly created generator • Note: Same as gen.__next__()
  18. 18. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Sending to a Generator • gen.send(item) - Send an item to a generator 18 def generator(): ... item = yield ... ... yield value • Wakes at last yield, returns sent value • Runs to the next yield and emits the value g = generator() next(g) # Advance to yield value = g.send(item)
  19. 19. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Closing a Generator • gen.close() - Terminate a generator 19 def generator(): ... try: yield except GeneratorExit: # Shutting down ... • Raises GeneratorExit at the yield • Only allowed action is to return • If uncaught, generator silently terminates g = generator() next(g) # Advance to yield g.close() # Terminate
  20. 20. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Raising Exceptions • gen.throw(typ [, val [,tb]]) - Throw exception 20 def generator(): ... try: yield except RuntimeError as e: ... ... yield val • Raises exception at yield • Returns the next yielded value (if any) g = generator() next(g) # Advance to yield val = g.throw(RuntimeError, 'Broken')
  21. 21. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator ReturnValues • StopIteration raised on generator exit 21 def generator(): ... yield ... return result • Return value (if any) passed with exception • Note: Python 3 only behavior (in Python 2, generators can't return values) g = generator() try: next(g) except StopIteration as e: result = e.value
  22. 22. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Delegation • yield from gen - Delegate to a subgenerator 22 def generator(): ... yield value ... return result def func(): result = yield from generator() • Allows generators to call other generators • Operations take place at the current yield • Return value (if any) is returned
  23. 23. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Delegation Example • Chain iterables together 23 def chain(x, y): yield from x yield from y • Example: >>> a = [1, 2, 3] >>> b = [4, 5, 6] >>> for x in chain(a, b): ... print(x,end=' ') ... 1 2 3 4 5 6 >>> c = [7,8,9] >>> for x in chain(a, chain(b, c)): ... print(x, end=' ') ... 1 2 3 4 5 6 7 8 9 >>>
  24. 24. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Mini-Reference • Generator instance operations 24 gen = generator() next(gen) # Advance to next yield gen.send(item) # Send an item gen.close() # Terminate gen.throw(exc, val, tb) # Raise exception result = yield from gen # Delegate • Using these, you can do a lot of neat stuff • Generator definition def generator(): ... yield ... return result
  25. 25. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 2 25 and now for something completely different
  26. 26. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Common Motif • Consider the following 26 f = open() ... f.close() lock.acquire() ... lock.release() db.start_transaction() ... db.commit() start = time.time() ... end = time.time() • It's so common, you'll see it everywhere!
  27. 27. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Managers • The 'with' statement 27 with open(filename) as f: statement statement ... with lock: statement statement ... • Allows control over entry/exit of a code block • Typical use: everything on the previous slide
  28. 28. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Management • It's easy to make your own (@contextmanager) 28 import time from contextlib import contextmanager @contextmanager def timethis(label): start = time.time() try: yield finally: end = time.time() print('%s: %0.3f' % (label, end-start) • This times a block of statements
  29. 29. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Management 29 • Usage with timethis('counting'): n = 1000000 while n > 0: n -= 1 • Output counting: 0.023
  30. 30. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Management • Another example: temporary directories 30 import tempfile, shutil from contextlib import contextmanager @contextmanager def tempdir(): outdir = tempfile.mkdtemp() try: yield outdir finally: shutil.rmtree(outdir) • Example with tempdir() as dirname: ...
  31. 31. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Whoa,Whoa, Stop! • Another example: temporary directories 31 import tempfile, shutil from contextlib import contextmanager @contextmanager def tempdir(): outdir = tempfile.mkdtemp() try: yield outdir finally: shutil.rmtree(outdir) • Example with tempdir() as dirname: ... What is this? • Not iteration • Not dataflow • Not concurrency • ????
  32. 32. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Management • Under the covers 32 with obj: statements statements statements ... statements • If an object implements these methods it can monitor entry/exit to the code block obj.__enter__() obj.__exit__()
  33. 33. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Manager • Implementation template 33 class Manager(object): def __enter__(self): return value def __exit__(self, exc_type, val, tb): if exc_type is None: return else: # Handle an exception (if you want) return True if handled else False • Use: with Manager() as value: statements statements
  34. 34. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Manager Example • Automatically deleted temp directories 34 import tempfile import shutil class tempdir(object): def __enter__(self): self.dirname = tempfile.mkdtemp() return self.dirname def __exit__(self, exc, val, tb): shutil.rmtree(self.dirname) • Use: with tempdir() as dirname: ...
  35. 35. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Alternate Formulation • @contextmanager is just a reformulation 35 import tempfile, shutil from contextlib import contextmanager @contextmanager def tempdir(): dirname = tempfile.mkdtemp() try: yield dirname finally: shutil.rmtree(dirname) • It's the same code, glued together differently
  36. 36. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • How does it work? 36 @contextmanager def tempdir(): dirname = tempfile.mkdtemp() try: yield dirname finally: shutil.rmtree(dirname) • Think of "yield" as scissors • Cuts the function in half
  37. 37. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • Each half maps to context manager methods 37 @contextmanager def tempdir(): dirname = tempfile.mkdtemp() try: yield dirname statements statements statements ... finally: shutil.rmtree(dirname) __enter__ __exit__ user statements ('with' block) • yield is the magic that makes it possible
  38. 38. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • There is a wrapper class (Context Manager) 38 class GeneratorCM(object): def __init__(self, gen): self.gen = gen def __enter__(self): ... def __exit__(self, exc, val, tb): ... • And a decorator def contextmanager(func): def run(*args, **kwargs): return GeneratorCM(func(*args, **kwargs)) return run
  39. 39. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • enter - Run the generator to the yield 39 class GeneratorCM(object): def __init__(self, gen): self.gen = gen def __enter__(self): return next(self.gen) def __exit__(self, exc, val, tb): ... • It runs a single "iteration" step • Returns the yielded value (if any)
  40. 40. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • exit - Resumes the generator 40 class GeneratorCM(object): ... def __exit__(self, etype, val, tb): try: if etype is None: next(self.gen) else: self.gen.throw(etype, val, tb) raise RuntimeError("Generator didn't stop") except StopIteration: return True except: if sys.exc_info()[1] is not val: raise • Either resumes it normally or raises exception
  41. 41. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Full Disclosure • Actual implementation is more complicated • There are some nasty corner cases • Exceptions with no associated value • StopIteration raised inside a with-block • Exceptions raised in context manager • Read source and see PEP-343 41
  42. 42. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Discussion • Why start with this example? • A completely different use of yield • Being used to reformulate control-flow • It simplifies programming for others (easy definition of context managers) • Maybe there's more... (of course there is) 42
  43. 43. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 3 43 Call me, maybe
  44. 44. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 3 44 Call me, maybe
  45. 45. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Async Processing • Consider the following execution model 45 def func(args): ... ... ... return result run_async(func, args) main thread • Examples: Run in separate process or thread, time delay, in response to event, etc.
  46. 46. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example:Thread Pool 46 from concurrent.futures import ThreadPoolExecutor def func(x, y): 'Some function. Nothing too interesting' import time time.sleep(5) return x + y pool = ThreadPoolExecutor(max_workers=8) fut = pool.submit(func, 2, 3) r = fut.result() print('Got:', r) • Runs the function in a separate thread • Waits for a result
  47. 47. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Futures 47 >>> fut = pool.submit(func, 2, 3) >>> fut <Future at 0x1011e6cf8 state=running> >>> • Future - A result to be computed later • You can wait for the result to return >>> fut.result() 5 >>> • However, this blocks the caller
  48. 48. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Futures 48 def run(): fut = pool.submit(func, 2, 3) fut.add_done_callback(result_handler) def result_handler(fut): result = fut.result() print('Got:', result) • Alternatively, you can register a callback • Triggered upon completion
  49. 49. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Exceptions 49 >>> fut = pool.submit(func, 2, 'Hello') >>> fut.result() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 395, in result return self.__get_result() File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 354, in __get_result raise self._exception File "/usr/local/lib/python3.4/concurrent/futures/thread.py", line 54, in run result = self.fn(*self.args, **self.kwargs) File "future2.py", line 6, in func return x + y TypeError: unsupported operand type(s) for +: 'int' and 'str' >>>
  50. 50. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Futures w/Errors 50 def run(): fut = pool.submit(func, 2, 3) fut.add_done_callback(result_handler) def result_handler(fut): try: result = fut.result() print('Got:', result) except Exception as e: print('Failed: %s: %s' % (type(e).__name__, e)) • Error handling with callbacks • Exception propagates out of fut.result() method
  51. 51. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Interlude 51 def run(): fut = pool.submit(func, 2, 3) fut.add_done_callback(result_handler) def result_handler(fut): try: result = fut.result() print('Got:', result) except Exception as e: print('Failed: %s: %s' % (type(e).__name__, e)) • Consider the structure of code using futures • Meditate on it... focus on the code. • This seems sort of familiar
  52. 52. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Callback Hell? 52 • No, no, no.... keep focusing.
  53. 53. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Interlude 53 def entry(): fut = pool.submit(func, 2, 3) fut.add_done_callback(exit) def exit(fut): try: result = fut.result() print('Got:', result) except Exception as e: print('Failed: %s: %s' % (type(e).__name__, e)) • What if the function names are changed? • Wait! This is almost a context manager (yes)
  54. 54. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Inlined Futures 54 @inlined_future def do_func(x, y): result = yield pool.submit(func, x, y) print('Got:', result) run_inline_future(do_func) • Thought: Maybe you could do that yield trick • The extra callback function is eliminated • Now, just one "simple" function • Inspired by @contextmanager
  55. 55. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com DéjàVu 55
  56. 56. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com DéjàVu 56 • This twisted idea has been used before...
  57. 57. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Preview 57 t = Task(gen) • There are two separate parts • Part 1:Wrapping generators with a "task" • Part 2: Implementing some runtime code run_inline_future(gen) • Forewarning: It will bend your mind a bit
  58. 58. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Commentary 58 • Will continue to use threads for examples • Mainly because they're easy to work with • And I don't want to get sucked into an event loop • Don't dwell on it too much • Key thing: There is some background processing
  59. 59. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator 59 def do_func(x, y): result = yield pool.submit(func, x, y) print('Got:', result) • Problem: Stepping through a generator • Involves gluing callbacks and yields together def do_func(x, y): yield pool.submit(func, x, y) result = print('Got:', result) add_done_callback() cut enter exit
  60. 60. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator 60 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result)
  61. 61. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator 61 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result) Task class wraps around and represents a running generator.
  62. 62. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator 62 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result) Advance the generator to the next yield, sending in a value (if any)
  63. 63. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator 63 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result) Attach a callback to the produced Future
  64. 64. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator 64 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result) Collect result and send back into the generator
  65. 65. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Does it Work? 65 pool = ThreadPoolExecutor(max_workers=8) def func(x, y): time.sleep(1) return x + y def do_func(x, y): result = yield pool.submit(func, x, y) print('Got:', result) t = Task(do_func(2, 3)) t.step() • Try it: • Output: Got: 5 • Yes, it works Note: must initiate first step of the task to get it to run
  66. 66. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Does it Work? 66 pool = ThreadPoolExecutor(max_workers=8) def func(x, y): time.sleep(1) return x + y def do_many(n): while n > 0: result = yield pool.submit(func, n, n) print('Got:', result) n -= 1 t = Task(do_many(10)) t.step() • More advanced: multiple yields/looping • Yes, this works too.
  67. 67. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Exception Handling 67 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): try: result = fut.result() self.step(result, None) except Exception as exc: self.step(None, exc)
  68. 68. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Exception Handling 68 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): try: result = fut.result() self.step(result, None) except Exception as exc: self.step(None, exc) send() or throw() depending on success
  69. 69. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Exception Handling 69 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): try: result = fut.result() self.step(result, None) except Exception as exc: self.step(None, exc) Catch exceptions and pass to next step as appropriate
  70. 70. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Error Example 70 def do_func(x, y): try: result = yield pool.submit(func, x, y) print('Got:', result) except Exception as e: print('Failed:', repr(e)) t = Task(do_func(2, 'Hello')) t.step() • Try it: • Output: Failed: TypeError("unsupported operand type(s) for +: 'int' and 'str'",) • Yep, that works too.
  71. 71. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Commentary 71 • This whole thing is rather bizarre • Execution of the inlined future takes place all on its own (concurrently with other code) • The normal rules don't apply
  72. 72. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Consider 72 def recursive(n): yield pool.submit(time.sleep, 0.001) print('Tick:', n) Task(recursive(n+1)).step() Task(recursive(0)).step() • Infinite recursion? • Output: Tick: 0 Tick: 1 Tick: 2 ... Tick: 1662773 Tick: 1662774 ...
  73. 73. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 4 73 yield from yield from yield from yield from future (maybe) source: @UrsulaWJ
  74. 74. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Singular Focus 74 • Focus on the future • Not the past • Not now • Yes, the future. • No, really, the future. (but not the singularity)
  75. 75. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Singular Focus 75 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass ... generator must only produce Futures
  76. 76. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler 76 def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) yield gen Task(after(10, do_func(2, 3))).step() • Can you make library functions? • It's trying to delay the execution of a user- supplied inlined future until later.
  77. 77. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler 77 def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) yield gen Task(after(10, do_func(2, 3))).step() • Can you make library functions? • No Traceback (most recent call last): ... AttributeError: 'generator' object has no attribute 'add_done_callback'
  78. 78. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler 78 def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) yield gen Task(after(10, do_func(2, 3))).step() • Can you make library functions? • This is busted • gen is a generator, not a Future
  79. 79. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (2nd Attempt) 79 def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) for f in gen: yield f Task(after(10, do_func(2, 3))).step() • What about this? • Idea: Just iterate the generator manually • Make it produce the required Futures
  80. 80. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (2nd Attempt) 80 def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) for f in gen: yield f Task(after(10, do_func(2, 3))).step() • What about this? • No luck.The result gets lost somewhere Got: None • Hmmm.
  81. 81. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (3rd Attempt) 81 def after(delay, gen): yield pool.submit(time.sleep, delay) result = None try: while True: f = gen.send(result) result = yield f except StopIteration: pass Task(after(10, do_func(2, 3))).step() • Obvious solution (duh!) • Hey, it works! Got: 5
  82. 82. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (3rd Attempt) 82 def after(delay, gen): yield pool.submit(time.sleep, delay) result = None try: while True: f = gen.send(result) result = yield f except StopIteration: pass Task(after(10, do_func(2, 3))).step() • Obvious solution (duh!) • Hey, it works! Got: 5 manual running of generator with results (ugh!)
  83. 83. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (4th Attempt) 83 def after(delay, gen): yield pool.submit(time.sleep, delay) yield from gen Task(after(10, do_func(2, 3))).step() • A better solution: yield from • 'yield from' - Runs the generator for you Got: 5 • And it works! (yay!) • Awesome
  84. 84. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com PEP 380 • yield from gen - Delegate to a subgenerator 84 def generator(): ... yield value ... return result def func(): result = yield from generator() • Transfer control to other generators • Operations take place at the current yield • Far more powerful than you might think
  85. 85. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Conundrum 85 def after(delay, gen): yield pool.submit(time.sleep, delay) yield from gen • "yield" and "yield from"? • Two different yields in the same function • Nobody will find that confusing (NOT!)
  86. 86. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (5th Attempt) 86 def after(delay, gen): yield from pool.submit(time.sleep, delay) yield from gen Task(after(10, do_func(2, 3))).step() • Maybe this will work? • Just use 'yield from'- always!
  87. 87. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (5th Attempt) 87 def after(delay, gen): yield from pool.submit(time.sleep, delay) yield from gen Task(after(10, do_func(2, 3))).step() • Maybe this will work? • Just use 'yield from'- always! • No. 'yield' and 'yield from' not interchangeable: Traceback (most recent call last): ... TypeError: 'Future' object is not iterable >>>
  88. 88. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com ?????? 88 (Can it be made to work?)
  89. 89. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Iterable Futures 89 def patch_future(cls): def __iter__(self): ! if not self.done(): yield self return self.result() cls.__iter__ = __iter__ from concurrent.futures import Future patch_future(Future) • A simple ingenious patch • It makes all Future instances iterable • They simply produce themselves and the result • It magically makes 'yield from' work!
  90. 90. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com All Roads Lead to Future 90 yield self • Future is the only thing that actually yields • Everything else delegates using 'yield from' • Future terminates the chain generator generator generator Future yield from yield from yield from run() next(gen)
  91. 91. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com The Decorator 91 import inspect def inlined_future(func): assert inspect.isgeneratorfunction(func) return func • Generators yielding futures is its own world • Probably a good idea to have some demarcation • Does nothing much at all, but serves as syntax @inlined_future def after(delay, gen): yield from pool.submit(time.sleep, delay) yield from gen • Alerts others about what you're doing
  92. 92. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Task Wrangling 92 t = Task(gen) t.step() • The "Task" object is just weird • No way to obtain a result • No way to join with it • Or do much of anything useful at all Task runs ???? t.step()
  93. 93. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tasks as Futures 93 class Task(Future): def __init__(self, gen): !super().__init__() ! self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: self.set_result(exc.value) • This tiny tweak makes it much more interesting
  94. 94. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tasks as Futures 94 class Task(Future): def __init__(self, gen): !super().__init__() ! self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: self.set_result(exc.value) • This tiny tweak makes it much more interesting A Task is a Future Set its result upon completion
  95. 95. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example 95 @inlined_future def do_func(x, y): result = yield pool.submit(func, x, y) return result t = Task(do_func(2,3)) t.step() ... print("Got:", t.result()) • Obtaining the result of task • So, you create a task that runs a generator producing Futures • The task is also a Future • Right. Got it.
  96. 96. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example 96 @inlined_future def do_func(x, y): result = yield pool.submit(func, x, y) return result t = Task(do_func(2,3)) t.step() ... print("Got:", t.result()) class Task(Future): ... def step(self, value=None, exc=None): try: ... except StopIteration as exc: self.set_result(exc.value) ...
  97. 97. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Task Runners 97 def start_inline_future(fut): t = Task(fut) t.step() return t def run_inline_future(fut): t = start_inline_future(fut) return t.result() • You can make utility functions to hide details result = run_inline_future(do_func(2,3)) print('Got:', result) • Example: Run an inline future to completion • Example: Run inline futures in parallel t1 = start_inline_future(do_func(2, 3)) t2 = start_inline_future(do_func(4, 5)) result1 = t1.result() result2 = t2.result()
  98. 98. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Step Back Slowly 98 • Built a generator-based task system for threads Tasks @inline_future run_inline_future() Threads pools concurrent.future Futures submit result • Execution of the future hidden in background • Note: that was on purpose (for now)
  99. 99. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com asyncio 99 • Ideas are the foundation asyncio coroutines Tasks @coroutine run_until_complete() Event Loop asyncio Futures result • In fact, it's almost exactly the same • Naturally, there are some details with event loop
  100. 100. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Simple Example 100 import asyncio def func(x, y): return x + y @asyncio.coroutine def do_func(x, y): yield from asyncio.sleep(1) return func(x, y) loop = asyncio.get_event_loop() result = loop.run_until_complete(do_func(2,3)) print("Got:", result) • asyncio "hello world"
  101. 101. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 101 import asyncio @asyncio.coroutine def echo_client(reader, writer): while True: line = yield from reader.readline() if not line: break resp = b'Got:' + line writer.write(resp) writer.close() loop = asyncio.get_event_loop() loop.run_until_complete( asyncio.start_server(echo_client, host='', port=25000) ) loop.run_forever() • asyncio - Echo Server
  102. 102. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Be on the Lookout! 102
  103. 103. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 103 (source: globalpost.com)
  104. 104. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 5 104 "Gil"
  105. 105. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Python Threads 105 • Threads, what are they good for? • Answer: Nothing, that's what! • Damn you GIL!!
  106. 106. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Actually... 106 • Threads are great at doing nothing! time.sleep(2) # Do nothing for awhile data = sock.recv(nbytes) # Wait around for data data = f.read(nbytes) • In fact, great for I/O! • Mostly just sitting around
  107. 107. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com CPU-Bound Work 107 • Threads are weak for computation • Global interpreter lock only allows 1 CPU • Multiple CPU-bound threads fight each other • Could be better http://www.dabeaz.com/GIL
  108. 108. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Solution 108 • Naturally, we must reinvent the one thing that threads are good at • Namely, waiting around. • Event-loops, async, coroutines, green threads. • Think about it: These are focused on I/O (yes, I know there are other potential issues with threads, but work with me here)
  109. 109. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com CPU-Bound Work 109 • Event-loops have their own issues • Don't bug me, I'm blocking right now (source: chicagotribune.com)
  110. 110. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Standard Solution 110 • Delegate the work out to a process pool python python python python python main program CPU CPU CPU CPU • multiprocessing, concurrent.futures, etc.
  111. 111. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thought Experiment 111 • Didn't we just do this with inlined futures? def fib(n): return 1 if n <= 2 else (fib(n-1) + fib(n-2)) @inlined_future def compute_fibs(n): result = [] for i in range(n): val = yield from pool.submit(fib, i) result.append(val) return result pool = ProcessPoolExecutor(4) result = run_inline_future(compute_fibs(35)) • It runs without crashing (let's ship it!)
  112. 112. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thought Experiment 112 • Can you launch tasks in parallel? t1 = start_inline_future(compute_fibs(34)) t2 = start_inline_future(compute_fibs(34)) result1 = t1.result() result2 = t2.result() • Sequential execution run_inline_future(compute_fibs(34)) run_inline_future(compute_fibs(34)) • Recall (from earlier) def start_inline_future(fut): t = Task(fut) t.step() return t
  113. 113. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thought Experiment 113 • Can you launch tasks in parallel? t1 = start_inline_future(compute_fibs(34)) t2 = start_inline_future(compute_fibs(34)) result1 = t1.result() result2 = t2.result() • Sequential execution run_inline_future(compute_fibs(34)) run_inline_future(compute_fibs(34)) • Recall (from earlier) def start_inline_future(fut): t = Task(fut) t.step() return t 9.56s
  114. 114. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thought Experiment 114 • Can you launch tasks in parallel? t1 = start_inline_future(compute_fibs(34)) t2 = start_inline_future(compute_fibs(34)) result1 = t1.result() result2 = t2.result() • Sequential execution run_inline_future(compute_fibs(34)) run_inline_future(compute_fibs(34)) 9.56s • Recall (from earlier) def start_inline_future(fut): t = Task(fut) t.step() return t 4.78s Inlined tasks running outside confines of the GIL?
  115. 115. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Execution Model 115 • The way in which it works is a little odd @inlined_future def compute_fibs(n): result = [] for i in range(n): print(threading.current_thread()) val = yield from pool.submit(fib, i) result.append(val) return result add this • Output: (2 Tasks) <_MainThread(MainThread, started 140735086636224)> <_MainThread(MainThread, started 140735086636224)> <Thread(Thread-1, started daemon 4320137216)> <Thread(Thread-1, started daemon 4320137216)> <Thread(Thread-1, started daemon 4320137216)> ... ????
  116. 116. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Process Pools 116 • Process pools involve a hidden result thread main thread python python CPU CPU submit result_thread • result thread reads returned values • Sets the result on the associated Future • Triggers the callback function (if any) results
  117. 117. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com The Issue 117 • Our inlined future switches execution threads @inlined_future def compute_fibs(n): result = [] for i in range(n): val = yield from pool.submit(fib, i) result.append(val) return result main thread result thread • Switch occurs at the first yield • All future execution occurs in result thread • That could be a little weird (especially if it blocked)
  118. 118. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Important Lesson 118 • If you're going to play with control flow, you must absolutely understand possible implications under the covers (i.e., switching threads across the yield statement).
  119. 119. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Insight 119 • The yield is not implementation @inlined_future def compute_fibs(n): result = [] for i in range(n): val = yield from pool.submit(fib, i) result.append(val) return result • You can implement different execution models • You don't have to follow a formulaic rule
  120. 120. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Inlined Thread Execution 120 • Variant: Run generator entirely in a single thread def run_inline_thread(gen): value = None exc = None while True: try: if exc: fut = gen.throw(exc) else: fut = gen.send(value) try: value = fut.result() exc = None except Exception as e: exc = e except StopIteration as exc: return exc.value • It just steps through... no callback function
  121. 121. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com New Execution 121 • Try it again with a thread pool (because why not?) @inlined_future def compute_fibs(n): result = [] for i in range(n): print(threading.current_thread()) val = yield from pool.submit(fib, i) result.append(val) return result tpool = ThreadPoolExecutor(8) t1 = tpool.submit(run_inline_thread(compute_fibs(34))) t2 = tpool.submit(run_inline_thread(compute_fibs(34))) result1 = t1.result() result2 = t2.result()
  122. 122. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com New Execution 122 • Output: (2 Threads) <Thread(Thread-1, started 4319916032)> <Thread(Thread-2, started 4326428672)> <Thread(Thread-1, started 4319916032)> <Thread(Thread-2, started 4326428672)> <Thread(Thread-1, started 4319916032)> <Thread(Thread-2, started 4326428672)> ... (works perfectly) 4.60s (a bit faster) • Processes, threads, and futures in perfect harmony • Uh... let's move along. Faster. Must go faster.
  123. 123. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Big Idea 123 • You can mold and adapt generator execution • That yield statement: magic!
  124. 124. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 6 124 Fake it until you make it
  125. 125. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Actors 125 • There is a striking similarity between coroutines and actors (i.e., the "actor" model) • Features of Actors • Receive messages • Send messages to other actors • Create new actors • No shared state (messages only) • Can coroutines serve as actors?
  126. 126. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example 126 • A very simple example @actor def printer(): while True: msg = yield print('printer:', msg) printer() n = 10 while n > 0: send('printer', n) n -=1 idea: use generators to define a kind of "named" actor task
  127. 127. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Attempt 1 127 _registry = { } def send(name, msg): _registry[name].send(msg) def actor(func): def wrapper(*args, **kwargs): gen = func(*args, **kwargs) next(gen) _registry[func.__name__] = gen return wrapper • Make a central coroutine registry and a decorator • Let's see if it works...
  128. 128. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example 128 • A very simple example @actor def printer(): while True: msg = yield print('printer:', msg) printer() n = 10 while n > 0: send('printer', n) n -=1 • It seems to work (maybe) printer: 10 printer: 9 printer: 8 ... printer: 1
  129. 129. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 129 • Recursive ping-pong (inspired by Stackless) @actor def ping(): while True: n = yield print('ping %d' % n) send('pong', n + 1) @actor def pong(): while True: n = yield print('pong %d' % n) send('ping', n + 1) ping() pong() send('ping', 0) ping pong
  130. 130. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 130 ping 0 pong 1 Traceback (most recent call last): File "actor.py", line 36, in <module> send('ping', 0) File "actor.py", line 8, in send _registry[name].send(msg) File "actor.py", line 24, in ping send('pong', n + 1) File "actor.py", line 8, in send _registry[name].send(msg) File "actor.py", line 31, in pong send('ping', n + 1) File "actor.py", line 8, in send _registry[name].send(msg) ValueError: generator already executing • Alas, it does not work
  131. 131. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Problems 131 • Important differences between actors/coroutines • Concurrent execution • Asynchronous message delivery • Although coroutines have a "send()", it's a normal method call • Synchronous • Involves the call-stack • Does not allow recursion/reentrancy
  132. 132. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Solution 1 132 class Actor(threading.Thread): def __init__(self, gen): super().__init__() self.daemon = True self.gen = gen self.mailbox = Queue() self.start() def send(self, msg): self.mailbox.put(msg) def run(self): while True: msg = self.mailbox.get() self.gen.send(msg) • Wrap the generator with a thread • Err...... no.
  133. 133. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Solution 2 133 _registry = { } _msg_queue = deque() def send(name, msg): _msg_queue.append((name, msg)) def run(): while _msg_queue: name, msg = _msg_queue.popleft() _registry[name].send(msg) • Write a tiny message scheduler • send() simply drops messages on a queue • run() executes as long as there are messages
  134. 134. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 134 • Recursive ping-pong (reprise) @actor def ping(): while True: n = yield print('ping %d' % n) send('pong', n + 1) @actor def pong(): while True: n = yield print('pong %d' % n) send('ping', n + 1) ping() pong() send('ping', 0) run() ping pong
  135. 135. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 135 ping 0 pong 1 ping 2 pong 3 ping 4 ping 5 ping 6 pong 7 ... ... forever • It works! • That's kind of amazing
  136. 136. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Comments 136 • It's still kind of a fake actor • Lacking in true concurrency • Easily blocked • Maybe it's good enough? • I don't know • Key idea: you can bend space-time with yield
  137. 137. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 7 137 A TerrifyingVisitor
  138. 138. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Let's Write a Compiler 138 • Well, an extremely simple one anyways... • Evaluating mathematical expressions 2 + 3 * 4 - 5 • Why? • Because eval() is for the weak, that's why
  139. 139. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Compilers 101 139 • Lexing : Make tokens 2 + 3 * 4 - 5 [NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM] • Parsing : Make a parse tree [NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM] PLUS NUM (2) TIMES NUM (3) NUM (4) MINUS NUM (5)
  140. 140. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Compilers 101 140 • Evaluation :Walk the parse tree PLUS NUM (2) TIMES NUM (3) NUM (4) MINUS NUM (5) 9 • It's almost too simple
  141. 141. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tokenizing 141 import re from collections import namedtuple tokens = [ r'(?P<NUM>d+)', r'(?P<PLUS>+)', r'(?P<MINUS>-)', r'(?P<TIMES>*)', r'(?P<DIVIDE>/)', r'(?P<WS>s+)', ] master_re = re.compile('|'.join(tokens)) Token = namedtuple('Token', ['type','value']) def tokenize(text): scan = master_re.scanner(text) return (Token(m.lastgroup, m.group()) for m in iter(scan.match, None) if m.lastgroup != 'WS')
  142. 142. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tokenizing 142 text = '2 + 3 * 4 - 5' for tok in tokenize(text): print(tok) Token(type='NUM', value='2') Token(type='PLUS', value='+') Token(type='NUM', value='3') Token(type='TIMES', value='*') Token(type='NUM', value='4') Token(type='MINUS', value='-') Token(type='NUM', value='5') • Example:
  143. 143. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Parsing 143 • Must match the token stream against a grammar expr ::= term { +|- term }* term ::= factor { *|/ factor}* factor ::= NUM • An expression is just a bunch of terms 2 + 3 * 4 - 5 term + term - term • A term is just one or more factors term term factor factor * factor (2) (3) (4)
  144. 144. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Recursive Descent Parse 144 expr ::= term { +|- term }* term ::= factor { *|/ factor}* factor ::= NUM def expr(): ! term() ! while accept('PLUS','MINUS'): term() !print('Matched expr') def term(): ! factor() while accept('TIMES','DIVIDE'): factor() print('Matched term') def factor(): if accept('NUM'): print('Matched factor') else: raise SyntaxError() Encode the grammar as a collection of functions Each function steps through the rule
  145. 145. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Recursive Descent Parse 145 def parse(toks): lookahead, current = next(toks, None), None def accept(*toktypes): nonlocal lookahead, current if lookahead and lookahead.type in toktypes: current, lookahead = lookahead, next(toks, None) return True def expr(): term() while accept('PLUS','MINUS'): term() print('Matched expr') ... expr()
  146. 146. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tree Building 146 class Node: _fields = [] def __init__(self, *args): for name, value in zip(self._fields, args): setattr(self, name, value) class BinOp(Node): _fields = ['op', 'left', 'right'] class Number(Node): _fields = ['value'] • Need some tree nodes for different things • Example: n1 = Number(3) n2 = Number(4) n3 = BinOp('*', n1, n2)
  147. 147. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tree Building 147 def parse(toks): ... def expr(): left = term() while accept('PLUS','MINUS'): left = BinOp(current.value, left) left.right = term() return left def term(): left = factor() while accept('TIMES','DIVIDE'): left = BinOp(current.value, left) left.right = factor() return left def factor(): if accept('NUM'): return Number(int(current.value)) else: raise SyntaxError() return expr() Building nodes and hooking them together
  148. 148. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Our Little Parser 148 text = '2 + 3*4 - 5' toks = tokenize(text) tree = parse(toks) • Story so far... BinOp('-', BinOp('+', Number(2), BinOp('*', Number(3), Number(4) ) ), Number(5) )
  149. 149. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 149 class NodeVisitor: def visit(self, node): return getattr(self, 'visit_' + type(node).__name__)(node) • The "Visitor" pattern • Example: class MyVisitor(NodeVisitor): def visit_Number(self, node): print(node.value) def visit_BinOp(self, node): self.visit(node.left) self.visit(node.right) print(node.op) MyVisitor().visit(tree) 2 3 4 * + 5 - output
  150. 150. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 150 class Evaluator(NodeVisitor): def visit_Number(self, node): ! return node.value def visit_BinOp(self, node): leftval = self.visit(node.left) rightval = self.visit(node.right) if node.op == '+': return leftval + rightval elif node.op == '-': return leftval - rightval elif node.op == '*': return leftval * rightval elif node.op == '/': return leftval / rightval print(Evaluator().visit(tree)) • An Expression Evaluator 9
  151. 151. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Digression 151 • Last 12 slides a whole graduate CS course • Plus at least one additional Python tutorial • Don't worry about it • Left as an exercise...
  152. 152. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Death Spiral 152 # Make '0+1+2+3+4+...+999' text = '+'.join(str(x) for x in range(1000)) toks = tokenize(text) tree = parse(toks) val = Evaluate().visit(tree) • And it almost works... Traceback (most recent call last): File "compiler.py", line 100, in <module> val = Evaluator().visit(tree) File "compiler.py", line 63, in visit return getattr(self, 'visit_' + type(node).__name__)(node) File "compiler.py", line 80, in visit_BinOp leftval = self.visit(node.left) ... RuntimeError: maximum recursion depth exceeded while calling a Python object
  153. 153. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 153 class Evaluator(NodeVisitor): def visit_Number(self, node): ! return node.value def visit_BinOp(self, node): leftval = self.visit(node.left) rightval = self.visit(node.right) if node.op == '+': return leftval + rightval elif node.op == '-': return leftval - rightval elif node.op == '*': return leftval * rightval elif node.op == '/': return leftval / rightval print(Evaluator().visit(tree)) • An Expression Evaluator !%*@*^#^# Recursion (damn you to hell)
  154. 154. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 154 + 0 + 1 + 2 + 3 + 4 ... + 999 0 + + + + 2 3 998 999 1 ... A deeply nested tree structure Blows up!
  155. 155. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 155 • The visitor pattern is bad idea • Better: Functional language with pattern matching and tail-call optimization I ToldYou So
  156. 156. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com QUESTION 156 How do you NOT do something?
  157. 157. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com QUESTION 157 How do you NOT do something? (yield?)
  158. 158. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 158 class Evaluator(NodeVisitor): def visit_Number(self, node): ! return node.value def visit_BinOp(self, node): leftval = yield node.left rightval = yield node.right if node.op == '+': return leftval + rightval elif node.op == '-': return leftval - rightval elif node.op == '*': return leftval * rightval elif node.op == '/': return leftval / rightval print(Evaluator().visit(tree)) • An Expression Evaluator Nope. Not doing that recursion.
  159. 159. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Wrapping 159 class NodeVisitor: def genvisit(self, node): result = getattr(self, 'visit_' + type(node).__name__)(node) if isinstance(result, types.GeneratorType): result = yield from result return result • Step 1:Wrap "visiting" with a generator • Thinking: No matter what the visit_() method produces, the result will be a generator • If already a generator, then just delegate to it
  160. 160. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Wrapping 160 >>> v = Evaluator() >>> n = Number(2) >>> gen = v.genvisit(n) >>> gen <generator object genvisit at 0x10070ab88> >>> gen.send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 2 >>> • Example:A method that simply returns a value • Result: Carried as value in StopIteration
  161. 161. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Wrapping 161 >>> v = Evaluator() >>> n = BinOp('*', Number(3), Number(4)) >>> gen = v.genvisit(n) >>> gen <generator object genvisit at 0x10070ab88> >>> gen.send(None) <__main__.Number object at 0x1058525c0> >>> gen.send(_.value) <__main__.Number object at 0x105852630> >>> gen.send(_.value) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 12 >>> • A method that yields nodes (iteration) Again, note the return mechanism
  162. 162. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Wrapping 162 >>> v = Evaluator() >>> n = BinOp('*', Number(3), Number(4)) >>> gen = v.genvisit(n) >>> gen <generator object genvisit at 0x10070ab88> >>> gen.send(None) <__main__.Number object at 0x1058525c0> >>> gen.send(_.value) <__main__.Number object at 0x105852630> >>> gen.send(_.value) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 12 >>> • A method that yields nodes def visit_Number(self, node): return node.value Manually carrying out this method in the example
  163. 163. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running Recursion 163 class NodeVisitor: def visit(self, node): stack = [ self.genvisit(node) ] ! result = None while stack: try: node = stack[-1].send(result) stack.append(self.genvisit(node)) result = None except StopIteration as exc: ! stack.pop() result = exc.value ! return result • Step 2: Run depth-first traversal with a stack • Basically, a stack of running generators
  164. 164. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Transcendence 164 # Make '0+1+2+3+4+...+999' text = '+'.join(str(x) for x in range(1000)) toks = tokenize(text) tree = parse(toks) val = Evaluate().visit(tree) print(val) • Does it work? • Yep 499500 • Yow!
  165. 165. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running Recursion 165 class Evaluator(NodeVisitor): def visit_BinOp(self, node): leftval = yield node.left rightval = yield node.right if node.op == '+': result = leftval + rightval ... return result • Each yield creates a new stack entry • Returned values (via StopIteration) get propagated as results generator generator generator stack yield yield return return
  166. 166. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running Recursion 166 >>> v = Evaluator() >>> n = BinOp('*', Number(3), Number(4)) >>> stack = [ v.genvisit(n) ] >>> stack[-1].send(None) <__main__.Number object at 0x1058525c0> >>> stack.append(v.genvisit(_)) >>> stack[-1].send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 3 >>> stack.pop() >>> stack[-1].send(3) <__main__.Number object at 0x105852630> >>> stack.append(v.genvisit(_)) >>> stack[-1].send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 4 >>> stack.pop() >>> stack[-1].send(4) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 12 >>> Nodes are visited and generators pushed onto a stack
  167. 167. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running Recursion 167 >>> v = Evaluator() >>> n = BinOp('*', Number(3), Number(4)) >>> stack = [ v.genvisit(n) ] >>> stack[-1].send(None) <__main__.Number object at 0x1058525c0> >>> stack.append(v.genvisit(_)) >>> stack[-1].send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 3 >>> stack.pop() >>> stack[-1].send(3) <__main__.Number object at 0x105852630> >>> stack.append(v.genvisit(_)) >>> stack[-1].send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 4 >>> stack.pop() >>> stack[-1].send(4) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 12 >>> Results propagate via StopIteration 12 (Final Result)
  168. 168. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 168 Final Words
  169. 169. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Historical Perspective 169 • Generators seem to have started as a simple way to implement iteration (Python 2.3) • Took an interesting turn with support for coroutines (Python 2.5) • Taken to a whole new level with delegation support in PEP-380 (Python 3.3).
  170. 170. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Control Flow Bending 170 • yield statement allows you to bend control-flow to adapt it to certain kinds of problems • Wrappers (context managers) • Futures/concurrency • Messaging • Recursion • Frankly, it blows my mind.
  171. 171. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com asyncio 171 • Inclusion of asyncio in standard library may be a game changer • To my knowledge, it's the only standard library module that uses coroutines/generator delegation in a significant manner • To really understand how it works, need to have your head wrapped around generators • Read the source for deep insight
  172. 172. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Is it Proper? 172 • Are coroutines/generators a good idea or not? • Answer: I still don't know • Issue: Coroutines seem like they're "all in" • Fraught with potential mind-bending issues • Example:Will there be two standard libraries?
  173. 173. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Two Libraries? 173 Python Standard Library Standard coroutine library (asyncio and friends) ????? • If two different worlds, do they interact? • If so, by what rules?
  174. 174. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Personal Use 174 • My own code is dreadfully boring • Generators for iteration: Yes. • Everything else: Threads, recursion, etc. (sorry) • Nevertheless:There may be something to all of this advanced coroutine/generator business
  175. 175. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Bit More Information 175
  176. 176. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thanks! 176 • I hope you got some new ideas • Please feel free to contact me http://www.dabeaz.com • Also, I teach Python classes (shameless plug) @dabeaz (Twitter) • Special Thanks: Brian Curtin, Ken Izzo, George Kappel, Christian Long, Michael Prentiss,Vladimir Urazov, Guido van Rossum

×