The document introduces doit, an automation tool that defines tasks as Python functions or generators. Doit provides more flexibility than traditional build tools by allowing tasks to be defined programmatically and checking dependencies through file timestamps or results of other tasks. It supports features like parallel and automatic execution of tasks to improve productivity.

1. doit – automation tool
Bringing the power of build-tools to
execute any kind of task
Eduardo Schettino
doit - automation tool 1

2. About me
 Brazil
 Beijing – China
 Exoweb http://www.exoweb.net
doit - automation tool 2

3. Talk overview
 build-tools (make)
 Who needs a build-tool?
 doit
 Questions & Answers
doit - automation tool 3

4. build-tools
Anything worth repeating is worth automating
 manage repetitive tasks and their dependencies
 speed-up development (faster turn-around)
 1977: Make (C & other compiled languages)
doit - automation tool 4

5. make model
Rules
Prerequisite 1
Prerequisite 2 Commands Targets
...
Prerequisite N
doit - automation tool 5

6. C project
foo.c
compile foo.o
link foobar
defs.h
compile bar.o
bar.c
doit - automation tool 6

7. 1 - compile foo
foo.c
compile foo.o
link foobar
defs.h
compile bar.o
bar.c
doit - automation tool 7

8. 2 - compile bar
foo.c
compile foo.o
link foobar
defs.h
compile bar.o
bar.c
doit - automation tool 8

9. 3 - foobar
foo.c
compile foo.o
link foobar
defs.h
compile bar.o
bar.c
doit - automation tool 9

10. initial build
foo.c
compile foo.o
link foobar
defs.h
compile bar.o
bar.c
All 3 operations executed
doit - automation tool 10

11. no-op rebuild
foo.c
compile foo.o
link foobar
defs.h
compile bar.o
bar.c
No changes. No operation executed
doit - automation tool 11

12. incremental rebuild
foo.c
compile foo.o
link foobar
defs.h
compile bar.o
bar.c
foo.c changed. 2 operations executed
doit - automation tool 12

13. how it works?
 file time-stamp comparison
 if any of the prerequisite files were modified
after last modification on target => re-execute
 if target is up-to-date => skip execution
doit - automation tool 13

14. who needs a build-tool?
 dynamic language => no compilation => static
checkers + unit-tests
 functional tests (DB + web) are slow
 maintenance tasks
doit - automation tool 14

15. python project
foo.py unit-test
main.py unit-test
bar.py unit-test
doit - automation tool 15

16. first run
foo.py unit-test
main.py unit-test
bar.py unit-test
doit - automation tool 16

17. no-op run???
foo.py unit-test
main.py unit-test
bar.py unit-test
 no modifications on source code
 all tests are executed again!
 no targets to compare time-stamps...
doit - automation tool 17

18. make problems
 ad-hoc language (hard to use, debug, ...)
 time-stamp based (fragile)
 restricted to operations that create files
 not good on dynamic creation of rules
doit - automation tool 18

19. doit goals
 more flexible model than traditional build-tools
 “real” language => python
 get out of your way
doit - automation tool 19

20. doit model
 based on “Tasks”, focus on “actions”
Tasks
dependency 1
dependency 2 Actions Targets
...
dependency N
doit - automation tool 20

21. how it works?
 same principle but...
 use a “db” file to save info from successful
execution
 tasks are defined on a “dodo” file, a plain
python module
 i.e. does not require targets to check if task is
up-to-date
doit - automation tool 21

22. Hello World
def task_hello():
return {'actions':
['echo Hello World > hello.txt']}
doit - automation tool 22

23. Hello World
A task generator ...
def task_hello():
return {'actions':
['echo Hello World > hello.txt']}
doit - automation tool 23

24. Hello World
A task generator ...
def task_hello():
... returns a dictionary with task meta-data
return {'actions':
['echo Hello World > hello.txt']}
doit - automation tool 24

25. Hello World
A task generator ...
def task_hello():
... returns a dictionary with task meta-data
return {'actions':
['echo Hello World > hello.txt']}
strings are shell commands
doit - automation tool 25

26. python actions
a plain python function
def pyhello():
with open('hello.txt', 'w') as hello_file:
hello_file.write("Hello Pythonn")
def task_hello():
return {'actions': [(pyhello,)]}
doit - automation tool 26

27. python actions
a plain python function
def pyhello():
with open('hello.txt', 'w') as hello_file:
hello_file.write("Hello Pythonn")
tuple (callable, args, kwargs)
def task_hello():
return {'actions': [(pyhello,)]}
doit - automation tool 27

28. compile
def task_compile():
return {'actions': ["cc ­c main.c"],
'file_dep': ["main.c","defs.h"],
'targets': ["main.o"]
}
dependencies targets
doit - automation tool 28

29. sub-tasks
define task meta-data
def task_create_file():
for i in range(3):
filename = "file%d.txt" % i
yield {'name': filename,
'actions':
["touch %s" % filename]}
doit - automation tool 29

30. sub-tasks
define task meta-data
def task_create_file():
for i in range(3):
filename = "file%d.txt" % i
yield {'name': filename,
'actions':
["touch %s" % filename]}
“yield” to create multiple tasks
doit - automation tool 30

31. sub-tasks
define task meta-data
def task_create_file():
for i in range(3):
filename = "file%d.txt" % i
required for sub-tasks
yield {'name': filename,
'actions':
["touch %s" % filename]}
“yield” to create multiple tasks
doit - automation tool 31

32. sub-tasks
define task meta-data
def task_create_file():
for i in range(3):
filename = "file%d.txt" % i
required for sub-tasks
yield {'name': filename,
'actions':
["touch %s" % filename]}
“yield” to create multiple tasks
doit - automation tool 32

33. task dependency
def task_foo():
return {'actions': ["echo foo"]}
def task_bar():
return {'actions': ["echo bar"]}
def task_mygroup():
return {'actions': None,
'task_dep': ['foo', 'bar']}
tasks that must be executed before this task
doit - automation tool 33

34. result dependency
def task_which_version():
return {'actions':
['bzr version­info ­­custom –template="{revno}n"']}
def task_do_something():
return {'actions': ['echo "TODO: send an email"'],
'result_dep': ['which_version']}
check result from another task instead of a file content
doit - automation tool 34

35. result dependency
def task_which_version():
return {'actions':
['bzr version­info ­­custom –template="{revno}n"']}
task “result” is string returned by action
def task_do_something():
return {'actions': ['echo "TODO: send an email"'],
'result_dep': ['which_version']}
check result from another task instead of a file content
doit - automation tool 35

36. execution
DOIT_CONFIG = {'default_tasks': ['t3']}
 by default all tasks are executed
 controlled by DOIT_CONFIG default_tasks
$ doit
. task3
$ doit task2 task1:foo
. task2
. task1:foo
doit - automation tool 36

37. up-to-date ?
 no file dependencies were modified
 no result dependencies were modified
 all targets exist
 compares => timestamp, size, checksum
doit - automation tool 37

38. up-to-date output
$ doit . (dot) => executed
. compile
$ doit -- (dashes) => skipped
­­ compile
$ rm main.o
$ doit
. compile
doit - automation tool 38

39. environment setup
def task_start_server():
for name in ('serverX', 'serverY'):
yield {'name': name,
'actions': [(start, (name,))],
'teardown': [(stop, (name,))],
}
def task_test_A():
return {'actions':['echo fun_test_a'],
'setup': ['start_server:serverX'],
}
test_A requires serverX to be running
start serverX only if test_A not up-to-date
doit - automation tool 39

40. environment setup
def task_start_server():
for name in ('serverX', 'serverY'):
yield {'name': name,
'actions': [(start, (name,))],
'teardown': [(stop, (name,))],
}
stop serverX after all tasks finish running
def task_test_A():
return {'actions':['echo fun_test_a'],
'setup': ['start_server:serverX'],
}
test_A requires serverX to be running
start serverX only if test_A not up-to-date
doit - automation tool 40

41. calculated dependency
def task_mod_deps():
return {'actions': [(print_deps,)],
'calc_dep': ["get_dep"],
}
dependencies are calculated on another task
def get_dep(mod):
return {'file_dep': [‘a’, ‘b’, ‘c’]}
def task_get_dep():
return {'actions':[(get_dep,)]}
doit - automation tool 41

42. calculated dependency
def task_mod_deps():
return {'actions': [(print_deps,)],
'calc_dep': ["get_dep"],
}
dependencies are calculated on another task
def get_dep(mod):
return {'file_dep': [‘a’, ‘b’, ‘c’]}
def task_get_dep():
return {'actions':[(get_dep,)]}
returns dictionary with same keys as task-dict
(file_dep, task_dep, result_dep, calc_dep)
doit - automation tool 42

43. other task features
 clean => clean actions ($doit clean <tasks>)
 doc => ($doit list)
 params => get parameters from command line
 getargs => get values computed in different
tasks
doit - automation tool 43

44. other runner features
 verbosity => capture/display stdout/stderr
 title => controls output (task name)
 custom_reporters => complete output control
 use wildcard to select tasks
doit - automation tool 44

45. parallel execution
 parallel execution of tasks in multiple processes
 uses multiprocessing lib
 subject to same limitations...
doit - automation tool 45

46. auto execution
 long running process
 watches for file modifications and automatically
re-execute outdated tasks
 works on linux and mac
 TDD, never leave the editor screen :)
doit - automation tool 46

47. thanks
 Questions ?
 website: http://python-doit.sourceforge.net
doit - automation tool 47