make is a basic tool to define pipelines of shell commands. It is useful if you have many shell scripts and commands, and you want to organize them. Even if it has been written to automatize the build of compiled language programs, make is also useful in bioinformatics and other fields.

1. BioEvo technical seminars GNU/Make and bioinformatics G.M. Dall'Olio Barcelona, 06/02/2009

3. g++ -c main.cpp; g++ -c func.cpp; g++ main.o func.o

6. perl/python/R/other scripts

7. Many suites of binary programs (emboss, phylip, blast, t-coffee, plink, genepop, gromacs, rosetta...)

8. etc...

10. Execute these commands only when necessary (don't calculate again some results, if they have already been calculated)

12. It is a declarative programming language

13. It belongs to a class of softwares called 'automated build tools'

15. Simplest Makefile example Makefile rule Target of the rule Commands associated with the rule This is a tabulation (not 8 spaces)

17. Write these instructions in it: print_hello : echo 'Hello, world!!'

19. Simplest Makefile example

21. When we type 'make print_hello', it executes any procedure (target) called 'print_hello' in the makefile

22. It then shows the commands executed and their output

24. If you omit this option, make will look for any file called 'Makefile' in the current directory

25. make -f Makefile all is equivalent to: make all

27. For example, this ' print_hello ' rule contains 5 commands

28. Note: ignore the '@' thing, it is only to disable verbose mode (explained later)

29. A more complex example

31. It allows you to use command-line tools which are more flexible;

32. Combined with a revision control software, it makes possible to reproduce all the operations made to your data;

33. Second part A closer look at make syntax (target and commands)

36. Everytime you call a make rule (example: 'make all'), the program looks for a file called like the target name (e.g. 'all', 'clean', 'inputdata.txt', 'results.txt')

37. The rule is executed only if that file doesn't exists.

40. When we call ' make testfile.txt ', make checks if a file called 'testfile.txt' already exists.

41. Filename as target names The commands associated with the rule ' testfile.txt ' are executed only if that file doesn't exists already

43. You can retrieve the matched target with the special variable $@

46. Special character % / creating more than a file at a time

48. If we use the '-j' option when invoking make, the three processess will be launched in parallel

53. '-mkdir /var' will cause an error anyway, but gnu/make will ignore it

55. So, this: lsvar : cd /var ls

56. Won't work (it will list only the files in the current directory, not /var)

57. The solution is to put everything in a single process:

58. lsvar : (cd /var; ls)

59. Third part Prerequisites and conditional execution

62. Example: result1.txt : data1.txt data2.txt cat data1.txt data2.txt > result1.txt @echo 'result1.txt' has been calculated'

63. Prerequisites are files (or rules) that need to exists already in order to create the target file.

64. If 'data1.txt' and 'data2.txt' don't exist, the rule 'result1.txt' will exit with an error (no rule to create them)

68. what happens if we remove the file 'result1.txt' we just created?

70. what happens if we remove the file 'result1.txt' we just created?

71. The second time we run the 'make result1.txt' command, it is not necessary to create data1.txt again, so only a rule is executed

73. Make all will calculate result1.txt and result2.txt, if they don't exist already (and they are older than their prerequisites)

75. We can do better: create or update a file only if it is newer than its prerequisites

77. A great feature of make is that it execute a rule not only if the target file doesn't exist, but also if it has a 'last modification date' earlier than all of its prerequisites

79. $: touch data1.txt calculate_result.py $: make result1.txt result1.txt has been calculated $: make result1.txt result1.txt is already up-to-date $: touch data1.txt $: make result1.txt result1.txt has been calculated

81. Let's say you discover an error in one of your input data: you will be able to repeat the analysis by executing only the operations needed

82. You can also use it to re-calculate results every time you modify a script: result.txt : scripts/calculate_result.py python calculate_result.py > result.py

83. Another example

84. Fourth part Variables and functions

86. In fact, it is a ~40 years old language

87. It uses special variables like $@, $^, and it can be worst than perl!!!

88. (perl developers – please don't get mad at me :-) )

90. They are called by including their name in ' $() ' WORKING_DIR is a variable

92. '$@' always corresponds to the target name: $: cat >Makefile %.txt : echo $@ $: make filename.txt echo filename.txt filename.txt $: $@ took the value of 'filename.txt'

93. Other special variables $@ The rule's target $< The rule's first prerequisite $? All the rule's out of date prerequisites $^ All Prerequisites

95. Most frequently used functions: $(addprefix <prefix>, list) -> add a prefix to a space-separated list example: FILES = file1 file2 file3 $(addprefix /home/user/data, $(FILES)

96. $(addsuffix) work similarly

98. Fifth part Testing, discussion, other examples and alternatives

100. You can pass variables to make: $: make say_hello MYNAME=”Giovanni” hello, Giovanni

101. Strongly suggested: use a Revision Control Software with support for branching (git, hg, bazaar) and create a branch for testing

103. we first want to mask out any repeats using rmask to create myseq.m

104. we then blastx myseq.m against a protein db called mydb

105. before blastx is run the protein db must be indexed using formatdb (slide taken from biomake web site)

108. Developed to annotate the Drosophila melanogaster genome (Berkeley university)

109. Cleaner syntax,derived from prolog

110. Separates the rule's name from the name of the target files

111. A BioMake example (slide taken from biomake web site)

113. o/q/dist/etc.. make

114. Ant (Java)

115. Scons (python)

116. Paver (python)

118. These tools are more oriented to software development

121. Working with clusters

122. Avoid re-calculations

124. Study it and understand its logic. Use it in the most basic way, without worrying about prerequisites and special variables. Later you can look for easier tools (biomake, rake, taverna, galaxy, your own, etc..)

126. A Makefile is a pipeline http://www.nodalpoint.org/2007/03/18/a_pipeline_is_a_makefile

127. BioMake and SKAM http://skam.sourceforge.net/

128. BioWiki Make Manifesto http://biowiki.org/MakefileManifesto

129. Discussion on the BIP mailing list http://www.mail-archive.com/biology-in-python@lists.idyll.org/msg00013.html

133. The people in the bip mailing list, for discussion

134. The author of bioinformaticszen.org and the people on nodalpoint for priming

138. Every time a command is called, make shows the exact line which is being executed This is the statement being executed

141. Example: results.txt : data1.txt cut -f 1-10 data.txt > results.txt