Fairly basic introduction to some principles and techniques for creating safe, fault-tolerant unix scripts using the Bash environment.

1. The Bash: Defensive Scripting
The Bash: Defensive Scripting
1
Montag, 29. August 11

2. The Bash: Defensive Scripting
2
Montag, 29. August 11

3. The Bash: Defensive Scripting
Unix
2
Montag, 29. August 11

4. The Bash: Defensive Scripting
Unix
Interfaces
2
Montag, 29. August 11

5. The Bash: Defensive Scripting
Unix
Interfaces
Re-use
2
Montag, 29. August 11

6. The Bash: Defensive Scripting
Unix
Interfaces
Re-use
Defensive Programming
2
Montag, 29. August 11

7. The Bash: Defensive Scripting
Unix
Interfaces
Re-use
Defensive Programming
Readings
2
Montag, 29. August 11

8. The Bash: Defensive Scripting
3
Montag, 29. August 11

9. The Bash: Defensive Scripting
Unix
3
Montag, 29. August 11

10. The Bash: Defensive Scripting
Unix
• Be forgiving in what you accept, but
strict in what you emit
3
Montag, 29. August 11

11. The Bash: Defensive Scripting
Unix
• Be forgiving in what you accept, but
strict in what you emit
• If you must fail, do so early and
noisily
3
Montag, 29. August 11

12. The Bash: Defensive Scripting
Unix
• Be forgiving in what you accept, but
strict in what you emit
• If you must fail, do so early and
noisily
• One tool for each task
3
Montag, 29. August 11

13. The Bash: Defensive Scripting
Unix
• Be forgiving in what you accept, but
strict in what you emit
• If you must fail, do so early and
noisily
• One tool for each task
• Human-readable data and protocols
3
Montag, 29. August 11

14. The Bash: Defensive Scripting
Unix
• Be forgiving in what you accept, but
strict in what you emit
• If you must fail, do so early and
noisily
• One tool for each task
• Human-readable data and protocols
• Do the simplest thing that will do
3
Montag, 29. August 11

15. The Bash: Defensive Scripting
4
Montag, 29. August 11

16. The Bash: Defensive Scripting
Unix: The Unix Toolbox
4
Montag, 29. August 11

17. The Bash: Defensive Scripting
Unix: The Unix Toolbox
Get familiar with cat, cut, tee, uniq,
sort, head, tail, but also grep, sed,
diff, find and possibly even awk. They
will save you a whole lot of work.
4
Montag, 29. August 11

18. The Bash: Defensive Scripting
Unix: The Unix Toolbox
Get familiar with cat, cut, tee, uniq,
sort, head, tail, but also grep, sed,
diff, find and possibly even awk. They
will save you a whole lot of work.
For more inspiration: ls /usr/bin
4
Montag, 29. August 11

19. The Bash: Defensive Scripting
Unix: The Unix Toolbox
Get familiar with cat, cut, tee, uniq,
sort, head, tail, but also grep, sed,
diff, find and possibly even awk. They
will save you a whole lot of work.
For more inspiration: ls /usr/bin
Extend your Unix Toolbox
4
Montag, 29. August 11

20. The Bash: Defensive Scripting
5
Montag, 29. August 11

21. The Bash: Defensive Scripting
Interfaces
5
Montag, 29. August 11

22. The Bash: Defensive Scripting
Interfaces
• stdin, stdout, stderr, exit code
5
Montag, 29. August 11

23. The Bash: Defensive Scripting
Interfaces
• stdin, stdout, stderr, exit code
• Favour plain text
5
Montag, 29. August 11

24. The Bash: Defensive Scripting
Interfaces
• stdin, stdout, stderr, exit code
• Favour plain text
• Treat comments as comments
5
Montag, 29. August 11

25. The Bash: Defensive Scripting
Interfaces
• stdin, stdout, stderr, exit code
• Favour plain text
• Treat comments as comments
• Filter, Source, Sink, Cantrip,
Compiler
5
Montag, 29. August 11

26. The Bash: Defensive Scripting
6
Montag, 29. August 11

27. The Bash: Defensive Scripting
Interfaces: Filter
6
Montag, 29. August 11

28. The Bash: Defensive Scripting
Interfaces: Filter
Read from stdin, write to stdout
6
Montag, 29. August 11

29. The Bash: Defensive Scripting
Interfaces: Filter
Read from stdin, write to stdout
Perfect for pipes
6
Montag, 29. August 11

30. The Bash: Defensive Scripting
Interfaces: Filter
Read from stdin, write to stdout
Perfect for pipes
Example: cut
6
Montag, 29. August 11

31. The Bash: Defensive Scripting
7
Montag, 29. August 11

32. The Bash: Defensive Scripting
Interfaces: Source
7
Montag, 29. August 11

33. The Bash: Defensive Scripting
Interfaces: Source
Read from file, write to stdout
7
Montag, 29. August 11

34. The Bash: Defensive Scripting
Interfaces: Source
Read from file, write to stdout
Interface to outer world, beginning of a
pipe sequence
7
Montag, 29. August 11

35. The Bash: Defensive Scripting
Interfaces: Source
Read from file, write to stdout
Interface to outer world, beginning of a
pipe sequence
Example: cat
7
Montag, 29. August 11

36. The Bash: Defensive Scripting
8
Montag, 29. August 11

37. The Bash: Defensive Scripting
Interfaces: Sink
8
Montag, 29. August 11

38. The Bash: Defensive Scripting
Interfaces: Sink
Read from stdin, write to file
8
Montag, 29. August 11

39. The Bash: Defensive Scripting
Interfaces: Sink
Read from stdin, write to file
Interface to outer world, end of a pipe
sequence
8
Montag, 29. August 11

40. The Bash: Defensive Scripting
Interfaces: Sink
Read from stdin, write to file
Interface to outer world, end of a pipe
sequence
Example: >
8
Montag, 29. August 11

41. The Bash: Defensive Scripting
9
Montag, 29. August 11

42. The Bash: Defensive Scripting
Interfaces: Cantrip
9
Montag, 29. August 11

43. The Bash: Defensive Scripting
Interfaces: Cantrip
Do sth. and return nothing but a status
code (or errors)
9
Montag, 29. August 11

44. The Bash: Defensive Scripting
Interfaces: Cantrip
Do sth. and return nothing but a status
code (or errors)
This is super scriptable
9
Montag, 29. August 11

45. The Bash: Defensive Scripting
Interfaces: Cantrip
Do sth. and return nothing but a status
code (or errors)
This is super scriptable
Example: rm
9
Montag, 29. August 11

46. The Bash: Defensive Scripting
10
Montag, 29. August 11

47. The Bash: Defensive Scripting
Interfaces: Compiler
10
Montag, 29. August 11

48. The Bash: Defensive Scripting
Interfaces: Compiler
Read from file, write to another file
10
Montag, 29. August 11

49. The Bash: Defensive Scripting
Interfaces: Compiler
Read from file, write to another file
Example: tar
10
Montag, 29. August 11

50. The Bash: Defensive Scripting
11
Montag, 29. August 11

51. The Bash: Defensive Scripting
Re-use
11
Montag, 29. August 11

52. The Bash: Defensive Scripting
Re-use
• Functions
11
Montag, 29. August 11

53. The Bash: Defensive Scripting
Re-use
• Functions
• Wrappers
11
Montag, 29. August 11

54. The Bash: Defensive Scripting
Re-use
• Functions
• Wrappers
• $() (aka ``)
11
Montag, 29. August 11

55. The Bash: Defensive Scripting
Re-use
• Functions
• Wrappers
• $() (aka ``)
• source (aka ‘.’)
11
Montag, 29. August 11

56. The Bash: Defensive Scripting
12
Montag, 29. August 11

57. The Bash: Defensive Scripting
Re-use: Functions
12
Montag, 29. August 11

58. The Bash: Defensive Scripting
Re-use: Functions
Will behave like a script on its own
12
Montag, 29. August 11

59. The Bash: Defensive Scripting
Re-use: Functions
Will behave like a script on its own
function log {
12
Montag, 29. August 11

60. The Bash: Defensive Scripting
Re-use: Functions
Will behave like a script on its own
function log {
echo “$(date) ERR $@” >&2
12
Montag, 29. August 11

61. The Bash: Defensive Scripting
Re-use: Functions
Will behave like a script on its own
function log {
echo “$(date) ERR $@” >&2
}
12
Montag, 29. August 11

62. The Bash: Defensive Scripting
Re-use: Functions
Will behave like a script on its own
function log {
echo “$(date) ERR $@” >&2
}
log “Cannot connect”
12
Montag, 29. August 11

63. The Bash: Defensive Scripting
Re-use: Functions
Will behave like a script on its own
function log {
echo “$(date) ERR $@” >&2
}
log “Cannot connect”
# Do 4 Aug 2011 10:38:58 CEST ERR
Cannot connect 12
Montag, 29. August 11

64. The Bash: Defensive Scripting
13
Montag, 29. August 11

65. The Bash: Defensive Scripting
Re-use: Wrappers
13
Montag, 29. August 11

66. The Bash: Defensive Scripting
Re-use: Wrappers
#!/bin/bash
13
Montag, 29. August 11

67. The Bash: Defensive Scripting
Re-use: Wrappers
#!/bin/bash
# file mcd -- make a dir if
13
Montag, 29. August 11

68. The Bash: Defensive Scripting
Re-use: Wrappers
#!/bin/bash
# file mcd -- make a dir if
# necessary and cd to it
13
Montag, 29. August 11

69. The Bash: Defensive Scripting
Re-use: Wrappers
#!/bin/bash
# file mcd -- make a dir if
# necessary and cd to it
test -n “$1” || exit 1
13
Montag, 29. August 11

70. The Bash: Defensive Scripting
Re-use: Wrappers
#!/bin/bash
# file mcd -- make a dir if
# necessary and cd to it
test -n “$1” || exit 1
newdir=”$1”
13
Montag, 29. August 11

71. The Bash: Defensive Scripting
Re-use: Wrappers
#!/bin/bash
# file mcd -- make a dir if
# necessary and cd to it
test -n “$1” || exit 1
newdir=”$1”
test -d $newdir || mkdir $newdir
13
Montag, 29. August 11

72. The Bash: Defensive Scripting
Re-use: Wrappers
#!/bin/bash
# file mcd -- make a dir if
# necessary and cd to it
test -n “$1” || exit 1
newdir=”$1”
test -d $newdir || mkdir $newdir
cd $newdir 13
Montag, 29. August 11

73. The Bash: Defensive Scripting
14
Montag, 29. August 11

74. The Bash: Defensive Scripting
Re-use: $()
14
Montag, 29. August 11

75. The Bash: Defensive Scripting
Re-use: $()
More readable than `backticks`
14
Montag, 29. August 11

76. The Bash: Defensive Scripting
Re-use: $()
More readable than `backticks`
myfiles=$(ls $HOME)
14
Montag, 29. August 11

77. The Bash: Defensive Scripting
15
Montag, 29. August 11

78. The Bash: Defensive Scripting
Re-use: source
15
Montag, 29. August 11

79. The Bash: Defensive Scripting
Re-use: source
More readable than “.”
15
Montag, 29. August 11

80. The Bash: Defensive Scripting
Re-use: source
More readable than “.”
source $HOME/lib/basics.sh
15
Montag, 29. August 11

81. The Bash: Defensive Scripting
Re-use: source
More readable than “.”
source $HOME/lib/basics.sh
source /etc/conf/script.cnf
15
Montag, 29. August 11

82. The Bash: Defensive Scripting
16
Montag, 29. August 11

83. The Bash: Defensive Scripting
Defensive Programming
16
Montag, 29. August 11

84. The Bash: Defensive Scripting
Defensive Programming
• test, test, test (for $?)
16
Montag, 29. August 11

85. The Bash: Defensive Scripting
Defensive Programming
• test, test, test (for $?)
• find
16
Montag, 29. August 11

86. The Bash: Defensive Scripting
Defensive Programming
• test, test, test (for $?)
• find
• && and || (AND and OR)
16
Montag, 29. August 11

87. The Bash: Defensive Scripting
Defensive Programming
• test, test, test (for $?)
• find
• && and || (AND and OR)
• trap
16
Montag, 29. August 11

88. The Bash: Defensive Scripting
Defensive Programming
• test, test, test (for $?)
• find
• && and || (AND and OR)
• trap
• log, stderr
16
Montag, 29. August 11

89. The Bash: Defensive Scripting
Defensive Programming
• test, test, test (for $?)
• find
• && and || (AND and OR)
• trap
• log, stderr
• --help, --version, usage()
16
Montag, 29. August 11

90. The Bash: Defensive Scripting
Defensive Programming
• test, test, test (for $?)
• find
• && and || (AND and OR)
• trap
• log, stderr
• --help, --version, usage()
• Use full path names
16
Montag, 29. August 11

91. The Bash: Defensive Scripting
17
Montag, 29. August 11

92. The Bash: Defensive Scripting
Defensive Programming: test
17
Montag, 29. August 11

93. The Bash: Defensive Scripting
Defensive Programming: test
test -r “$sourcefile”
17
Montag, 29. August 11

94. The Bash: Defensive Scripting
Defensive Programming: test
test -r “$sourcefile”
test -d “$destdir”
17
Montag, 29. August 11

95. The Bash: Defensive Scripting
Defensive Programming: test
test -r “$sourcefile”
test -d “$destdir”
test -x “$script”
17
Montag, 29. August 11

96. The Bash: Defensive Scripting
Defensive Programming: test
test -r “$sourcefile”
test -d “$destdir”
test -x “$script”
test -w “$destfile”
17
Montag, 29. August 11

97. The Bash: Defensive Scripting
Defensive Programming: test
test -r “$sourcefile”
test -d “$destdir”
test -x “$script”
test -w “$destfile”
test -n “$variable”
17
Montag, 29. August 11

98. The Bash: Defensive Scripting
Defensive Programming: test
test -r “$sourcefile”
test -d “$destdir”
test -x “$script”
test -w “$destfile”
test -n “$variable”
test -z “$nothing”
17
Montag, 29. August 11

99. The Bash: Defensive Scripting
18
Montag, 29. August 11

100. The Bash: Defensive Scripting
Defensive Programming: test
18
Montag, 29. August 11

101. The Bash: Defensive Scripting
Defensive Programming: test
test -r “$sourcefile”
18
Montag, 29. August 11

102. The Bash: Defensive Scripting
Defensive Programming: test
test -r “$sourcefile”
if [[ $? = 0 ]]; then
18
Montag, 29. August 11

103. The Bash: Defensive Scripting
Defensive Programming: test
test -r “$sourcefile”
if [[ $? = 0 ]]; then
echo “yes, it’s readable”
18
Montag, 29. August 11

104. The Bash: Defensive Scripting
Defensive Programming: test
test -r “$sourcefile”
if [[ $? = 0 ]]; then
echo “yes, it’s readable”
else
18
Montag, 29. August 11

105. The Bash: Defensive Scripting
Defensive Programming: test
test -r “$sourcefile”
if [[ $? = 0 ]]; then
echo “yes, it’s readable”
else
echo “cannot read $sourcefile”
18
Montag, 29. August 11

106. The Bash: Defensive Scripting
Defensive Programming: test
test -r “$sourcefile”
if [[ $? = 0 ]]; then
echo “yes, it’s readable”
else
echo “cannot read $sourcefile”
fi
18
Montag, 29. August 11

107. The Bash: Defensive Scripting
19
Montag, 29. August 11

108. The Bash: Defensive Scripting
Defensive Programming: && and ||
19
Montag, 29. August 11

109. The Bash: Defensive Scripting
Defensive Programming: && and ||
test -r “$sourcefile” || exit 1
19
Montag, 29. August 11

110. The Bash: Defensive Scripting
20
Montag, 29. August 11

111. The Bash: Defensive Scripting
Defensive Programming: && and ||
20
Montag, 29. August 11

112. The Bash: Defensive Scripting
Defensive Programming: && and ||
test -r “$sourcefile” || {
20
Montag, 29. August 11

113. The Bash: Defensive Scripting
Defensive Programming: && and ||
test -r “$sourcefile” || {
echo “Cannot read source file” >&2
20
Montag, 29. August 11

114. The Bash: Defensive Scripting
Defensive Programming: && and ||
test -r “$sourcefile” || {
echo “Cannot read source file” >&2
exit 1
20
Montag, 29. August 11

115. The Bash: Defensive Scripting
Defensive Programming: && and ||
test -r “$sourcefile” || {
echo “Cannot read source file” >&2
exit 1
}
20
Montag, 29. August 11

116. The Bash: Defensive Scripting
21
Montag, 29. August 11

117. The Bash: Defensive Scripting
Defensive Programming: && and ||
21
Montag, 29. August 11

118. The Bash: Defensive Scripting
Defensive Programming: && and ||
test -r “$sourcefile” && {
21
Montag, 29. August 11

119. The Bash: Defensive Scripting
Defensive Programming: && and ||
test -r “$sourcefile” && {
cat -n “$sourcefile”
21
Montag, 29. August 11

120. The Bash: Defensive Scripting
Defensive Programming: && and ||
test -r “$sourcefile” && {
cat -n “$sourcefile”
} || {
21
Montag, 29. August 11

121. The Bash: Defensive Scripting
Defensive Programming: && and ||
test -r “$sourcefile” && {
cat -n “$sourcefile”
} || {
echo “Cannot read source file” >&2
21
Montag, 29. August 11

122. The Bash: Defensive Scripting
Defensive Programming: && and ||
test -r “$sourcefile” && {
cat -n “$sourcefile”
} || {
echo “Cannot read source file” >&2
exit 1
21
Montag, 29. August 11

123. The Bash: Defensive Scripting
Defensive Programming: && and ||
test -r “$sourcefile” && {
cat -n “$sourcefile”
} || {
echo “Cannot read source file” >&2
exit 1
}
21
Montag, 29. August 11

124. The Bash: Defensive Scripting
22
Montag, 29. August 11

125. The Bash: Defensive Scripting
Defensive Programming: find
22
Montag, 29. August 11

126. The Bash: Defensive Scripting
Defensive Programming: find
Don’t rely on a file being in some exact
place. If it needs to be, tell the user.
22
Montag, 29. August 11

127. The Bash: Defensive Scripting
Defensive Programming: find
Don’t rely on a file being in some exact
place. If it needs to be, tell the user.
find Syntax is a bit peculiar.
22
Montag, 29. August 11

128. The Bash: Defensive Scripting
Defensive Programming: find
Don’t rely on a file being in some exact
place. If it needs to be, tell the user.
find Syntax is a bit peculiar.
find $install_dir -name “*README*” -type
f -exec mv {} $install_dir/docs/ ;
22
Montag, 29. August 11

129. The Bash: Defensive Scripting
23
Montag, 29. August 11

130. The Bash: Defensive Scripting
Defensive Programming: trap
23
Montag, 29. August 11

131. The Bash: Defensive Scripting
Defensive Programming: trap
trap “/bin/rm -f $tmpf; exit” EXIT
23
Montag, 29. August 11

132. The Bash: Defensive Scripting
Defensive Programming: trap
trap “/bin/rm -f $tmpf; exit” EXIT
trap “source $config” SIGHUP
23
Montag, 29. August 11

133. The Bash: Defensive Scripting
function log {
msg=“$(date) ERR $@”
test -w $LOGF && echo $msg > $LOGF
echo $msg >&2
}
24
Montag, 29. August 11

134. The Bash: Defensive Scripting
Defensive Programming: Logs
function log {
msg=“$(date) ERR $@”
test -w $LOGF && echo $msg > $LOGF
echo $msg >&2
}
24
Montag, 29. August 11

135. The Bash: Defensive Scripting
Defensive Programming: Logs
Write status to stderr. stderr is not
for *errors* only, but for all kind of
metainformation
function log {
msg=“$(date) ERR $@”
test -w $LOGF && echo $msg > $LOGF
echo $msg >&2
}
24
Montag, 29. August 11

136. The Bash: Defensive Scripting
Defensive Programming: Logs
Write status to stderr. stderr is not
for *errors* only, but for all kind of
metainformation
Use syslog/logger or your own logging
function function log {
msg=“$(date) ERR $@”
test -w $LOGF && echo $msg > $LOGF
echo $msg >&2
}
24
Montag, 29. August 11

137. The Bash: Defensive Scripting
25
Montag, 29. August 11

138. The Bash: Defensive Scripting
Defensive Programming:
25
Montag, 29. August 11

139. The Bash: Defensive Scripting
Defensive Programming:
--help, --version
25
Montag, 29. August 11

140. The Bash: Defensive Scripting
Defensive Programming:
--help, --version
• Simplify the use of your program by
providing --help and --version
25
Montag, 29. August 11

141. The Bash: Defensive Scripting
Defensive Programming:
--help, --version
• Simplify the use of your program by
providing --help and --version
• On error print a usage message
25
Montag, 29. August 11

142. The Bash: Defensive Scripting
26
Montag, 29. August 11

143. The Bash: Defensive Scripting
Defensive Programming:
26
Montag, 29. August 11

144. The Bash: Defensive Scripting
Defensive Programming:
use full path names
26
Montag, 29. August 11

145. The Bash: Defensive Scripting
Defensive Programming:
use full path names
base=$HOME/install
26
Montag, 29. August 11

146. The Bash: Defensive Scripting
Defensive Programming:
use full path names
base=$HOME/install
srcf=$base/source.txt
26
Montag, 29. August 11

147. The Bash: Defensive Scripting
Defensive Programming:
use full path names
base=$HOME/install
srcf=$base/source.txt
dstf=$base/destination.txt
26
Montag, 29. August 11

148. The Bash: Defensive Scripting
Defensive Programming:
use full path names
base=$HOME/install
srcf=$base/source.txt
dstf=$base/destination.txt
test -r $srcf && cp -i $srcf $dstf
26
Montag, 29. August 11

149. The Bash: Defensive Scripting
27
Montag, 29. August 11

150. The Bash: Defensive Scripting
Readings
27
Montag, 29. August 11

151. The Bash: Defensive Scripting
Readings
• Advanced Bash-Scripting Guide http://
tldp.org/LDP/abs/html/
27
Montag, 29. August 11

152. The Bash: Defensive Scripting
Readings
• Advanced Bash-Scripting Guide http://
tldp.org/LDP/abs/html/
• The Art of Unix Programming (by Eric S.
Raymond; http://www.faqs.org/docs/artu/)
27
Montag, 29. August 11

153. The Bash: Defensive Scripting
Readings
• Advanced Bash-Scripting Guide http://
tldp.org/LDP/abs/html/
• The Art of Unix Programming (by Eric S.
Raymond; http://www.faqs.org/docs/artu/)
• Wicked Cool Shell Scripts (No Starch Press)
27
Montag, 29. August 11

154. The Bash: Defensive Scripting
Readings
• Advanced Bash-Scripting Guide http://
tldp.org/LDP/abs/html/
• The Art of Unix Programming (by Eric S.
Raymond; http://www.faqs.org/docs/artu/)
• Wicked Cool Shell Scripts (No Starch Press)
• Patrick Jezek: Bash Techtalk (08/2010)
http://slides.liip.ch/shell_scripting/
index.html 27
Montag, 29. August 11