Ce diaporama a bien été signalé.
Nous utilisons votre profil LinkedIn et vos données d’activité pour vous proposer des publicités personnalisées et pertinentes. Vous pouvez changer vos préférences de publicités à tout moment.

Unix command-line tools

48 061 vues

Publié le

Publié dans : Technologie
  • //DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT ......................................................................................................................... ......................................................................................................................... //DOWNLOAD PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... //DOWNLOAD EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... //DOWNLOAD doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... //DOWNLOAD PDF EBOOK here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... //DOWNLOAD EPUB Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... //DOWNLOAD doc Ebook here { https://tinyurl.com/y8nn3gmc } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • i saw a much Better PPT on ThesisScientist.com on the same Topic
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • @SexxxMaster I agree. An .odp would make more sense. :) Still, the content is fine.
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici
  • A .ppt to explain linux command-line tools???? WTF!?!
       Répondre 
    Voulez-vous vraiment ?  Oui  Non
    Votre message apparaîtra ici

Unix command-line tools

  1. 1. find, grep, sed, & awk Increasing productivity with command-line tools.
  2. 2. Unix philosophy <ul><li>“ This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” </li></ul><ul><li>--Doug McIlroy, inventor of Unix pipes </li></ul>
  3. 3. Why learn command-line utils? <ul><li>Simple – “do one thing” </li></ul><ul><li>Flexible – built for re-use </li></ul><ul><li>Fast – no graphics, no overhead </li></ul><ul><li>Ubiquitous – available on every machine </li></ul><ul><li>Permanent – 40 years so far … </li></ul>
  4. 4. Part 0 – pipes and xargs
  5. 5. Some simple programs <ul><li>List files in current working directory: </li></ul><ul><li>$ ls </li></ul><ul><li>foo bar bazoo </li></ul><ul><li>Count lines in file foo : </li></ul><ul><li>$ wc –l foo </li></ul><ul><li>42 foo </li></ul>
  6. 6. Putting programs together <ul><li>$ ls | wc –l </li></ul><ul><li>3 </li></ul><ul><li>$ ls | xargs wc –l </li></ul><ul><li>42 foo </li></ul><ul><li>31 bar </li></ul><ul><li>12 bazoo </li></ul><ul><li>85 total </li></ul>
  7. 7. Part 1: find
  8. 8. Basic find examples <ul><li>$ find . –name Account.java </li></ul>
  9. 9. Basic find examples <ul><li>$ find . –name Account.java </li></ul><ul><li>$ find /etc –name '*.conf' </li></ul>
  10. 10. Basic find examples <ul><li>$ find . –name Account.java </li></ul><ul><li>$ find /etc –name '*.conf' </li></ul><ul><li>$ find . –name '*.xml' </li></ul>
  11. 11. Basic find examples <ul><li>$ find . –name Account.java </li></ul><ul><li>$ find /etc –name '*.conf' </li></ul><ul><li>$ find . –name '*.xml' </li></ul><ul><li>$ find . -not -name '*.java' -maxdepth 4 </li></ul>
  12. 12. Basic find examples <ul><li>$ find . –name Account.java </li></ul><ul><li>$ find /etc –name '*.conf' </li></ul><ul><li>$ find . –name '*.xml' </li></ul><ul><li>$ find . -not -name '*.java' -maxdepth 4 </li></ul><ul><li>$ find . (-name '*jsp' –o –name '*xml') </li></ul>
  13. 13. Basic find examples <ul><li>$ find . –name Account.java </li></ul><ul><li>$ find /etc –name '*.conf' </li></ul><ul><li>$ find . –name '*.xml' </li></ul><ul><li>$ find . -not -name '*.java' -maxdepth 4 </li></ul><ul><li>$ find . (-name '*jsp' –o –name '*xml') </li></ul><ul><li>-iname case-insensitive </li></ul><ul><li>! == -not </li></ul><ul><li>Quotes keep shell from expanding wildcards. </li></ul>
  14. 14. Find and do stuff <ul><li>$ find . –name '*.java' | xargs wc –l | sort </li></ul>
  15. 15. Find and do stuff <ul><li>$ find . –name '*.java' | xargs wc –l | sort </li></ul><ul><li>Other options: </li></ul><ul><li>$ find . –name '*.java' -exec wc –l {} ; | sort </li></ul><ul><li>$ find . –name '*.java' -exec wc –l {} + | sort </li></ul>
  16. 16. Find and do stuff <ul><li>$ find . –name '*.java' | xargs wc –l | sort </li></ul><ul><li>Other options: </li></ul><ul><li>$ find . –name '*.java' -exec wc –l {} ; | sort </li></ul><ul><li>$ find . –name '*.java' -exec wc –l {} + | sort </li></ul><ul><li>Use your imagination. mv, rm, cp, chmod . . . </li></ul>
  17. 17. -exec or | xargs ? <ul><li>-exec has crazy syntax. </li></ul><ul><li>| xargs fits Unix philosophy. </li></ul><ul><li>; is slow, executes command once for each line. </li></ul><ul><li>; not sensible, sorts 'alphabetically.' </li></ul><ul><li>| xargs may fail with filenames containing whitespace, quotes or slashes. </li></ul>
  18. 18. Find by type <ul><li>Files: </li></ul><ul><li>$ find . –type f </li></ul>
  19. 19. Find by type <ul><li>Files: </li></ul><ul><li>$ find . –type f </li></ul><ul><li>Directories: </li></ul><ul><li>$ find . –type d </li></ul>
  20. 20. Find by type <ul><li>Files: </li></ul><ul><li>$ find . –type f </li></ul><ul><li>Directories: </li></ul><ul><li>$ find . –type d </li></ul><ul><li>Links: </li></ul><ul><li>$ find . –type l </li></ul>
  21. 21. By modification time <ul><li>Changed within day: </li></ul><ul><li>$ find . –mtime -1 </li></ul>
  22. 22. By modification time <ul><li>Changed within day: </li></ul><ul><li>$ find . –mtime -1 </li></ul><ul><li>Changed within minute: </li></ul><ul><li>$ find . –mmin -15 </li></ul>
  23. 23. By modification time <ul><li>Changed within day: </li></ul><ul><li>$ find . –mtime -1 </li></ul><ul><li>Changed within minute: </li></ul><ul><li>$ find . –mmin -15 </li></ul><ul><li>Variants –ctime , -cmin , -atime , -amin aren't especially useful. </li></ul>
  24. 24. By modification time, II <ul><li>Compare to file </li></ul><ul><li>$ find . –newer foo.txt </li></ul><ul><li>$ find . ! –newer foo.txt </li></ul>
  25. 25. By modification time, III <ul><li>Compare to date </li></ul><ul><li>$ find . -type f -newermt '2010-01-01' </li></ul>
  26. 26. By modification time, III <ul><li>Compare to date </li></ul><ul><li>$ find . -type f -newermt '2010-01-01' </li></ul><ul><li>Between dates! </li></ul><ul><li>$ find . -type f -newermt '2010-01-01' </li></ul><ul><li>> ! -newermt '2010-06-01' </li></ul>
  27. 27. Find by permissions <ul><li>$ find . –perm 644 </li></ul><ul><li>$ find . –perm –u=w </li></ul><ul><li>$ find . –perm –ug=w </li></ul><ul><li>$ find . –perm –o=x </li></ul>
  28. 28. Find by size <ul><li>Less than 1 kB: </li></ul><ul><li>$ find . –size -1k </li></ul>
  29. 29. Find by size <ul><li>Less than 1 kB: </li></ul><ul><li>$ find . –size -1k </li></ul><ul><li>More than 100MB: </li></ul><ul><li>$ find . –size +100M </li></ul>
  30. 30. find summary: <ul><li>Can search by name, path, depth, permissions, type, size, modification time, and more. </li></ul>
  31. 31. find summary: <ul><li>Can search by name, path, depth, permissions, type, size, modification time, and more. </li></ul><ul><li>Once you find what you want, pipe it to xargs if you want to do something with it. </li></ul>
  32. 32. find summary: <ul><li>Can search by name, path, depth, permissions, type, size, modification time, and more. </li></ul><ul><li>Once you find what you want, pipe it to xargs if you want to do something with it. </li></ul><ul><li>The puppy is for your grandmother. </li></ul>
  33. 33. Part 2: grep <ul><li>g lobal / r egular e xpression / p rint </li></ul><ul><li>From ed command g/re/p </li></ul><ul><li>For finding text inside files. </li></ul>
  34. 34. Basic usage: <ul><li>$ grep <string> <file or directory> </li></ul>
  35. 35. Basic usage: <ul><li>$ grep <string> <file or directory> </li></ul><ul><li>$ grep 'new FooDao' Bar.java </li></ul>
  36. 36. Basic usage: <ul><li>$ grep <string> <file or directory> </li></ul><ul><li>$ grep 'new FooDao' Bar.java </li></ul><ul><li>$ grep Account *.xml </li></ul>
  37. 37. Basic usage: <ul><li>$ grep <string> <file or directory> </li></ul><ul><li>$ grep 'new FooDao' Bar.java </li></ul><ul><li>$ grep Account *.xml </li></ul><ul><li>$ grep –r 'Dao[Impl|Mock]' src </li></ul>
  38. 38. Basic usage: <ul><li>$ grep <string> <file or directory> </li></ul><ul><li>$ grep 'new FooDao' Bar.java </li></ul><ul><li>$ grep Account *.xml </li></ul><ul><li>$ grep –r 'Dao[Impl|Mock]' src </li></ul><ul><li>Quote string if spaces or regex. </li></ul><ul><li>Recursive flag is typical </li></ul><ul><li>Don't quote filename with wildcards! </li></ul>
  39. 39. Common grep options <ul><li>Case-insensitive search: </li></ul><ul><li>$ grep –i foo bar.txt </li></ul>
  40. 40. Common grep options <ul><li>Case-insensitive search: </li></ul><ul><li>$ grep –i foo bar.txt </li></ul><ul><li>Only find word matches: </li></ul><ul><li>$ grep –rw foo src </li></ul>
  41. 41. Common grep options <ul><li>Case-insensitive search: </li></ul><ul><li>$ grep –i foo bar.txt </li></ul><ul><li>Only find word matches: </li></ul><ul><li>$ grep –rw foo src </li></ul><ul><li>Display line number: </li></ul><ul><li>$ grep –nr 'new Foo()' src </li></ul>
  42. 42. Filtering results <ul><li>In v erted search: </li></ul><ul><li>$ grep –v foo bar.txt </li></ul><ul><li>Prints lines not containing foo. </li></ul>
  43. 43. Filtering results <ul><li>In v erted search: </li></ul><ul><li>$ grep –v foo bar.txt </li></ul><ul><li>Prints lines not containing foo. </li></ul><ul><li>Typical use: </li></ul><ul><li>$ grep –r User src | grep –v svn </li></ul>
  44. 44. Filtering results <ul><li>In v erted search: </li></ul><ul><li>$ grep –v foo bar.txt </li></ul><ul><li>Prints lines not containing foo. </li></ul><ul><li>Typical use: </li></ul><ul><li>$ grep –r User src | grep –v svn </li></ul><ul><li>Using find … | xargs grep … is faster. </li></ul>
  45. 45. More grep options <ul><li>Search for multiple terms: </li></ul><ul><li>$ grep -e foo –e bar baz.txt </li></ul>
  46. 46. More grep options <ul><li>Search for multiple terms: </li></ul><ul><li>$ grep -e foo –e bar baz.txt </li></ul><ul><li>Find surrounding lines: </li></ul><ul><li>$ grep –r –C 2 foo src </li></ul>
  47. 47. More grep options <ul><li>Search for multiple terms: </li></ul><ul><li>$ grep -e foo –e bar baz.txt </li></ul><ul><li>Find surrounding lines: </li></ul><ul><li>$ grep –r –C 2 foo src </li></ul><ul><li>Similarly –A or –B will print lines before and after the line containing match. </li></ul>
  48. 48. Example <ul><li>Find tests that use the AccountDao interface. </li></ul>
  49. 49. Example <ul><li>Find tests that use the AccountDao interface. </li></ul><ul><li>Possible solution (arrive at incrementally): </li></ul><ul><li>$ grep –rwn –C 3 AccountDao src/test </li></ul><ul><li>> | grep –v svn </li></ul>
  50. 50. grep summary: <ul><li>-r r ecursive search </li></ul><ul><li>-i case i nsensitive </li></ul><ul><li>-w whole w ord </li></ul><ul><li>-n line n umber </li></ul><ul><li>-e multipl e searches </li></ul><ul><li>-A A fter </li></ul><ul><li>-B B efore </li></ul><ul><li>-C C entered </li></ul>
  51. 51. Part 3: sed <ul><li>s tream ed itor </li></ul><ul><li>For modifying files and streams of text. </li></ul>
  52. 52. sed command #1: s <ul><li>$ echo 'foo' | sed 's/foo/bar/' </li></ul>
  53. 53. sed command #1: s <ul><li>$ echo 'foo' | sed 's/foo/bar/' </li></ul><ul><li>bar </li></ul>
  54. 54. sed command #1: s <ul><li>$ echo 'foo' | sed 's/foo/bar/' </li></ul><ul><li>bar </li></ul><ul><li>$ echo 'foo foo' | sed 's/foo/bar/' </li></ul>
  55. 55. sed command #1: s <ul><li>$ echo 'foo' | sed 's/foo/bar/' </li></ul><ul><li>bar </li></ul><ul><li>$ echo 'foo foo' | sed 's/foo/bar/' </li></ul><ul><li>bar foo </li></ul>
  56. 56. sed command #1: s <ul><li>$ echo 'foo' | sed 's/foo/bar/' </li></ul><ul><li>bar </li></ul><ul><li>$ echo 'foo foo' | sed 's/foo/bar/' </li></ul><ul><li>bar foo </li></ul><ul><li>'s/foo/bar/g' – global (within line) </li></ul>
  57. 57. Typical uses <ul><li>$ sed 's/foo/bar/g' old </li></ul><ul><li><output> </li></ul>
  58. 58. Typical uses <ul><li>$ sed 's/foo/bar/g' old </li></ul><ul><li><output> </li></ul><ul><li>$ sed 's/foo/bar/g' old > new </li></ul>
  59. 59. Typical uses <ul><li>$ sed 's/foo/bar/g' old </li></ul><ul><li><output> </li></ul><ul><li>$ sed 's/foo/bar/g' old > new </li></ul><ul><li>$ sed –i 's/foo/bar/g' file </li></ul>
  60. 60. Typical uses <ul><li>$ sed 's/foo/bar/g' old </li></ul><ul><li><output> </li></ul><ul><li>$ sed 's/foo/bar/g' old > new </li></ul><ul><li>$ sed –i 's/foo/bar/g' file </li></ul><ul><li>$ <stuff> | xargs sed –i 's/foo/bar/g' </li></ul>
  61. 61. Real life example I <ul><li>Each time I test a batch job, a flag file gets it's only line set to YES , and the job can't be tested again until it is reverted to NO . </li></ul>
  62. 62. Real life example I <ul><li>Each time I test a batch job, a flag file gets it's only line set to YES , and the job can't be tested again until it is reverted to NO . </li></ul><ul><li>$ sed –i 's/YES/NO/' flagfile </li></ul><ul><li>Can change file again with up-arrow. </li></ul><ul><li>No context switch. </li></ul>
  63. 63. Real life example II <ul><li>A bunch of test cases say: </li></ul><ul><li>Assert.assertStuff which could be assertStuff , since using JUnit 3. </li></ul>
  64. 64. Real life example II <ul><li>A bunch of test cases say: </li></ul><ul><li>Assert.assertStuff which could be assertStuff , since using JUnit 3. </li></ul><ul><li>$ find src/test/ -name '*Test.java' </li></ul><ul><li>> | xargs sed –i 's/Assert.assert/assert/' </li></ul>
  65. 65. Real life example III <ul><li>Windows CR-LF is mucking things up. </li></ul>
  66. 66. Real life example III <ul><li>Windows CR-LF is mucking things up. </li></ul><ul><li>$ sed 's/.$//' winfile > unixfile </li></ul><ul><li>Replaces rn with (always inserted) n </li></ul>
  67. 67. Real life example III <ul><li>Windows CR-LF is mucking things up. </li></ul><ul><li>$ sed 's/.$//' winfile > unixfile </li></ul><ul><li>Replaces rn with (always inserted) n </li></ul><ul><li>$ sed 's/$/r/' unixfile > winfile </li></ul><ul><li>Replaces n with rn . </li></ul>
  68. 68. Capturing groups <ul><li>$ echo 'Dog Cat Pig' | sed 's/b(w)/(1)/g' </li></ul>
  69. 69. Capturing groups <ul><li>$ echo 'Dog Cat Pig' | sed 's/b(w)/(1)/g' </li></ul><ul><li>(D)og (C)at (P)ig </li></ul>
  70. 70. Capturing groups <ul><li>$ echo 'Dog Cat Pig' | sed 's/b(w)/(1)/g' </li></ul><ul><li>(D)og (C)at (P)ig </li></ul><ul><li>$ echo 'john doe' | sed 's/b(w)/U1/g' </li></ul>
  71. 71. Capturing groups <ul><li>$ echo 'Dog Cat Pig' | sed 's/b(w)/(1)/g' </li></ul><ul><li>(D)og (C)at (P)ig </li></ul><ul><li>$ echo 'john doe' | sed 's/b(w)/U1/g' </li></ul><ul><li>John Doe </li></ul>
  72. 72. Capturing groups <ul><li>$ echo 'Dog Cat Pig' | sed 's/b(w)/(1)/g' </li></ul><ul><li>(D)og (C)at (P)ig </li></ul><ul><li>$ echo 'john doe' | sed 's/b(w)/U1/g' </li></ul><ul><li>John Doe </li></ul><ul><li>Must escape parenthesis and braces. </li></ul><ul><li>Brackets are not escaped. </li></ul><ul><li>d and + not supported in sed regex. </li></ul>
  73. 73. Exercise: formatting phone #. <ul><li>Convert all strings of 10 digits to (###) ###-####. </li></ul>
  74. 74. Exercise: formatting phone #. <ul><li>Convert all strings of 10 digits to (###) ###-####. </li></ul><ul><li>Conceptually, we want: </li></ul><ul><li>'s/(d{3})(d{3})(d{4})/(1) 2-3/g' </li></ul>
  75. 75. Exercise: formatting phone #. <ul><li>Convert all strings of 10 digits to (###) ###-####. </li></ul><ul><li>Conceptually, we want: </li></ul><ul><li>'s/(d{3})(d{3})(d{4})/(1) 2-3/g' </li></ul><ul><li>In sed regex, that amounts to: </li></ul><ul><li>'s/([0-9]{3})([0-9]{3})([0-9]{4})/(1) 2-3/g' </li></ul>
  76. 76. Exercise: trim whitespace <ul><li>Trim leading whitespace: </li></ul>
  77. 77. Exercise: trim whitespace <ul><li>Trim leading whitespace: </li></ul><ul><li>$ sed -i 's/^[ t]*//' t.txt </li></ul>
  78. 78. Exercise: trim whitespace <ul><li>Trim leading whitespace: </li></ul><ul><li>$ sed -i 's/^[ t]*//' t.txt </li></ul><ul><li>Trim trailing whitespace: </li></ul>
  79. 79. Exercise: trim whitespace <ul><li>Trim leading whitespace: </li></ul><ul><li>$ sed -i 's/^[ t]*//' t.txt </li></ul><ul><li>Trim trailing whitespace: </li></ul><ul><li>$ sed -i 's/[ t]*$//' t.txt </li></ul>
  80. 80. Exercise: trim whitespace <ul><li>Trim leading whitespace: </li></ul><ul><li>$ sed -i 's/^[ t]*//' t.txt </li></ul><ul><li>Trim trailing whitespace: </li></ul><ul><li>$ sed -i 's/[ t]*$//' t.txt </li></ul><ul><li>Trim leading and trailing whitespace: </li></ul>
  81. 81. Exercise: trim whitespace <ul><li>Trim leading whitespace: </li></ul><ul><li>$ sed -i 's/^[ t]*//' t.txt </li></ul><ul><li>Trim trailing whitespace: </li></ul><ul><li>$ sed –i 's/[ t]*$//' t.txt </li></ul><ul><li>Trim leading and trailing whitespace: </li></ul><ul><li>$ sed -i 's/^[ t]*//;s/[ t]*$//' t.txt </li></ul>
  82. 82. Add comment line to file with s : <ul><li>'1s/^/// Copyright FooCorpn/' </li></ul>
  83. 83. Add comment line to file with s : <ul><li>'1s/^/// Copyright FooCorpn/' </li></ul><ul><li>Prepends // Copyright FooCorpn </li></ul><ul><li>1 restricts to first line, similar to vi search. </li></ul><ul><li>^ matches start of line. </li></ul><ul><li>With find & sed insert in all .java files. </li></ul>
  84. 84. Shebang! <ul><li>In my .bashrc : </li></ul><ul><li>function shebang { </li></ul><ul><li>sed –i '1s/^/#!/usr/bin/env pythonnn' $1 </li></ul><ul><li>chmod +x $1 </li></ul><ul><li>} </li></ul><ul><li>Prepends #!/usr/bin/env python and makes file executable </li></ul>
  85. 85. sed command #2: d <ul><li>Delete lines containing foo: </li></ul><ul><li>$ sed –i '/foo/ d' file </li></ul>
  86. 86. sed command #2: d <ul><li>Delete lines containing foo: </li></ul><ul><li>$ sed –i '/foo/ d' file </li></ul><ul><li>Delete lines starting with # : </li></ul><ul><li>$ sed –i '/^#/ d' file </li></ul>
  87. 87. sed command #2: d <ul><li>Delete lines containing foo: </li></ul><ul><li>$ sed –i '/foo/ d' file </li></ul><ul><li>Delete lines starting with # : </li></ul><ul><li>$ sed –i '/^#/ d' file </li></ul><ul><li>Delete first two lines: </li></ul><ul><li>$ sed –i '1,2 d' file </li></ul>
  88. 88. More delete examples: <ul><li>Delete blank lines: </li></ul>
  89. 89. More delete examples: <ul><li>Delete blank lines: </li></ul><ul><li>$ sed '/^$/ d' file </li></ul>
  90. 90. More delete examples: <ul><li>Delete blank lines: </li></ul><ul><li>$ sed '/^$/ d' file </li></ul><ul><li>Delete up to first blank line (email header): </li></ul>
  91. 91. More delete examples: <ul><li>Delete blank lines: </li></ul><ul><li>$ sed '/^$/ d' file </li></ul><ul><li>Delete up to first blank line (email header): </li></ul><ul><li>$ sed '1,/^$/ d' file </li></ul>
  92. 92. More delete examples: <ul><li>Delete blank lines: </li></ul><ul><li>$ sed '/^$/ d' file </li></ul><ul><li>Delete up to first blank line (email header): </li></ul><ul><li>$ sed '1,/^$/ d' file </li></ul><ul><li>Note that we can combine range with regex. </li></ul>
  93. 93. Real life example II, ctd <ul><li>A bunch of test classes have the following unnecessary line: </li></ul><ul><li>import junit.framework.Assert; </li></ul>
  94. 94. Real life example II, ctd <ul><li>A bunch of test classes have the following unnecessary line: </li></ul><ul><li>import junit.framework.Assert; </li></ul><ul><li>$find src/test/ -name *.java | xargs </li></ul><ul><li>> sed -i '/import junit.framework.Assert;/d' </li></ul>
  95. 95. sed summary <ul><li>With only s and d you should probably find a use for sed once a week. </li></ul>
  96. 96. sed summary <ul><li>With only s and d you should probably find a use for sed once a week. </li></ul><ul><li>Combine with find for better results. </li></ul>
  97. 97. sed summary <ul><li>With only s and d you should probably find a use for sed once a week. </li></ul><ul><li>Combine with find for better results. </li></ul><ul><li>sed gets better as your regex improves. </li></ul>
  98. 98. sed summary <ul><li>With only s and d you should probably find a use for sed once a week. </li></ul><ul><li>Combine with find for better results. </li></ul><ul><li>sed gets better as your regex improves. </li></ul><ul><li>Syntax often matches vi . </li></ul>
  99. 99. Part 4: awk <ul><li>A ho, W einberger, K ernighan </li></ul><ul><li>pronounced auk. </li></ul><ul><li>Useful for text-munging. </li></ul>
  100. 100. Simple awk programs <ul><li>$ echo 'Jones 123' | awk '{print $0}' </li></ul><ul><li>Jones 123 </li></ul><ul><li>$ echo 'Jones 123' | awk '{print $1}' </li></ul><ul><li>Jones </li></ul><ul><li>$ echo 'Jones 123' | awk '{print $2}' </li></ul><ul><li>123 </li></ul>
  101. 101. Example server.log file: <ul><li>fcrawler.looksmart.com [26/Apr/2000:00:00:12] &quot;GET </li></ul><ul><li>/contacts.html HTTP/1.0&quot; 200 4595 &quot;-&quot; </li></ul><ul><li>fcrawler.looksmart.com [26/Apr/2000:00:17:19] &quot;GET </li></ul><ul><li>/news/news.html HTTP/1.0&quot; 200 16716 &quot;-&quot; </li></ul><ul><li>ppp931.on.bellglobal.com [26/Apr/2000:00:16:12] &quot;GET </li></ul><ul><li>/download/windows/asctab31.zip HTTP/1.0&quot; 200 1540096 </li></ul><ul><li>&quot;http://www.htmlgoodies.com/downloads/freeware/webdevelopment/15.html&quot; </li></ul><ul><li>123.123.123.123 [26/Apr/2000:00:23:48] &quot;GET /pics/wpaper.gif HTTP/1.0“ </li></ul><ul><li>200 6248 &quot;http://www.jafsoft.com/asctortf/&quot; </li></ul><ul><li>123.123.123.123 [26/Apr/2000:00:23:47] &quot;GET /asctortf/ HTTP/1.0&quot; 200 </li></ul><ul><li>8130 </li></ul><ul><li>&quot;http://search.netscape.com/Computers/Data_Formats/Document/Text/RTF&quot; </li></ul><ul><li>123.123.123.123 [26/Apr/2000:00:23:48] &quot;GET /pics/5star2000.gif </li></ul><ul><li>HTTP/1.0&quot; 200 4005 &quot;http://www.jafsoft.com/asctortf/&quot; </li></ul><ul><li>123.123.123.123 [26/Apr/2000:00:23:50] &quot;GET /pics/5star.gif HTTP/1.0&quot; </li></ul><ul><li>200 1031 &quot;http://www.jafsoft.com/asctortf/&quot; </li></ul><ul><li>123.123.123.123 [26/Apr/2000:00:23:51] &quot;GET /pics/a2hlogo.jpg HTTP/1.0&quot; </li></ul><ul><li>200 4282 &quot;http://www.jafsoft.com/asctortf/&quot; </li></ul><ul><li><snip> </li></ul>
  102. 102. Built-in variables: NF , NR <ul><li>NR – Number of Record </li></ul><ul><li>NF – Number of Fields </li></ul><ul><li>With $ , gives field, otherwise number </li></ul>
  103. 103. Built-in variables: NF , NR <ul><li>NR – Number of Record </li></ul><ul><li>NF – Number of Fields </li></ul><ul><li>With $ , gives field, otherwise number </li></ul><ul><li>$ awk '{print NR, $(NF-2)}' server.log </li></ul><ul><li>1 200 </li></ul><ul><li>2 200 </li></ul>
  104. 104. Structure of an awk program <ul><li>condition { actions } </li></ul>
  105. 105. Structure of an awk program <ul><li>condition { actions } </li></ul><ul><li>$ awk 'END { print NR }' server.log </li></ul>
  106. 106. Structure of an awk program <ul><li>condition { actions } </li></ul><ul><li>$ awk 'END { print NR }' server.log </li></ul><ul><li>9 </li></ul>
  107. 107. Structure of an awk program <ul><li>condition { actions } </li></ul><ul><li>$ awk 'END { print NR }' server.log </li></ul><ul><li>9 </li></ul><ul><li>$ awk '$1 ~ /^[0-9]+.*/ { print $1,$7}' </li></ul><ul><li>> server.log </li></ul>
  108. 108. Structure of an awk program <ul><li>condition { actions } </li></ul><ul><li>$ awk 'END { print NR }' server.log </li></ul><ul><li>9 </li></ul><ul><li>$ awk '$1 ~ /^[0-9]+.*/ { print $1,$7}' </li></ul><ul><li>> server.log </li></ul><ul><li>123.123.123.123 6248 </li></ul><ul><li>123.123.123.123 8130 </li></ul>
  109. 109. Changing delimiter <ul><li>$ awk 'BEGIN {FS = &quot;:&quot;} ; {print $2}' </li></ul>
  110. 110. Changing delimiter <ul><li>$ awk 'BEGIN {FS = &quot;:&quot;} ; {print $2}' </li></ul><ul><li>FS – Field Seperator </li></ul><ul><li>BEGIN and END are special patterns </li></ul>
  111. 111. Changing delimiter <ul><li>$ awk 'BEGIN {FS = &quot;:&quot;} ; {print $2}' </li></ul><ul><li>FS – Field Seperator </li></ul><ul><li>BEGIN and END are special patterns </li></ul><ul><li>Or from the command line: </li></ul><ul><li>$ awk –F: '{ print $2 }' </li></ul>
  112. 112. Get date out of server.log <ul><li>$ awk '{ print $2 }' server.log </li></ul><ul><li>[26/Apr/2000:00:00:12] </li></ul>
  113. 113. Get date out of server.log <ul><li>$ awk '{ print $2 }' server.log </li></ul><ul><li>[26/Apr/2000:00:00:12] </li></ul><ul><li>$ awk '{ print $2 }' server.log </li></ul><ul><li>> | awk –F: '{print $1} </li></ul>
  114. 114. Get date out of server.log <ul><li>$ awk '{ print $2 }' server.log </li></ul><ul><li>[26/Apr/2000:00:00:12] </li></ul><ul><li>$ awk '{ print $2 }' server.log </li></ul><ul><li>> | awk –F: '{print $1} </li></ul><ul><li>[26/Apr/2000 </li></ul>
  115. 115. Get date out of server.log <ul><li>$ awk '{ print $2 }' server.log </li></ul><ul><li>[26/Apr/2000:00:00:12] </li></ul><ul><li>$ awk '{ print $2 }' server.log </li></ul><ul><li>> | awk –F: '{print $1} </li></ul><ul><li>[26/Apr/2000 </li></ul><ul><li>$ awk '{ print $2 }' server.log </li></ul><ul><li>> | awk –F: '{print $1} | sed 's/[//' </li></ul>
  116. 116. Get date out of server.log <ul><li>$ awk '{ print $2 }' server.log </li></ul><ul><li>[26/Apr/2000:00:00:12] </li></ul><ul><li>$ awk '{ print $2 }' server.log </li></ul><ul><li>> | awk –F: '{print $1} </li></ul><ul><li>[26/Apr/2000 </li></ul><ul><li>$ awk '{ print $2 }' server.log </li></ul><ul><li>> | awk –F: '{print $1} | sed 's/[//' </li></ul><ul><li>26/Apr/2000 </li></ul>
  117. 117. Maintaining state in awk <ul><li>Find total bytes transferred from server.log </li></ul>
  118. 118. Maintaining state in awk <ul><li>Find total bytes transferred from server.log </li></ul><ul><li>$ awk '{ b += $(NF-1) } END { print b }' server.log </li></ul><ul><li>1585139 </li></ul>
  119. 119. Maintaining state in awk <ul><li>Find total bytes transferred from server.log </li></ul><ul><li>$ awk '{ b += $(NF-1) } END { print b }' server.log </li></ul><ul><li>1585139 </li></ul><ul><li>Find total bytes transferred to fcrawler </li></ul>
  120. 120. Maintaining state in awk <ul><li>Find total bytes transferred from server.log </li></ul><ul><li>$ awk '{ b += $(NF-1) } END { print b }' server.log </li></ul><ul><li>1585139 </li></ul><ul><li>Find total bytes transferred to fcrawler </li></ul><ul><li>$ awk '$1 ~ /^fcraw.*/ { b += $(NF-1) } END { print b }' </li></ul><ul><li>> server.log </li></ul>
  121. 121. Maintaining state in awk <ul><li>Find total bytes transferred from server.log </li></ul><ul><li>$ awk '{ b += $(NF-1) } END { print b }' server.log </li></ul><ul><li>1585139 </li></ul><ul><li>Find total bytes transferred to fcrawler </li></ul><ul><li>$ awk '$1 ~ /^fcraw.*/ { b += $(NF-1) } END { print b }' </li></ul><ul><li>> server.log </li></ul><ul><li>21311 </li></ul>
  122. 122. One more example <ul><li>Want to eliminate commented out code in large codebase. </li></ul><ul><li>Let's construct a one-liner to identify classes that are more than 50% comments. </li></ul>
  123. 123. One more example <ul><li>Want to eliminate commented out code in large codebase. </li></ul><ul><li>Let's construct a one-liner to identify classes that are more than 50% comments. </li></ul><ul><li>$ awk '$1 == &quot;//&quot; { a+=1 } END { if (a*2 > NR) {print FILENAME, NR, a}}' </li></ul>
  124. 124. One more example <ul><li>Want to eliminate commented out code in large codebase. </li></ul><ul><li>Let's construct a one-liner to identify classes that are more than 50% comments. </li></ul><ul><li>$ awk '$1 == &quot;//&quot; { a+=1 } END { if (a*2 > NR) {print FILENAME, NR, a}}' </li></ul><ul><li>To execute on all Java classes: </li></ul>
  125. 125. Example, ctd. <ul><li>$ find src -name '*.java' -exec awk '$1 == &quot;//&quot; { a+=1 } END { if (a * 2 > NR) {print FILENAME, NR, a}}' {} ; </li></ul>
  126. 126. Example, ctd. <ul><li>$ find src -name '*.java' -exec awk '$1 == &quot;//&quot; { a+=1 } END { if (a * 2 > NR) {print FILENAME, NR, a}}' {} ; </li></ul><ul><li>Here –exec with ; is the right choice, as the awk program is executed for each file individually. </li></ul>
  127. 127. Example, ctd. <ul><li>$ find src -name '*.java' -exec awk '$1 == &quot;//&quot; { a+=1 } END { if (a * 2 > NR) {print FILENAME, NR, a}}' {} ; </li></ul><ul><li>Here –exec with ; is the right choice, as the awk program is executed for each file individually. </li></ul><ul><li>It should be possible to use xargs and FNR , but I'm trying to keep the awk simple. </li></ul>
  128. 128. awk summary <ul><li>NF – Number of Field </li></ul>
  129. 129. awk summary <ul><li>NF – Number of Field </li></ul><ul><li>NR – Number of Records </li></ul>
  130. 130. awk summary <ul><li>NF – Number of Field </li></ul><ul><li>NR – Number of Records </li></ul><ul><li>FILENAME – filename </li></ul>
  131. 131. awk summary <ul><li>NF – Number of Field </li></ul><ul><li>NR – Number of Records </li></ul><ul><li>FILENAME – filename </li></ul><ul><li>BEGIN, END – special events </li></ul>
  132. 132. awk summary <ul><li>NF – Number of Field </li></ul><ul><li>NR – Number of Records </li></ul><ul><li>FILENAME – filename </li></ul><ul><li>BEGIN, END – special events </li></ul><ul><li>FS – Field Seperator (or –F ). </li></ul>
  133. 133. awk summary <ul><li>NF – Number of Field </li></ul><ul><li>NR – Number of Records </li></ul><ul><li>FILENAME – filename </li></ul><ul><li>BEGIN, END – special events </li></ul><ul><li>FS – Field Seperator (or –F ). </li></ul><ul><li>awk 'condition { actions }' </li></ul>
  134. 134. More information <ul><li>To see slides and helpful links, go to: </li></ul><ul><li>http://wilsonericn.wordpress.com </li></ul><ul><li>To find me at Nationwide: </li></ul><ul><li>WILSOE18 </li></ul><ul><li>To find me on twitter: </li></ul><ul><li>@wilsonericn </li></ul>

×