openlab.html 42 KB

12345678910111213141516171819202122232425262728293031323334353637383940414243444546474849505152535455565758596061626364656667686970717273747576777879808182838485868788899091929394959697989910010110210310410510610710810911011111211311411511611711811912012112212312412512612712812913013113213313413513613713813914014114214314414514614714814915015115215315415515615715815916016116216316416516616716816917017117217317417517617717817918018118218318418518618718818919019119219319419519619719819920020120220320420520620720820921021121221321421521621721821922022122222322422522622722822923023123223323423523623723823924024124224324424524624724824925025125225325425525625725825926026126226326426526626726826927027127227327427527627727827928028128228328428528628728828929029129229329429529629729829930030130230330430530630730830931031131231331431531631731831932032132232332432532632732832933033133233333433533633733833934034134234334434534634734834935035135235335435535635735835936036136236336436536636736836937037137237337437537637737837938038138238338438538638738838939039139239339439539639739839940040140240340440540640740840941041141241341441541641741841942042142242342442542642742842943043143243343443543643743843944044144244344444544644744844945045145245345445545645745845946046146246346446546646746846947047147247347447547647747847948048148248348448548648748848949049149249349449549649749849950050150250350450550650750850951051151251351451551651751851952052152252352452552652752852953053153253353453553653753853954054154254354454554654754854955055155255355455555655755855956056156256356456556656756856957057157257357457557657757857958058158258358458558658758858959059159259359459559659759859960060160260360460560660760860961061161261361461561661761861962062162262362462562662762862963063163263363463563663763863964064164264364464564664764864965065165265365465565665765865966066166266366466566666766866967067167267367467567667767867968068168268368468568668768868969069169269369469569669769869970070170270370470570670770870971071171271371471571671771871972072172272372472572672772872973073173273373473573673773873974074174274374474574674774874975075175275375475575675775875976076176276376476576676776876977077177277377477577677777877978078178278378478578678778878979079179279379479579679779879980080180280380480580680780880981081181281381481581681781881982082182282382482582682782882983083183283383483583683783883984084184284384484584684784884985085185285385485585685785885986086186286386486586686786886987087187287387487587687787887988088188288388488588688788888989089189289389489589689789889990090190290390490590690790890991091191291391491591691791891992092192292392492592692792892993093193293393493593693793893994094194294394494594694794894995095195295395495595695795895996096196296396496596696796896997097197297397497597697797897998098198298398498598698798898999099199299399499599699799899910001001100210031004100510061007100810091010101110121013101410151016101710181019102010211022102310241025102610271028102910301031103210331034103510361037103810391040104110421043104410451046104710481049105010511052105310541055105610571058105910601061106210631064106510661067106810691070107110721073107410751076107710781079108010811082108310841085108610871088108910901091109210931094109510961097109810991100110111021103110411051106110711081109111011111112111311141115111611171118111911201121112211231124112511261127112811291130113111321133113411351136113711381139114011411142114311441145114611471148114911501151115211531154115511561157115811591160116111621163116411651166116711681169117011711172117311741175117611771178117911801181118211831184118511861187118811891190119111921193119411951196119711981199120012011202120312041205120612071208120912101211121212131214121512161217121812191220122112221223122412251226122712281229123012311232123312341235123612371238123912401241124212431244124512461247124812491250125112521253125412551256125712581259126012611262126312641265126612671268126912701271127212731274127512761277127812791280128112821283128412851286128712881289129012911292129312941295129612971298129913001301130213031304130513061307130813091310131113121313131413151316131713181319132013211322
  1. <?xml version="1.0" encoding="utf-8"?>
  2. <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  3. "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
  4. <html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
  5. <head>
  6. <title>OpenLab: open tools for physicists</title>
  7. <!-- 2017-05-10 Wed 14:42 -->
  8. <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
  9. <meta name="generator" content="Org-mode" />
  10. <meta name="author" content="Andrea Miglietta" />
  11. <style type="text/css">
  12. <!--/*--><![CDATA[/*><!--*/
  13. .title { text-align: center; }
  14. .todo { font-family: monospace; color: red; }
  15. .done { color: green; }
  16. .tag { background-color: #eee; font-family: monospace;
  17. padding: 2px; font-size: 80%; font-weight: normal; }
  18. .timestamp { color: #bebebe; }
  19. .timestamp-kwd { color: #5f9ea0; }
  20. .right { margin-left: auto; margin-right: 0px; text-align: right; }
  21. .left { margin-left: 0px; margin-right: auto; text-align: left; }
  22. .center { margin-left: auto; margin-right: auto; text-align: center; }
  23. .underline { text-decoration: underline; }
  24. #postamble p, #preamble p { font-size: 90%; margin: .2em; }
  25. p.verse { margin-left: 3%; }
  26. pre {
  27. border: 1px solid #ccc;
  28. box-shadow: 3px 3px 3px #eee;
  29. padding: 8pt;
  30. font-family: monospace;
  31. overflow: auto;
  32. margin: 1.2em;
  33. }
  34. pre.src {
  35. position: relative;
  36. overflow: visible;
  37. padding-top: 1.2em;
  38. }
  39. pre.src:before {
  40. display: none;
  41. position: absolute;
  42. background-color: white;
  43. top: -10px;
  44. right: 10px;
  45. padding: 3px;
  46. border: 1px solid black;
  47. }
  48. pre.src:hover:before { display: inline;}
  49. pre.src-sh:before { content: 'sh'; }
  50. pre.src-bash:before { content: 'sh'; }
  51. pre.src-emacs-lisp:before { content: 'Emacs Lisp'; }
  52. pre.src-R:before { content: 'R'; }
  53. pre.src-perl:before { content: 'Perl'; }
  54. pre.src-java:before { content: 'Java'; }
  55. pre.src-sql:before { content: 'SQL'; }
  56. table { border-collapse:collapse; }
  57. caption.t-above { caption-side: top; }
  58. caption.t-bottom { caption-side: bottom; }
  59. td, th { vertical-align:top; }
  60. th.right { text-align: center; }
  61. th.left { text-align: center; }
  62. th.center { text-align: center; }
  63. td.right { text-align: right; }
  64. td.left { text-align: left; }
  65. td.center { text-align: center; }
  66. dt { font-weight: bold; }
  67. .footpara:nth-child(2) { display: inline; }
  68. .footpara { display: block; }
  69. .footdef { margin-bottom: 1em; }
  70. .figure { padding: 1em; }
  71. .figure p { text-align: center; }
  72. .inlinetask {
  73. padding: 10px;
  74. border: 2px solid gray;
  75. margin: 10px;
  76. background: #ffffcc;
  77. }
  78. #org-div-home-and-up
  79. { text-align: right; font-size: 70%; white-space: nowrap; }
  80. textarea { overflow-x: auto; }
  81. .linenr { font-size: smaller }
  82. .code-highlighted { background-color: #ffff00; }
  83. .org-info-js_info-navigation { border-style: none; }
  84. #org-info-js_console-label
  85. { font-size: 10px; font-weight: bold; white-space: nowrap; }
  86. .org-info-js_search-highlight
  87. { background-color: #ffff00; color: #000000; font-weight: bold; }
  88. /*]]>*/-->
  89. </style>
  90. <script type="text/javascript">
  91. /*
  92. @licstart The following is the entire license notice for the
  93. JavaScript code in this tag.
  94. Copyright (C) 2012-2013 Free Software Foundation, Inc.
  95. The JavaScript code in this tag is free software: you can
  96. redistribute it and/or modify it under the terms of the GNU
  97. General Public License (GNU GPL) as published by the Free Software
  98. Foundation, either version 3 of the License, or (at your option)
  99. any later version. The code is distributed WITHOUT ANY WARRANTY;
  100. without even the implied warranty of MERCHANTABILITY or FITNESS
  101. FOR A PARTICULAR PURPOSE. See the GNU GPL for more details.
  102. As additional permission under GNU GPL version 3 section 7, you
  103. may distribute non-source (e.g., minimized or compacted) forms of
  104. that code without the copy of the GNU GPL normally required by
  105. section 4, provided you include this license notice and a URL
  106. through which recipients can access the Corresponding Source.
  107. @licend The above is the entire license notice
  108. for the JavaScript code in this tag.
  109. */
  110. <!--/*--><![CDATA[/*><!--*/
  111. function CodeHighlightOn(elem, id)
  112. {
  113. var target = document.getElementById(id);
  114. if(null != target) {
  115. elem.cacheClassElem = elem.className;
  116. elem.cacheClassTarget = target.className;
  117. target.className = "code-highlighted";
  118. elem.className = "code-highlighted";
  119. }
  120. }
  121. function CodeHighlightOff(elem, id)
  122. {
  123. var target = document.getElementById(id);
  124. if(elem.cacheClassElem)
  125. elem.className = elem.cacheClassElem;
  126. if(elem.cacheClassTarget)
  127. target.className = elem.cacheClassTarget;
  128. }
  129. /*]]>*///-->
  130. </script>
  131. </head>
  132. <body>
  133. <div id="content">
  134. <h1 class="title">OpenLab: open tools for physicists</h1>
  135. <div id="table-of-contents">
  136. <h2>Table of Contents</h2>
  137. <div id="text-table-of-contents">
  138. <ul>
  139. <li><a href="#sec-1">1. Introduction</a>
  140. <ul>
  141. <li><a href="#sec-1-1">1.1. Presentation of me:</a></li>
  142. <li><a href="#sec-1-2">1.2. Goals of this conference:</a></li>
  143. <li><a href="#sec-1-3">1.3. Quick review of things I'd like to assume you know:</a>
  144. <ul>
  145. <li><a href="#sec-1-3-1">1.3.1. Open a terminal</a></li>
  146. <li><a href="#sec-1-3-2">1.3.2. Open a file with an editor of your choice</a></li>
  147. <li><a href="#sec-1-3-3">1.3.3. Elementary commands:</a></li>
  148. <li><a href="#sec-1-3-4">1.3.4. Standard input, output and error</a></li>
  149. <li><a href="#sec-1-3-5">1.3.5. Difference between absolute and relative path</a></li>
  150. <li><a href="#sec-1-3-6">1.3.6. How to get help?</a></li>
  151. </ul>
  152. </li>
  153. <li><a href="#sec-1-4">1.4. How can I improve my skills?</a></li>
  154. </ul>
  155. </li>
  156. <li><a href="#sec-2">2. Start learning Bash</a>
  157. <ul>
  158. <li><a href="#sec-2-1">2.1. What is Bash?</a></li>
  159. <li><a href="#sec-2-2">2.2. Why using Bash?</a>
  160. <ul>
  161. <li><a href="#sec-2-2-1">2.2.1. How to verify which shell you are using?</a></li>
  162. </ul>
  163. </li>
  164. <li><a href="#sec-2-3">2.3. Bash scripting</a>
  165. <ul>
  166. <li><a href="#sec-2-3-1">2.3.1. What is a script?</a></li>
  167. <li><a href="#sec-2-3-2">2.3.2. How can I run my script?</a></li>
  168. <li><a href="#sec-2-3-3">2.3.3. First step: Hello World!</a></li>
  169. <li><a href="#sec-2-3-4">2.3.4. Why is important quoting variables?</a></li>
  170. <li><a href="#sec-2-3-5">2.3.5. Command substitutuion: backticks vs $()</a></li>
  171. <li><a href="#sec-2-3-6">2.3.6. Why is dangerous shell scripting?</a></li>
  172. <li><a href="#sec-2-3-7">2.3.7. Use bashisms: use Bash, don't half-use it.</a></li>
  173. <li><a href="#sec-2-3-8">2.3.8. How to debug a script:</a></li>
  174. </ul>
  175. </li>
  176. <li><a href="#sec-2-4">2.4. 7 personal advice to write a good script:</a></li>
  177. </ul>
  178. </li>
  179. <li><a href="#sec-3">3. One step further: how to schedule our operations</a>
  180. <ul>
  181. <li><a href="#sec-3-1">3.1. Cron:</a>
  182. <ul>
  183. <li><a href="#sec-3-1-1">3.1.1. What is cron?</a></li>
  184. <li><a href="#sec-3-1-2">3.1.2. How can I schedule use cron?</a></li>
  185. </ul>
  186. </li>
  187. <li><a href="#sec-3-2">3.2. Systemd timers:</a>
  188. <ul>
  189. <li><a href="#sec-3-2-1">3.2.1. What is a systemd timer?</a></li>
  190. <li><a href="#sec-3-2-2">3.2.2. How can I create a timer?</a></li>
  191. </ul>
  192. </li>
  193. <li><a href="#sec-3-3">3.3. Exercises:</a></li>
  194. </ul>
  195. </li>
  196. <li><a href="#sec-4">4. How to 'rename' large number of files?</a></li>
  197. <li><a href="#sec-5">5. How to 'find' and do operations over multiple files?</a></li>
  198. <li><a href="#sec-6">6. Text processing by examples: awk</a>
  199. <ul>
  200. <li>
  201. <ul>
  202. <li><a href="#sec-6-0-1">6.0.1. The worst thing you can do with Awk! (imho)</a></li>
  203. <li><a href="#sec-6-0-2">6.0.2. Print lines based on pattern</a></li>
  204. <li><a href="#sec-6-0-3">6.0.3. Print lines in range</a></li>
  205. <li><a href="#sec-6-0-4">6.0.4. Remove (non consecutive) duplicated lines</a></li>
  206. <li><a href="#sec-6-0-5">6.0.5. Find and replace</a></li>
  207. <li><a href="#sec-6-0-6">6.0.6. Print intersection of two files</a></li>
  208. <li><a href="#sec-6-0-7">6.0.7. Parse a CSV file</a></li>
  209. </ul>
  210. </li>
  211. </ul>
  212. </li>
  213. <li><a href="#sec-7">7. Project example: dice rolling simulations</a></li>
  214. <li><a href="#sec-8">8. Not covered by this talk but truly useful and important:</a>
  215. <ul>
  216. <li><a href="#sec-8-1">8.1. ssh</a>
  217. <ul>
  218. <li><a href="#sec-8-1-1">8.1.1. Generate public/private key pair for authentication</a></li>
  219. <li><a href="#sec-8-1-2">8.1.2. Basic configuration example</a></li>
  220. </ul>
  221. </li>
  222. <li><a href="#sec-8-2">8.2. git</a></li>
  223. <li><a href="#sec-8-3">8.3. tmux</a></li>
  224. </ul>
  225. </li>
  226. </ul>
  227. </div>
  228. </div>
  229. <div id="outline-container-sec-1" class="outline-2">
  230. <h2 id="sec-1"><span class="section-number-2">1</span> Introduction</h2>
  231. <div class="outline-text-2" id="text-1">
  232. </div><div id="outline-container-sec-1-1" class="outline-3">
  233. <h3 id="sec-1-1"><span class="section-number-3">1.1</span> Presentation of me:</h3>
  234. <div class="outline-text-3" id="text-1-1">
  235. <ul class="org-ul">
  236. <li>Who I am?
  237. </li>
  238. <li>My contacts:
  239. <ol class="org-ol">
  240. <li>andrea.miglietta92@gmail.com
  241. </li>
  242. <li>andreatsh@lcm.mi.infn.it
  243. </li>
  244. </ol>
  245. </li>
  246. <li>GitHub: <a href="https://github.com/andreatsh/openlab">https://github.com/andreatsh/openlab</a>
  247. </li>
  248. </ul>
  249. </div>
  250. </div>
  251. <div id="outline-container-sec-1-2" class="outline-3">
  252. <h3 id="sec-1-2"><span class="section-number-3">1.2</span> Goals of this conference:</h3>
  253. <div class="outline-text-3" id="text-1-2">
  254. <p>
  255. Why using command line?
  256. Why bother?
  257. Why do I have to waste my time to learn command line?
  258. </p>
  259. <p>
  260. When you realize that you spend most of your time doing repetitive tasks, you
  261. could suspect you are not doing things quite well. What you want is a way to
  262. perform repetitive tasks with less typing and actions as possible.
  263. Stop waste your time with boring routine stuff!
  264. </p>
  265. <p>
  266. Moreover, you can have a good comprehension and a fine control of what you
  267. are actually doing. There are lots of tasks you can perfom easily with
  268. command line that in other way are particular complicated.
  269. If you do not believe it, try to implement a program (for example in C++)
  270. that:
  271. </p>
  272. <ul class="org-ul">
  273. <li>search all your files into a directory
  274. </li>
  275. <li>evaluate the disk space occupied by each file
  276. </li>
  277. <li>order by size the results
  278. </li>
  279. <li>print them on a file
  280. </li>
  281. </ul>
  282. <p>
  283. Once you have done, compare the time spent and the length of your code with
  284. this simple line:
  285. </p>
  286. <div class="org-src-container">
  287. <pre class="src src-bash">du -sh * | sort -nr &gt; file.results
  288. </pre>
  289. </div>
  290. <p>
  291. If you are not convinced yet, please teach me your way to code in C++!
  292. Seriously!
  293. </p>
  294. </div>
  295. </div>
  296. <div id="outline-container-sec-1-3" class="outline-3">
  297. <h3 id="sec-1-3"><span class="section-number-3">1.3</span> Quick review of things I'd like to assume you know:</h3>
  298. <div class="outline-text-3" id="text-1-3">
  299. </div><div id="outline-container-sec-1-3-1" class="outline-4">
  300. <h4 id="sec-1-3-1"><span class="section-number-4">1.3.1</span> Open a terminal</h4>
  301. </div>
  302. <div id="outline-container-sec-1-3-2" class="outline-4">
  303. <h4 id="sec-1-3-2"><span class="section-number-4">1.3.2</span> Open a file with an editor of your choice</h4>
  304. <div class="outline-text-4" id="text-1-3-2">
  305. <p>
  306. Choose your tools wisely and spend time to customize them.
  307. There are many text editor, some example:
  308. </p>
  309. </div>
  310. <ol class="org-ol"><li><a id="sec-1-3-2-1" name="sec-1-3-2-1"></a>Emacs, Vim<br /><div class="outline-text-5" id="text-1-3-2-1">
  311. <p>
  312. Advanced and powerful editors. (Recommended)
  313. </p>
  314. </div>
  315. </li>
  316. <li><a id="sec-1-3-2-2" name="sec-1-3-2-2"></a>Nano<br /></li>
  317. <li><a id="sec-1-3-2-3" name="sec-1-3-2-3"></a>Gedit<br /></li></ol>
  318. </div>
  319. <div id="outline-container-sec-1-3-3" class="outline-4">
  320. <h4 id="sec-1-3-3"><span class="section-number-4">1.3.3</span> Elementary commands:</h4>
  321. <div class="outline-text-4" id="text-1-3-3">
  322. <ul class="org-ul">
  323. <li>cd
  324. </li>
  325. <li>ls
  326. </li>
  327. <li>cp
  328. </li>
  329. <li>mv
  330. </li>
  331. <li>rm
  332. </li>
  333. <li>mkdir
  334. </li>
  335. </ul>
  336. </div>
  337. </div>
  338. <div id="outline-container-sec-1-3-4" class="outline-4">
  339. <h4 id="sec-1-3-4"><span class="section-number-4">1.3.4</span> Standard input, output and error</h4>
  340. <div class="outline-text-4" id="text-1-3-4">
  341. </div><ol class="org-ol"><li><a id="sec-1-3-4-1" name="sec-1-3-4-1"></a>&gt;<br /><div class="outline-text-5" id="text-1-3-4-1">
  342. <p>
  343. Redirect stdout to a file.
  344. Creates the file if not present, otherwise overwrites it.
  345. </p>
  346. </div>
  347. </li>
  348. <li><a id="sec-1-3-4-2" name="sec-1-3-4-2"></a>&gt;&gt;<br /><div class="outline-text-5" id="text-1-3-4-2">
  349. <p>
  350. Redirect stdout to a file.
  351. Creates the file if not present, otherwise appends to it.
  352. </p>
  353. </div>
  354. </li>
  355. <li><a id="sec-1-3-4-3" name="sec-1-3-4-3"></a>1&gt;<br /><div class="outline-text-5" id="text-1-3-4-3">
  356. <p>
  357. Redirect stdout
  358. </p>
  359. </div>
  360. </li>
  361. <li><a id="sec-1-3-4-4" name="sec-1-3-4-4"></a>1&gt;&gt;<br /><div class="outline-text-5" id="text-1-3-4-4">
  362. <p>
  363. Redirect and append stdout
  364. </p>
  365. </div>
  366. </li>
  367. <li><a id="sec-1-3-4-5" name="sec-1-3-4-5"></a>2&gt;<br /><div class="outline-text-5" id="text-1-3-4-5">
  368. <p>
  369. Redirect stderr
  370. </p>
  371. </div>
  372. </li>
  373. <li><a id="sec-1-3-4-6" name="sec-1-3-4-6"></a>2&gt;&gt;<br /><div class="outline-text-5" id="text-1-3-4-6">
  374. <p>
  375. Redirect and append stderr
  376. </p>
  377. </div>
  378. </li>
  379. <li><a id="sec-1-3-4-7" name="sec-1-3-4-7"></a>&amp;&gt;<br /><div class="outline-text-5" id="text-1-3-4-7">
  380. <p>
  381. Redirect both stdout and stderr
  382. </p>
  383. </div>
  384. </li>
  385. <li><a id="sec-1-3-4-8" name="sec-1-3-4-8"></a>2&gt;&amp;1<br /><div class="outline-text-5" id="text-1-3-4-8">
  386. <p>
  387. Redirect both stdout and stderr
  388. </p>
  389. </div>
  390. </li>
  391. <li><a id="sec-1-3-4-9" name="sec-1-3-4-9"></a>What is /dev/null?<br /></li></ol>
  392. </div>
  393. <div id="outline-container-sec-1-3-5" class="outline-4">
  394. <h4 id="sec-1-3-5"><span class="section-number-4">1.3.5</span> Difference between absolute and relative path</h4>
  395. </div>
  396. <div id="outline-container-sec-1-3-6" class="outline-4">
  397. <h4 id="sec-1-3-6"><span class="section-number-4">1.3.6</span> How to get help?</h4>
  398. <div class="outline-text-4" id="text-1-3-6">
  399. </div><ol class="org-ol"><li><a id="sec-1-3-6-1" name="sec-1-3-6-1"></a>man [command]<br /><div class="outline-text-5" id="text-1-3-6-1">
  400. <p>
  401. Show the complete manual page of the command
  402. </p>
  403. </div>
  404. </li>
  405. <li><a id="sec-1-3-6-2" name="sec-1-3-6-2"></a>&#x2013;help, -h<br /><div class="outline-text-5" id="text-1-3-6-2">
  406. <p>
  407. These flags are provided by almost every commands and show the basic usage
  408. </p>
  409. </div>
  410. </li>
  411. <li><a id="sec-1-3-6-3" name="sec-1-3-6-3"></a>GIYF<br /><div class="outline-text-5" id="text-1-3-6-3">
  412. <p>
  413. Google Is Your Friend
  414. </p>
  415. </div>
  416. </li></ol>
  417. </div>
  418. </div>
  419. <div id="outline-container-sec-1-4" class="outline-3">
  420. <h3 id="sec-1-4"><span class="section-number-3">1.4</span> How can I improve my skills?</h3>
  421. <div class="outline-text-3" id="text-1-4">
  422. <p>
  423. A lot of study and practice, practice and practice again!
  424. There are no other ways to gain confidence with command line.
  425. More confidence you have, less time you spend on silly routine tasks!
  426. </p>
  427. </div>
  428. </div>
  429. </div>
  430. <div id="outline-container-sec-2" class="outline-2">
  431. <h2 id="sec-2"><span class="section-number-2">2</span> Start learning Bash</h2>
  432. <div class="outline-text-2" id="text-2">
  433. </div><div id="outline-container-sec-2-1" class="outline-3">
  434. <h3 id="sec-2-1"><span class="section-number-3">2.1</span> What is Bash?</h3>
  435. <div class="outline-text-3" id="text-2-1">
  436. <p>
  437. Bash is a Unix shell and command language.
  438. </p>
  439. </div>
  440. </div>
  441. <div id="outline-container-sec-2-2" class="outline-3">
  442. <h3 id="sec-2-2"><span class="section-number-3">2.2</span> Why using Bash?</h3>
  443. <div class="outline-text-3" id="text-2-2">
  444. <p>
  445. Bash is the standard GNU shell. It's the default shell on most Linux
  446. distributions, so your script will virtually works everywhere.
  447. Bash is intuitive for beginner users and at the same time is a powerful tool
  448. for advavenced and professional users.
  449. </p>
  450. </div>
  451. <div id="outline-container-sec-2-2-1" class="outline-4">
  452. <h4 id="sec-2-2-1"><span class="section-number-4">2.2.1</span> How to verify which shell you are using?</h4>
  453. <div class="outline-text-4" id="text-2-2-1">
  454. <p>
  455. There are many shells available (bash, sh, zsh, ksh, fish, etc..).
  456. Typing commands for a shell when you are running another one can cause
  457. confusion, errors or unwanted behaviors.
  458. To identify your default shell open a terminal and type:
  459. </p>
  460. <div class="org-src-container">
  461. <pre class="src src-bash">echo "$SHELL"
  462. </pre>
  463. </div>
  464. <p>
  465. /bin/bash
  466. </p>
  467. </div>
  468. </div>
  469. </div>
  470. <div id="outline-container-sec-2-3" class="outline-3">
  471. <h3 id="sec-2-3"><span class="section-number-3">2.3</span> Bash scripting</h3>
  472. <div class="outline-text-3" id="text-2-3">
  473. </div><div id="outline-container-sec-2-3-1" class="outline-4">
  474. <h4 id="sec-2-3-1"><span class="section-number-4">2.3.1</span> What is a script?</h4>
  475. <div class="outline-text-4" id="text-2-3-1">
  476. <p>
  477. Shell scripts are interpreted, not compiled. So in few words, a shell script
  478. is just a sequence of instructions that the shell reads from the script line
  479. by line and executes them.
  480. </p>
  481. </div>
  482. <ol class="org-ol"><li><a id="sec-2-3-1-1" name="sec-2-3-1-1"></a>Shebang: #!<br /></li></ol>
  483. </div>
  484. <div id="outline-container-sec-2-3-2" class="outline-4">
  485. <h4 id="sec-2-3-2"><span class="section-number-4">2.3.2</span> How can I run my script?</h4>
  486. <div class="outline-text-4" id="text-2-3-2">
  487. </div><ol class="org-ol"><li><a id="sec-2-3-2-1" name="sec-2-3-2-1"></a>Make a file executable<br /><div class="outline-text-5" id="text-2-3-2-1">
  488. <p>
  489. The permissions of a file granted to a user are:
  490. </p>
  491. <ul class="org-ul">
  492. <li>read (<b>r</b>)
  493. </li>
  494. <li>write (<b>w</b>)
  495. </li>
  496. <li>execute (<b>x</b>)
  497. </li>
  498. </ul>
  499. <p>
  500. You have to assign the execute attribute to your script:
  501. </p>
  502. <div class="org-src-container">
  503. <pre class="src src-bash">chmod +x script.sh
  504. </pre>
  505. </div>
  506. </div>
  507. </li></ol>
  508. </div>
  509. <div id="outline-container-sec-2-3-3" class="outline-4">
  510. <h4 id="sec-2-3-3"><span class="section-number-4">2.3.3</span> First step: Hello World!</h4>
  511. <div class="outline-text-4" id="text-2-3-3">
  512. </div><ol class="org-ol"><li><a id="sec-2-3-3-1" name="sec-2-3-3-1"></a>A simple "Hello World!"<br /><div class="outline-text-5" id="text-2-3-3-1">
  513. <div class="org-src-container">
  514. <pre class="src src-bash">#!/bin/bash
  515. echo "Hello World!"
  516. </pre>
  517. </div>
  518. </div>
  519. </li>
  520. <li><a id="sec-2-3-3-2" name="sec-2-3-3-2"></a>Another simple "Hello World!"<br /><div class="outline-text-5" id="text-2-3-3-2">
  521. <div class="org-src-container">
  522. <pre class="src src-bash">#!/bin/bash
  523. # This is a variable
  524. HELLO="Hello World!"
  525. echo "$HELLO"
  526. </pre>
  527. </div>
  528. </div>
  529. </li></ol>
  530. </div>
  531. <div id="outline-container-sec-2-3-4" class="outline-4">
  532. <h4 id="sec-2-3-4"><span class="section-number-4">2.3.4</span> Why is important quoting variables?</h4>
  533. <div class="outline-text-4" id="text-2-3-4">
  534. <p>
  535. It's a good practice to quote your variables.
  536. </p>
  537. </div>
  538. <ol class="org-ol"><li><a id="sec-2-3-4-1" name="sec-2-3-4-1"></a>Differences between using single quotes or double quotes<br /><ol class="org-ol"><li><a id="sec-2-3-4-1-1" name="sec-2-3-4-1-1"></a>Example 1<br /><div class="outline-text-6" id="text-2-3-4-1-1">
  539. <ul class="org-ul">
  540. <li>Single quotes example:
  541. <div class="org-src-container">
  542. <pre class="src src-sh">echo 'This is my $HOME'
  543. </pre>
  544. </div>
  545. <p>
  546. This is my $HOME
  547. </p>
  548. </li>
  549. <li>Double quotes example:
  550. <div class="org-src-container">
  551. <pre class="src src-sh">echo "This is my $HOME"
  552. </pre>
  553. </div>
  554. <p>
  555. This is my /home/andreatsh
  556. </p>
  557. </li>
  558. </ul>
  559. </div>
  560. </li>
  561. <li><a id="sec-2-3-4-1-2" name="sec-2-3-4-1-2"></a>Example 2<br /><div class="outline-text-6" id="text-2-3-4-1-2">
  562. <ul class="org-ul">
  563. <li>Single quotes:
  564. <div class="org-src-container">
  565. <pre class="src src-sh">[ '$HOME' =='/home/andreatsh' ] &amp;&amp; echo "Yuppi" || echo "Error"
  566. </pre>
  567. </div>
  568. <p>
  569. Error
  570. </p>
  571. </li>
  572. <li>Example with double quotes:
  573. <div class="org-src-container">
  574. <pre class="src src-sh">[ "$HOME" == "/home/andreatsh" ] &amp;&amp; echo "Yuppi" || echo "Error`"
  575. </pre>
  576. </div>
  577. <p>
  578. Yuppi
  579. </p>
  580. </li>
  581. </ul>
  582. </div>
  583. </li></ol>
  584. </li>
  585. <li><a id="sec-2-3-4-2" name="sec-2-3-4-2"></a>A bit more advanced example:<br /><div class="outline-text-5" id="text-2-3-4-2">
  586. <div class="org-src-container">
  587. <pre class="src src-bash">#!/bin/bash
  588. # Personal example of why quoting is important.
  589. # What output do you expect? You might be surprised.
  590. HELLO=Hello!; ls -l
  591. echo "$HELLO"
  592. # This is a script designed to learn, you can run it and nothing bad will
  593. # happen. But if you do not write your script with care and good practices
  594. # (even on-the-fly scripts) can do real damages!
  595. </pre>
  596. </div>
  597. </div>
  598. </li></ol>
  599. </div>
  600. <div id="outline-container-sec-2-3-5" class="outline-4">
  601. <h4 id="sec-2-3-5"><span class="section-number-4">2.3.5</span> Command substitutuion: backticks vs $()</h4>
  602. <div class="outline-text-4" id="text-2-3-5">
  603. </div><ol class="org-ol"><li><a id="sec-2-3-5-1" name="sec-2-3-5-1"></a>From Bash manual page:<br /><div class="outline-text-5" id="text-2-3-5-1">
  604. <blockquote>
  605. <p>
  606. Command substitution allows the output of a command to replace the command
  607. name. There are two forms:
  608. </p>
  609. </blockquote>
  610. <p class="verse">
  611. &#xa0;&#xa0;&#xa0;&#xa0;$(command)<br />
  612. or<br />
  613. &#xa0;&#xa0;&#xa0;&#xa0;`command`<br />
  614. </p>
  615. <blockquote>
  616. <p>
  617. Bash performs the expansion by executing command in a subshell environment
  618. and replacing the command substitution with the standard output of the
  619. command, with any trailing newlines deleted. Embedded newlines are not
  620. deleted, but they may be removed during word splitting.
  621. The command substitution $(cat file) can be replaced by the equivalent but
  622. faster $(&lt; file). <br />
  623. </p>
  624. <p>
  625. When the old-style backquote form of substitution is used, backslash retains
  626. its literal meaning except when followed by $, `, or \. The first backquote
  627. not preceded by a backslash terminates the command substitution. When using
  628. the $(command) form, all characters between the parentheses make up the
  629. command; none are treated specially. <br />
  630. </p>
  631. <p>
  632. Command substitutions may be nested. To nest when using the backquoted form,
  633. escape the inner backquotes with backslashes. <br />
  634. </p>
  635. <p>
  636. If the substitution appears within double quotes, word splitting and
  637. pathname expansion are not performed on the results.
  638. </p>
  639. </blockquote>
  640. </div>
  641. </li>
  642. <li><a id="sec-2-3-5-2" name="sec-2-3-5-2"></a>Examples:<br /><div class="outline-text-5" id="text-2-3-5-2">
  643. <ol class="org-ol">
  644. <li>Simple use of command substitutuions:
  645. <div class="org-src-container">
  646. <pre class="src src-sh">ex1=`echo ciao`; echo "$ex1"
  647. ex1=$(echo ciao); echo "$ex1"
  648. </pre>
  649. </div>
  650. </li>
  651. <li>Nested command substitutions:
  652. <div class="org-src-container">
  653. <pre class="src src-sh">ex2=`echo `ls``; echo "$ex2" # WRONG!
  654. ex2=`echo \`ls\``; echo "$ex2" # RIGHT
  655. ex2=$(echo $(ls)); echo "$ex2" # RIGHT
  656. </pre>
  657. </div>
  658. </li>
  659. </ol>
  660. </div>
  661. </li></ol>
  662. </div>
  663. <div id="outline-container-sec-2-3-6" class="outline-4">
  664. <h4 id="sec-2-3-6"><span class="section-number-4">2.3.6</span> Why is dangerous shell scripting?</h4>
  665. <div class="outline-text-4" id="text-2-3-6">
  666. <p>
  667. Many reasons. An example is worth a thousand words:
  668. Example:
  669. </p>
  670. <div class="org-src-container">
  671. <pre class="src src-bash"># This is an example of really bad code!
  672. MYDIR=
  673. rm -rf "$MYDIR/"*
  674. # MYDIR variable may not be defined or be an empty string for many reasons!
  675. # You just ran "rm -rf /*", maybe even as superuser!
  676. </pre>
  677. </div>
  678. </div>
  679. </div>
  680. <div id="outline-container-sec-2-3-7" class="outline-4">
  681. <h4 id="sec-2-3-7"><span class="section-number-4">2.3.7</span> Use bashisms: use Bash, don't half-use it.</h4>
  682. <div class="outline-text-4" id="text-2-3-7">
  683. <ul class="org-ul">
  684. <li>parameters expansions
  685. </li>
  686. <li>local and readonly variables
  687. </li>
  688. <li>improved conditional expressions
  689. </li>
  690. </ul>
  691. </div>
  692. </div>
  693. <div id="outline-container-sec-2-3-8" class="outline-4">
  694. <h4 id="sec-2-3-8"><span class="section-number-4">2.3.8</span> How to debug a script:</h4>
  695. <div class="outline-text-4" id="text-2-3-8">
  696. <p>
  697. I'm used to write very carefully my script, even on-the-fly or one-time
  698. scripts. However, there is always something that could be have undesired
  699. behaviors. To debug my Bash script I find useful this way to proceed:
  700. </p>
  701. <div class="org-src-container">
  702. <pre class="src src-bash"># Debug mode: ON
  703. set -x
  704. # ...
  705. # Your script goes here!
  706. # ...
  707. # Debug mode: OFF
  708. set +x
  709. </pre>
  710. </div>
  711. </div>
  712. </div>
  713. </div>
  714. <div id="outline-container-sec-2-4" class="outline-3">
  715. <h3 id="sec-2-4"><span class="section-number-3">2.4</span> 7 personal advice to write a good script:</h3>
  716. <div class="outline-text-3" id="text-2-4">
  717. <p>
  718. I summarize here the main recommendations that will be used in this
  719. document. I think it's a good habit to use them all the time, even just
  720. for single time scripts. A well documented, structered and organized,
  721. versioned collection of scripts will bring you many benefits over time.
  722. </p>
  723. <ol class="org-ol">
  724. <li>Comment your code!
  725. Use comments to explain why are you doing something more than what
  726. you are doing, and write them in english! Do not underestimate the
  727. importance of english in writing code! You don't write production
  728. scripts for yourself! The whole team need to understand what you have
  729. done and why.
  730. </li>
  731. <li>Choose descriptive names for variables and functions!
  732. </li>
  733. <li>Try to write reliable and reusable code.
  734. In this way is easier to make changes and updates without break the
  735. application, and you can rearrange in a jiffy functions to fit your
  736. temporary needs instead of rewrite code from scratch.
  737. </li>
  738. <li>Don't use a complex construct where a simpler one will do.
  739. Complex construct are difficult to understand and decrease readability.
  740. </li>
  741. <li>Break complex scripts into simpler modules. Use functions where appropriate.
  742. </li>
  743. <li>Log what your script is doing!
  744. Make sure to always known what your scripts are doing.
  745. In my experience the correct way is to use standard error for logging,
  746. so you can redirect your simulation's output (i.e. your data) into a file
  747. without mix them with useful (and hopefully verbose) log messages.
  748. </li>
  749. <li>Last but not least: Keep your code under versioning!
  750. This is really important to me! Personally I keep under versioning even
  751. my filesystem configurations directories, like those in /etc.
  752. Making mistakes or having unwanted behavior is easy with shell scripts,
  753. as we will see soon in this document. A repository is really usefull to
  754. review your code and have structured logs of what you have done and
  755. bugs fixed.
  756. </li>
  757. </ol>
  758. </div>
  759. </div>
  760. </div>
  761. <div id="outline-container-sec-3" class="outline-2">
  762. <h2 id="sec-3"><span class="section-number-2">3</span> One step further: how to schedule our operations</h2>
  763. <div class="outline-text-2" id="text-3">
  764. <p>
  765. It's often useful run a script periodically, to do some cleanup, monitor a
  766. simulation, or backup your files. To take efficiency one step further, it
  767. would be nice if we could not sit in front of our computers and run these
  768. scritps manually. You can get this job done by using cron or systemd-timers.
  769. Here is a brief introduction to these topics.
  770. </p>
  771. </div>
  772. <div id="outline-container-sec-3-1" class="outline-3">
  773. <h3 id="sec-3-1"><span class="section-number-3">3.1</span> Cron:</h3>
  774. <div class="outline-text-3" id="text-3-1">
  775. </div><div id="outline-container-sec-3-1-1" class="outline-4">
  776. <h4 id="sec-3-1-1"><span class="section-number-4">3.1.1</span> What is cron?</h4>
  777. <div class="outline-text-4" id="text-3-1-1">
  778. <p>
  779. <i>cron</i> is a daemon to execute scheduled commands.
  780. </p>
  781. </div>
  782. </div>
  783. <div id="outline-container-sec-3-1-2" class="outline-4">
  784. <h4 id="sec-3-1-2"><span class="section-number-4">3.1.2</span> How can I schedule use cron?</h4>
  785. <div class="outline-text-4" id="text-3-1-2">
  786. <p>
  787. Each user can have his own crontab file in <i>var/spool/cron/crontabs</i>,
  788. which can be created/modified/removed by <i>crontab</i> command.
  789. The crontab file should not be accessed directly, the <i>crontab</i> command
  790. should be used to access and update it.
  791. To edit (or create) your crontab:
  792. </p>
  793. <div class="org-src-container">
  794. <pre class="src src-bash">crontab -e
  795. </pre>
  796. </div>
  797. <p>
  798. To list your current crontab:
  799. </p>
  800. <div class="org-src-container">
  801. <pre class="src src-bash">crontab -l
  802. </pre>
  803. </div>
  804. <p>
  805. Use 'man cron' and 'man crontab' to find all the documentation you need.
  806. </p>
  807. </div>
  808. </div>
  809. </div>
  810. <div id="outline-container-sec-3-2" class="outline-3">
  811. <h3 id="sec-3-2"><span class="section-number-3">3.2</span> Systemd timers:</h3>
  812. <div class="outline-text-3" id="text-3-2">
  813. <p>
  814. This is an advanced topic for this conference. If you don't know what SystemD is, don't worry,
  815. you can skip this part without losing anything.
  816. </p>
  817. </div>
  818. <div id="outline-container-sec-3-2-1" class="outline-4">
  819. <h4 id="sec-3-2-1"><span class="section-number-4">3.2.1</span> What is a systemd timer?</h4>
  820. <div class="outline-text-4" id="text-3-2-1">
  821. <p>
  822. Timers are systemd unit files with a suffix of <i>.timer</i>.
  823. </p>
  824. </div>
  825. </div>
  826. <div id="outline-container-sec-3-2-2" class="outline-4">
  827. <h4 id="sec-3-2-2"><span class="section-number-4">3.2.2</span> How can I create a timer?</h4>
  828. <div class="outline-text-4" id="text-3-2-2">
  829. <p>
  830. Let's say you have a script you want to run every hour.
  831. In <i>$HOME/</i>.config/systemd/user/ you have to create two files: one is the
  832. the service file, the other is the timer file.
  833. </p>
  834. </div>
  835. <ol class="org-ol"><li><a id="sec-3-2-2-1" name="sec-3-2-2-1"></a>Create a service file<br /><div class="outline-text-5" id="text-3-2-2-1">
  836. <p>
  837. We start creating a test.service file like this:
  838. </p>
  839. <pre class="example">
  840. [Unit]
  841. Description=Simple test script
  842. [Service]
  843. Type=simple
  844. ExecStart=/home/andreatsh/bin/test.sh
  845. </pre>
  846. </div>
  847. </li>
  848. <li><a id="sec-3-2-2-2" name="sec-3-2-2-2"></a>Create a timer file<br /><div class="outline-text-5" id="text-3-2-2-2">
  849. <p>
  850. Now we create test.timer file:
  851. </p>
  852. <pre class="example">
  853. [Unit]
  854. Description=Runs every minute
  855. RefuseManualStart=no
  856. RefuseManualStop=no
  857. [Timer]
  858. OnCalendar=*:0/1
  859. Unit=test.service
  860. [Install]
  861. WantedBy=timers.target
  862. </pre>
  863. </div>
  864. </li>
  865. <li><a id="sec-3-2-2-3" name="sec-3-2-2-3"></a>Enable/Start service and timer<br /><div class="outline-text-5" id="text-3-2-2-3">
  866. <div class="org-src-container">
  867. <pre class="src src-bash">systemctl --user enable testscript.timer
  868. systemctl --user start testscript.timer
  869. </pre>
  870. </div>
  871. </div>
  872. </li></ol>
  873. </div>
  874. </div>
  875. <div id="outline-container-sec-3-3" class="outline-3">
  876. <h3 id="sec-3-3"><span class="section-number-3">3.3</span> Exercises:</h3>
  877. <div class="outline-text-3" id="text-3-3">
  878. <ol class="org-ol">
  879. <li>Find a way to check periodically (for example every hour) your available
  880. disk space and send the result via email. For example, if your simulation
  881. do an heavy i/o on disk and you have a limited disk quota it's a good
  882. habit monitor it.
  883. Difficulty: Basic
  884. Estimated time to complete task: ~ 1 minute
  885. </li>
  886. </ol>
  887. </div>
  888. </div>
  889. </div>
  890. <div id="outline-container-sec-4" class="outline-2">
  891. <h2 id="sec-4"><span class="section-number-2">4</span> How to 'rename' large number of files?</h2>
  892. <div class="outline-text-2" id="text-4">
  893. <p>
  894. I have a large number of file I want to rename them,
  895. I can do that in a easy and fast way?
  896. </p>
  897. <div class="org-src-container">
  898. <pre class="src src-bash">#!/bin/bash
  899. # Debian/Ubuntu version
  900. rename 's/pattern/replacement/' files
  901. # Fedora/CentOS version
  902. rename pattern replacement files
  903. </pre>
  904. </div>
  905. </div>
  906. </div>
  907. <div id="outline-container-sec-5" class="outline-2">
  908. <h2 id="sec-5"><span class="section-number-2">5</span> How to 'find' and do operations over multiple files?</h2>
  909. <div class="outline-text-2" id="text-5">
  910. <div class="org-src-container">
  911. <pre class="src src-bash">for i in $(ls);
  912. do
  913. # Do something ..
  914. done
  915. </pre>
  916. </div>
  917. <p>
  918. You don't have control on the output, ls does not distinguish between regular
  919. file or directory, or files with different permission.
  920. Sure, you can use a more complex command like
  921. </p>
  922. <div class="org-src-container">
  923. <pre class="src src-bash">for i in $(ls -l | awk '/^-rwx-rw--x/{print $9}');
  924. do
  925. # Do something ..
  926. done
  927. </pre>
  928. </div>
  929. <p>
  930. but is it worth? This is not an efficient way to do loop over file.
  931. It's always better use appropriate tools: for example I prefer a syntax like:
  932. </p>
  933. <div class="org-src-container">
  934. <pre class="src src-bash">find /path/to/dir -type f -exec touch {} \;
  935. </pre>
  936. </div>
  937. </div>
  938. </div>
  939. <div id="outline-container-sec-6" class="outline-2">
  940. <h2 id="sec-6"><span class="section-number-2">6</span> Text processing by examples: awk</h2>
  941. <div class="outline-text-2" id="text-6">
  942. <p>
  943. Awk is a really useful tool for text processing. If you have to easily
  944. manipulate data organized in columns, strange format, or rearrange data
  945. based on pattern this is for you.
  946. Here you can find some basic examples that in my opinion show how much
  947. intuitive and powerful Awk is.
  948. These examples are deliberately more idiomatic than those that often you can
  949. learn from standard manuals, they want to be hints on how to write shorter
  950. and sometimes more efficient Awk programs.
  951. </p>
  952. </div>
  953. <div id="outline-container-sec-6-0-1" class="outline-4">
  954. <h4 id="sec-6-0-1"><span class="section-number-4">6.0.1</span> The worst thing you can do with Awk! (imho)</h4>
  955. <div class="outline-text-4" id="text-6-0-1">
  956. <p>
  957. Suppose we want to read a file line-by-line, and we want to print the
  958. second column. You might be tempted to do something like:
  959. </p>
  960. <div class="org-src-container">
  961. <pre class="src src-bash"># Example of bad code!
  962. while IFS= read -r line; do
  963. echo "$line" | awk '{ print $2 }'
  964. done &lt; file
  965. </pre>
  966. </div>
  967. <p>
  968. Why this is not a good idea? Awk is designed and optimized to read a file
  969. line-by-line and do operations on these lines. In this case you are calling
  970. an instance of Awk for each line of your file, it's a waste of resources and
  971. time! If you have to cut ten onions you take the knife, cut all the onions
  972. and then you wash the knife and put it away; in this case you are washing
  973. and putting the knife away after you have cut every single onion.
  974. The right way to do this task is:
  975. </p>
  976. <div class="org-src-container">
  977. <pre class="src src-bash">awk '{ print $2 }' file
  978. </pre>
  979. </div>
  980. <p>
  981. Note: Whenever is possible avoid loops in Bash!
  982. </p>
  983. </div>
  984. </div>
  985. <div id="outline-container-sec-6-0-2" class="outline-4">
  986. <h4 id="sec-6-0-2"><span class="section-number-4">6.0.2</span> Print lines based on pattern</h4>
  987. <div class="outline-text-4" id="text-6-0-2">
  988. <p>
  989. Let's say we simply want to print all lines in a file that match a pattern.
  990. Our first attempt might be something like that:
  991. </p>
  992. <div class="org-src-container">
  993. <pre class="src src-bash">awk '{ if( $0 ~ /pattern/ ) print $0 }' file
  994. </pre>
  995. </div>
  996. <p>
  997. This code works, but we can go further. If this is not the first time you
  998. use Awk, you know it works by definition in this way:
  999. </p>
  1000. <div class="org-src-container">
  1001. <pre class="src src-bash">awk ' CONDITION_1 { ACTION_1 } ... CONDITION_N { ACTION_N} ' file
  1002. </pre>
  1003. </div>
  1004. <p>
  1005. So the previous one-liner become:
  1006. </p>
  1007. <div class="org-src-container">
  1008. <pre class="src src-bash">awk ' $0 ~ /pattern/ { print $0 }' file
  1009. </pre>
  1010. </div>
  1011. <p>
  1012. But it's not enough! We also know from theory that every regular expression
  1013. is implicitly applied by Awk to $0; moreover, we know the default action
  1014. applied to a true condition without specified actions is 'print' ($0 is a
  1015. redundant option since 'print' alone by default print '$0').
  1016. I omit all these working intermidiate steps and come to the point:
  1017. </p>
  1018. <div class="org-src-container">
  1019. <pre class="src src-bash"># This command emulates 'grep'
  1020. awk '/pattern/' file
  1021. </pre>
  1022. </div>
  1023. <p>
  1024. Really more readable and typing saving than our first try.
  1025. </p>
  1026. <p>
  1027. And if you want to print all lines that does not match a pattern:
  1028. </p>
  1029. <div class="org-src-container">
  1030. <pre class="src src-bash"># This command emulates 'grep -v'
  1031. awk '!/pattern/' file
  1032. </pre>
  1033. </div>
  1034. </div>
  1035. <ol class="org-ol"><li><a id="sec-6-0-2-1" name="sec-6-0-2-1"></a>Exercises:<br /><div class="outline-text-5" id="text-6-0-2-1">
  1036. <ol class="org-ol">
  1037. <li>Print lines that match both pattern1 AND pattern2
  1038. </li>
  1039. <li>Print lines that match pattern1 BUT NOT pattern2
  1040. </li>
  1041. </ol>
  1042. </div>
  1043. </li></ol>
  1044. </div>
  1045. <div id="outline-container-sec-6-0-3" class="outline-4">
  1046. <h4 id="sec-6-0-3"><span class="section-number-4">6.0.3</span> Print lines in range</h4>
  1047. <div class="outline-text-4" id="text-6-0-3">
  1048. <div class="org-src-container">
  1049. <pre class="src src-bash"># NR: The total number of input records seen so far.
  1050. # Print from line 3 to line 7 (inclusive)
  1051. awk 'NR==3,NR==7'
  1052. # Print lines between to regular expression (inclusive)
  1053. awk '/beginregex/,/endregex/'
  1054. # Print lines between to regular expression (not inclusive)
  1055. awk '/endregex/{p=0}; p; /beginregex/{p=1}/
  1056. </pre>
  1057. </div>
  1058. </div>
  1059. <ol class="org-ol"><li><a id="sec-6-0-3-1" name="sec-6-0-3-1"></a>Exercises:<br /><div class="outline-text-5" id="text-6-0-3-1">
  1060. <ol class="org-ol">
  1061. <li>Print lines from beginregex to endregex, excluding beginregex.
  1062. </li>
  1063. <li>Print lines from beginregex to endregex, excluding endregex.
  1064. </li>
  1065. </ol>
  1066. </div>
  1067. </li></ol>
  1068. </div>
  1069. <div id="outline-container-sec-6-0-4" class="outline-4">
  1070. <h4 id="sec-6-0-4"><span class="section-number-4">6.0.4</span> Remove (non consecutive) duplicated lines</h4>
  1071. <div class="outline-text-4" id="text-6-0-4">
  1072. <div class="org-src-container">
  1073. <pre class="src src-bash">awk '!z[$0]++' file
  1074. </pre>
  1075. </div>
  1076. </div>
  1077. <ol class="org-ol"><li><a id="sec-6-0-4-1" name="sec-6-0-4-1"></a>Exercises:<br /><div class="outline-text-5" id="text-6-0-4-1">
  1078. <ol class="org-ol">
  1079. <li>Remove duplicated lines only if these lines are consecutive.
  1080. (Emulates uniq command)
  1081. </li>
  1082. </ol>
  1083. </div>
  1084. </li></ol>
  1085. </div>
  1086. <div id="outline-container-sec-6-0-5" class="outline-4">
  1087. <h4 id="sec-6-0-5"><span class="section-number-4">6.0.5</span> Find and replace</h4>
  1088. <div class="outline-text-4" id="text-6-0-5">
  1089. <div class="org-src-container">
  1090. <pre class="src src-bash"># Find and replace only the first occurrence
  1091. awk '{sub("pattern","replacement")} 1'
  1092. # Find and replace all occurrences
  1093. awk '{gsub("pattern","replacement")} 1'
  1094. </pre>
  1095. </div>
  1096. </div>
  1097. <ol class="org-ol"><li><a id="sec-6-0-5-1" name="sec-6-0-5-1"></a>Exercises:<br /><div class="outline-text-5" id="text-6-0-5-1">
  1098. <ol class="org-ol">
  1099. <li>Print only the lines where a replacement occurs.
  1100. </li>
  1101. </ol>
  1102. </div>
  1103. </li></ol>
  1104. </div>
  1105. <div id="outline-container-sec-6-0-6" class="outline-4">
  1106. <h4 id="sec-6-0-6"><span class="section-number-4">6.0.6</span> Print intersection of two files</h4>
  1107. <div class="outline-text-4" id="text-6-0-6">
  1108. <p>
  1109. This is a really common and powerful construct in Awk.
  1110. </p>
  1111. <div class="org-src-container">
  1112. <pre class="src src-bash"># Print only the lines that are both in file1 and file2
  1113. awk 'FNR==NR{ z[$0]++; next } $0 in z' file1 file2
  1114. </pre>
  1115. </div>
  1116. </div>
  1117. </div>
  1118. <div id="outline-container-sec-6-0-7" class="outline-4">
  1119. <h4 id="sec-6-0-7"><span class="section-number-4">6.0.7</span> Parse a CSV file</h4>
  1120. <div class="outline-text-4" id="text-6-0-7">
  1121. <div class="org-src-container">
  1122. <pre class="src src-bash"># With the flag -F you can set the Input Field Separator (IFS)
  1123. # If the separator is comma "," you can write something like:
  1124. awk -F, '{ print $1,$3 }' file.csv
  1125. NOTE: In this case we have set IFS=, but when we print, as you already know,
  1126. there are spaces between fields and not commas! This is not desirable if we,
  1127. after some manipulation, want to have again a csv file as output.
  1128. Sure, we can print explicitly all commas:
  1129. awk -F, '{ print $1","$3 }' file.csv
  1130. # However, this is not a good solution.
  1131. # In many situation is preferable set the Output Field Separator (OFS):
  1132. awk -F, '{ print $1,$3 }' OFS=, file.csv
  1133. </pre>
  1134. </div>
  1135. </div>
  1136. </div>
  1137. </div>
  1138. <div id="outline-container-sec-7" class="outline-2">
  1139. <h2 id="sec-7"><span class="section-number-2">7</span> Project example: dice rolling simulations</h2>
  1140. <div class="outline-text-2" id="text-7">
  1141. <p>
  1142. I choose this example because it's simple and everyone has experience of
  1143. dice rolling; moreover, I write the program in C++ because there are several
  1144. courses that cover this language and I hope you are familiar with it.
  1145. The C++ code I wrote is straightforward, so you can focus on:
  1146. </p>
  1147. <ul class="org-ul">
  1148. <li>how to pass different arguments to a program
  1149. </li>
  1150. <li>how to redirect program's output to specific files
  1151. </li>
  1152. <li>how to manipulate output files
  1153. </li>
  1154. </ul>
  1155. <p>
  1156. You can find the program's code in the <i>project:dice</i> subfolder of this seminar
  1157. repository. Compile. Run. An usage message should be returned:
  1158. Usage: ./dice &lt;dice number&gt; &lt;faces number&gt; &lt;rolls number&gt;
  1159. </p>
  1160. </div>
  1161. </div>
  1162. <div id="outline-container-sec-8" class="outline-2">
  1163. <h2 id="sec-8"><span class="section-number-2">8</span> Not covered by this talk but truly useful and important:</h2>
  1164. <div class="outline-text-2" id="text-8">
  1165. </div><div id="outline-container-sec-8-1" class="outline-3">
  1166. <h3 id="sec-8-1"><span class="section-number-3">8.1</span> ssh</h3>
  1167. <div class="outline-text-3" id="text-8-1">
  1168. <p>
  1169. Some tips to improve your experience.
  1170. </p>
  1171. </div>
  1172. <div id="outline-container-sec-8-1-1" class="outline-4">
  1173. <h4 id="sec-8-1-1"><span class="section-number-4">8.1.1</span> Generate public/private key pair for authentication</h4>
  1174. <div class="outline-text-4" id="text-8-1-1">
  1175. <p>
  1176. You can generate SSH keys with the command:
  1177. </p>
  1178. <div class="org-src-container">
  1179. <pre class="src src-bash">ssh-keygen
  1180. </pre>
  1181. </div>
  1182. <p>
  1183. Once you have a key pair you have to append your public key in the file
  1184. '~/.ssh/authorized<sub>keys'</sub> on your remote host, you can do it manually or with
  1185. the command:
  1186. </p>
  1187. <div class="org-src-container">
  1188. <pre class="src src-bash">ssh-copy-id -i ~/.ssh/id_rsa.pub user@domain
  1189. </pre>
  1190. </div>
  1191. </div>
  1192. </div>
  1193. <div id="outline-container-sec-8-1-2" class="outline-4">
  1194. <h4 id="sec-8-1-2"><span class="section-number-4">8.1.2</span> Basic configuration example</h4>
  1195. <div class="outline-text-4" id="text-8-1-2">
  1196. <p>
  1197. Create file ~/.ssh/config (or edit it if you already have one).
  1198. A basic configuration file looks like:
  1199. </p>
  1200. <pre class="example">
  1201. Host *
  1202. ControlMaster auto
  1203. ControlPath ~/.ssh/master-%r@%h:%p
  1204. Host mylab
  1205. Hostname mylab.example
  1206. User mylogin
  1207. CheckHostIP no
  1208. Compression yes
  1209. Protocol 2
  1210. </pre>
  1211. <p>
  1212. From now on, when you want to open a connection via ssh just type:
  1213. </p>
  1214. <div class="org-src-container">
  1215. <pre class="src src-bash">ssh mylab
  1216. </pre>
  1217. </div>
  1218. <p>
  1219. When you connect the first time ControlMaster create a socket and from the
  1220. second time you don't need to reinsert your password or key's passphrase (if
  1221. you have set one). Note that you can configure multiple hosts and add or
  1222. remove options as you like to fit your needs.
  1223. </p>
  1224. </div>
  1225. </div>
  1226. </div>
  1227. <div id="outline-container-sec-8-2" class="outline-3">
  1228. <h3 id="sec-8-2"><span class="section-number-3">8.2</span> git</h3>
  1229. <div class="outline-text-3" id="text-8-2">
  1230. <p>
  1231. Love it! Never again without!
  1232. I truly recommend you to inform on this irreplaceable tool!
  1233. </p>
  1234. </div>
  1235. </div>
  1236. <div id="outline-container-sec-8-3" class="outline-3">
  1237. <h3 id="sec-8-3"><span class="section-number-3">8.3</span> tmux</h3>
  1238. <div class="outline-text-3" id="text-8-3">
  1239. <p>
  1240. Tmux is a terminal multiplexer.
  1241. </p>
  1242. </div>
  1243. </div>
  1244. </div>
  1245. </div>
  1246. <div id="postamble" class="status">
  1247. <p class="date">Date: 2017.05.10</p>
  1248. <p class="author">Author: Andrea Miglietta</p>
  1249. <p class="date">Created: 2017-05-10 Wed 14:42</p>
  1250. <p class="creator"><a href="http://www.gnu.org/software/emacs/">Emacs</a> 24.5.1 (<a href="http://orgmode.org">Org</a> mode 8.2.10)</p>
  1251. <p class="validation"><a href="http://validator.w3.org/check?uri=referer">Validate</a></p>
  1252. </div>
  1253. </body>
  1254. </html>