diff options
Diffstat (limited to 'ult/ult_5/script.rst')
| -rw-r--r-- | ult/ult_5/script.rst | 333 |
1 files changed, 161 insertions, 172 deletions
diff --git a/ult/ult_5/script.rst b/ult/ult_5/script.rst index e112ff7..2937397 100644 --- a/ult/ult_5/script.rst +++ b/ult/ult_5/script.rst @@ -3,17 +3,15 @@ .. At the end of this tutorial, you will be able to: - .. 1. - .. 2. + .. 1. Understand what is Redirection and Piping. + .. 2. Learn various features of shell. .. Prerequisites .. ------------- -.. 1. Using Linux tools - Part 1 -.. 2. Using Linux tools - Part 2 -.. 3. Using Linux tools - Part 3 -.. 4. Using Linux tools - Part 4 - +.. 1. Getting started with Linux +.. 2. Basic File Handling +.. 4. Advanced file handling Script ------ @@ -26,7 +24,7 @@ team along with the logo of MHRD }}} .. R1 Hello friends and Welcome to the tutorial on -'Using linux tools - Part 5'. +'Redirection and Piping'. .. L2 @@ -36,11 +34,8 @@ Hello friends and Welcome to the tutorial on At the end of this tutorial, you will be able to, - 1. Sort lines of text files - #. Print lines matching a pattern - #. Translate or delete characters - #. Omit repeated lines. - + 1. Understand what is Redirection. + #. Learn the concept of Piping. .. L3 @@ -49,14 +44,15 @@ At the end of this tutorial, you will be able to, .. R3 Before beginning this tutorial,we would suggest you to complete the -tutorial on "Using Linux tools from Part 1 to Part 4". +former tutorials as being displayed currently. .. R4 -In this tutorial, we shall learn about text processing. -TO begin with, consider data kept in two files, namely marks1.txt and -students.txt -Let us see what data they contain. Open a terminal and type, +Let us begin with the concept of 'Redirection and Piping' which +performs the same operations as the ``cut`` and ``paste`` commands. + +Consider the files ``marks.txt`` and ``students.txt``.The contents of +the files are as following: .. L4 @@ -68,274 +64,267 @@ Let us see what data they contain. Open a terminal and type, .. R5 -Let's say we wish to sort the output in the alphabetical order -of the names of the files. We can use the ``sort`` command for this -purpose. - -We just pipe the previous output to the ``sort`` command as, +Now, let us view the contents of both these files side-by-side. .. L5 :: - cut -d " " -f 2- marks1.txt | paste -d " " students.txt -| sort + cut -d " " -f 2- marks1.txt | paste -d " " students.txt - .. R6 -Let's say we wish to sort the names, based on the marks in the first -subject i.e. the first column after the name. ``sort`` command also allows us to -specify the delimiter between the fields and sort the data on a particular -field. ``-t`` option is used to specify the delimiter and ``-k`` option -is used to specify the field. +Now, in order to view the same output in a new file at an other +location, we say, .. L6 :: - cut -d " " -f 2- marks1.txt | paste -d " " students.txt -| sort -t " " -k 2 + cut -d " " -f 2- marks1.txt > /tmp/m_tmp.txt + paste -d " " students.txt m_tmp.txt -.. L7 +.. R7 -{{{ Show slide with, Sort... }}} +First, let us try to understand the second solution,which is a two +step approach. +Later, we shall look at the first solution. -.. R7 +.. L7 + +.. L8 -This command give us a sorted output as required. But, what if we would -like the output to appear in the reverse order. ``-r`` option allows the output -to be sorted in the reverse order and the ``-n`` option is used to choose -a numerical sorting. +{{{ Show slide, with Redirection }}} .. R8 -Let us do it on the terminal and see for ourselves, +The standard output, in general, goes to the display. +But, this may not be what we always require. -.. L8 +For instance, in the solution above, we use the cut command and get only +the required columns of the file and write the output to a new temporary +file. The ``>`` character is used to state that we wish to redirect the +output, and it is followed by the location to which we wish to redirect. +For example, -{{{ Switch to the terminal }}} -:: + command > file1 - cut -d " " -f 2- marks1.txt | paste -d " " students.txt -| - sort -t " " -k 2 -rn +.. L9 + +{{{ Show slide, with Redirection.. }}} .. R9 -Suppose, While you are compiling the student marklist, Anne walks up to you and -wants to know her marks. You, being a kind person that you are, oblige. -But you do not wish to her to see the marks that others have scored. What -do you do? Here, the ``grep`` command comes to your rescue. +Similarly, the standard input (stdin) can be redirected as, + + command < file1 -``grep`` is a command line text search utility. You can use it to search -for Anne and show her, what she scored. ``grep`` allows us to search for a -search string in files. But we could, like any other command, pipe the -output of other commands to it. So, we shall use the previous combination -of cut and paste that we had, to get the marks of students along with their -names and search for Anne in that. +The input and the output redirection could be combined in a single command, +as, -.. L9 -:: + command < infile > outfile - cut -d " " -f 2- marks1.txt | paste -d " " students.txt - | grep Anne +.. L10 -.. R10 +{{{ Show slide, with stderr }}} -This will give us only the line containing the word Anne as the output. -The grep command is by default case-sensitive. So, we wouldn't have got -the result if we had searched for anne, with a small a, instead of -Anne, with a capital a. But, what if we didn't know, whether the name was -capitalized or not? ``grep`` allows you to do case-insensitive searches -by using the ``-i`` option. -.. L10 -:: +.. R10 - cut -d " " -f 2- marks1.txt | paste -d " " students.txt - | grep -i Anne +There is actually a third kind of standard stream, called the Standard +error (stderr). Any error messages that you get, are coming through this +stream. Like ``stdout``, ``stderr`` also streams to the display by default, +but it could be redirected to a file, as well. .. R11 -Now, in another scenario, if we wished to print all the lines, which do -not contain the word Anne, we could use the ``-v`` option. +For instance, let's reproduce an error using the ``cut`` command used +before. We shall change the ``-f`` option to ``-c`` .. L11 + +{{{ Switch to terminal }}} :: - cut -d " " -f 2- marks1.txt | paste -d " " students.txt - | grep -iv Anne + cut -d " " -c 2- marks1.txt > /tmp/m_tmp.txt .. R12 -grep allows us to do more complex searches, for instance, searching for -sentences starting or ending with a particular pattern and regular -expression based searches. - -{{{ Show slide with, tr }}} - -``tr`` is a command that takes two sets of characters as parameters, and -replaces occurrences of the characters in the first set with the -corresponding elements from the other set. It reads from the standard -output and writes to the standard output. - -For instance, if we wish to replace all the lower case letters in the -students file with upper case, we can do it as, +This displays an error saying that the delimiter option should be used +with the fields option only. You may verify this by looking at the +``m_tmp.txt`` file, which is now empty.We can now, redirect the +``stderr`` also to a file, instead of showing it on the display. .. L12 - -{{{ Switch to the terminal }}} :: - cat students.txt | tr a-z A-Z + cut -d " " -f 2- marks1.txt 1> /tmp/m_tmp.txt 2> /tmp/m_err.txt .. R13 -A common task is to remove empty newlines from a file. The ``-s`` flag -causes ``tr`` to compress sequences of identical adjacent characters in its -output to a single token. For example, +The above command redirects all the errors to the ``m_err.txt`` file +and the output to the ``m_tmp.txt`` file. When redirecting, 1 stands +for ``stdout`` and 2 stands for ``stderr``. + +Let us complete the solution by using the ``paste`` command. .. L13 :: - tr -s '\n' '\n' + paste -d " " students.txt m_tmp.txt .. R14 -Hit enter 2-3 times and see that every time we hit enter we get a newline. +So, in two steps we solved the problem of getting rid of the roll numbers +from the marks file and displaying the marks along with the names of the +students. Now, that we know how to redirect output, we could choose to +write the output to a file, instead of showing on the display. + +Let us now look at the first solution. .. L14 :: - <Enter> - <Enter> + cut -d " " -f 2- marks1.txt | paste -d " " students.txt - + +.. L15 + +{{{ Show slide, with Piping }}} .. R15 -It replaces sequences of one or more newline characters with a single newline. +First of all, the hyphen at the end is to ask the paste command to +read the standard input, instead of looking for a FILE. The ``man`` +page of ``paste`` command gives us this information. -The ``-d`` flag causes ``tr`` to delete all tokens of the specified set of -characters from its input. In this case, only a single character set -argument is used. The following command removes carriage return characters, -thereby converting a file in DOS/Windows format to the Unix format. +Now, let us observe the ``cut`` command. If we look at the command only +upto the ``|`` character, it appears as a normal ``cut`` command . -.. L15 -:: +.. L16 - cat foo.txt | tr -d '\r' > bar.txt +{{{ Show slide, with Piping.. }}} .. R16 -The ``-c`` flag complements the first set of characters. +So, the ``|`` character here, seems +to be joining the two commands in some way. +Essentially, what we are doing is, to redirect the output of the first +command to ``stdin`` and the second command takes the input from the ``stdin``. +This activity is commonly called piping and the character ``|`` is called +a pipe. -.. L16 -:: +.. L17 - tr -cd '[:alnum:]' +{{{ Show slide, with Piping... }}} .. R17 -It therefore removes all non-alphanumeric characters. +This is roughly equivalent to using two redirects and a temporary file. -Let us consider one more scenario.Suppose we have a list of items, say books, -and we wish to obtain a list which names of all the books only once, without -any duplicates. To achieve this, we use the ``uniq`` command. Let us first -have a look at our file + command1 > tempfile + command2 < tempfile + rm tempfile -.. L17 -:: +Also, given that a pipe is just a way to send the output of a command to +the ``stdin``, it should be obvious to you that we can use a chain of +pipes. Any number of commands can be piped together and therefore it should + be noted that it is not restricted to only two commands. - cat items.txt +.. L18 -.. R18 +{{{ Switch to Summary slide }}} -Now, let us try and get rid of the duplicate lines from this file using -the ``uniq`` command. +.. R18 -.. L18 -:: +This brings us to the end of the end of this tutorial. +In this tutorial, we have learnt to, - uniq items.txt + 1. Use the ``cut`` and ``paste`` commands in redirection. + 2. Use the pipe ( | ) character. + +.. L19 + +{{{ Show self assessment questions slide }}} .. R19 -Nothing happens! Why? The ``uniq`` command removes duplicate lines only when -they are next to each other. So, henceforth, we get a sorted file from the -original file and work with that file. +Here are some self assessment questions for you to solve: -.. L19 -:: +1. How will you redirect the content of a file to a device ? + +2. How to view last field(30), in a file located at /home/test.txt whose + first line is "data:myscripts:20:30" + + - cut -d : -f 4 /home/test.txt + - cut -f 3 /home/test.txt + - cut -d : -f 3 /home/test.txt + + +.. L20 - sort items.txt | uniq +{{{ Solutions for the self assessment questions on slide }}} .. R20 -``uniq -u`` command gives the lines which are unique and do not have any -duplicates in the file. ``uniq -d`` outputs only those lines which -have duplicates. +And the answers: -.. L20 +1. A file can be redirected to a device as, +:: + + cat filename > device +For eg: :: - uniq -u items-sorted.txt - -.. R21 + cat sound.wav > /dev/audio + -The ``-c`` option displays the number of times each line occurs in the file. +2. The correct option would be +:: + + cut -d : -f 4 /home/test.txt .. L21 -:: - uniq -dc items-sorted.txt +{{{ Show the SDES & FOSSEE slide }}} + +.. R21 + +Software Development techniques for Engineers and Scientists - SDES, is an +initiative by FOSSEE. For more information, please visit the given link. + +Free and Open-source Software for Science and Engineering Education - FOSSEE, is +based at IIT Bombay which is funded by MHRD as part of National Mission on +Education through ICT. .. L22 -{{{ Show summary slide }}} +{{{ Show the ``About the Spoken Tutorial Project'' slide }}} .. R22 -This brings us to the end of the end of this tutorial. -In this tutorial, we have learnt to, - - 1. Use the ``sort`` command to sort lines of text files. - #. Use the ``grep`` command to search text pattern. - #. Use the ``tr`` command to translate and/or delete characters. - #. Use the ``uniq`` command to omit repeated lines in a text. +Watch the video available at the following link. It summarises the Spoken +Tutorial project.If you do not have good bandwidth, you can download and +watch it. .. L23 -{{{ Show self assessment questions slide }}} +{{{ Show the `` Spoken Tutorial Workshops'' slide }}} .. R23 -Here are some self assessment questions for you to solve +The Spoken Tutorial Project Team conducts workshops using spoken tutorials, +gives certificates to those who pass an online test. - 1. To obtain patterns; one per line, which of the following command is used ? - - - grep -f - - grep -i - - grep -v - - grep -e - - 2. Translate the word 'linux' to upper-case. - - 3. Sort the output of the ``ls -al`` command. +For more details, contact contact@spoken-tutorial.org .. L24 -{{{ Solution of self assessment questions on slide }}} +{{{ Show the ``Acknowledgements'' slide }}} .. R24 -And the answers, - - 1. In order to obtain patterns one per line, we use the ``grep`` command - alongwith the -f option. - - 2. We use the tr command to change the word into uppercase -:: - - echo 'linux' | tr a-z A-Z - - - 3. We use the sort command as, -:: - - ls -al | sort -n -k5 -The -n means "sort numerically", and the -k5 option means to key off of -column five. +Spoken Tutorial Project is a part of the "Talk to a Teacher" project. +It is supported by the National Mission on Education through ICT, MHRD, +Government of India. More information on this mission is available at the +given link. .. L25 |