diff options
Diffstat (limited to 'ult/ult_7/script.rst')
| -rw-r--r-- | ult/ult_7/script.rst | 410 |
1 files changed, 201 insertions, 209 deletions
diff --git a/ult/ult_7/script.rst b/ult/ult_7/script.rst index 4efdfd9..b4cb55c 100644 --- a/ult/ult_7/script.rst +++ b/ult/ult_7/script.rst @@ -3,18 +3,17 @@ .. At the end of this tutorial, you will be able to: - .. 1. Prepare scripts using 'Control Operators'. - .. 2. Understand what 'Environment Variables' are. + .. 1. Sort lines of text files + .. 2. Print lines matching a pattern + .. 3. Translate or delete characters + .. 4. Omit repeated lines + .. Prerequisites .. ------------- -.. 1. Using Linux tools - Part 1 -.. 2. Using Linux tools - Part 2 -.. 3. Using Linux tools - Part 3 -.. 4. Using Linux tools - Part 4 -.. 5. Using Linux tools - Part 5 -.. 6. Using Linux tools - Part 6 +.. 1. Getting started with Linux +.. 2. Redirection and Piping @@ -28,374 +27,367 @@ team along with the logo of MHRD }}} .. R1 -Hello friends and Welcome to the tutorial on -'Using linux tools - Part 7'. +Hello friends and Welcome to the tutorial on 'Text Processing'. .. L2 -{{{ Show the 'Objectives' slide }}} +{{{ Show slide with objectives }}} .. R2 At the end of this tutorial, you will be able to, - 1. Prepare scripts using 'Control Operators'. - 2. Understand what 'Environment Variables' are. - + 1. Sort lines of text files + #. Print lines matching a pattern + #. Translate or delete characters + #. Omit repeated lines. + + .. L3 {{{ Switch to the pre-requisite slide }}} .. R3 -Before beginning this tutorial, we suggest you to complete the -tutorials, "Using Linux tools, Part 1 to Part 6". +Before beginning this tutorial,we would suggest you to complete the +former tutorials as being displayed currently. .. R4 -We have many 'Control Structures and Operators' available in the linux bash. -Let us look at how to use them. -To write an 'if', or an 'if-else' construct, we need to check or test for a -condition(s). The ``test`` command allows us to test for condition(s). It has -a whole range of tests that can be performed. The man page of ``test`` -gives you the complete listing of various types of tests that can be performed -with it. - -Let's write a simple script with an ``if`` condition that tests whether a -directory with a particular name, exists or not. +In this tutorial, we shall learn about text processing. +TO begin with, consider data kept in two files, namely marks1.txt and +students.txt +Let us see what data they contain. Open a terminal and type, .. L4 -.. L5 +{{{ Open the terminal }}} +:: -{{{ Show slide, 'if' }}} + cat marks1.txt + cat students.txt .. R5 -Let us create a script named ``dir-test.sh`` with this code. +Let's say we wish to sort the output in the alphabetical order +of the names of the files. We can use the ``sort`` command for this +purpose. - #!/bin/bash - if test -d $1 - then - echo "Yes, the directory" $1 "is present" - fi +We just pipe the previous output to the ``sort`` command as, -When the script is run with an argument, it will print a message, if a -directory with the said name exists in the current working directory. +.. L5 +:: + + cut -d " " -f 2- marks1.txt | paste -d " " students.txt -| sort .. R6 -Let's write a simple script which returns back whether the argument passed -is negative or not. +Let's say we wish to sort the names, based on the marks in the first +subject i.e. the first column after the name. ``sort`` command also allows us to +specify the delimiter between the fields and sort the data on a particular +field. ``-t`` option is used to specify the delimiter and ``-k`` option +is used to specify the field. .. L6 - -{{{ Open the file sign.sh and show }}} :: - #!/bin/bash - if test $1 -lt 0 - then - echo "number is negative" - else - echo "number is non-negative" - fi - -.. R7 - -We can run the file with a set of different inputs and see if it works. + cut -d " " -f 2- marks1.txt | paste -d " " students.txt -| sort -t " " -k 2 .. L7 -{{{ Switch to terminal }}} -:: +{{{ Show slide with, Sort... }}} + +.. R7 - ./sign.sh -11 +This command give us a sorted output as required. But, what if we would +like the output to appear in the reverse order. ``-r`` option allows the output +to be sorted in the reverse order and the ``-n`` option is used to choose +a numerical sorting. .. R8 -Instead of using the ``test`` command, square brackets may also be used. +Let us do it on the terminal and see for ourselves, .. L8 -.. L9 +{{{ Switch to the terminal }}} +:: -{{{ Show slide, [ ] - alias for test }}} + cut -d " " -f 2- marks1.txt | paste -d " " students.txt -| + sort -t " " -k 2 -rn .. R9 -Note that the spacing is important, when using the square brackets. -The left square bracket ( ``[`` ) should be followed by a space and the right -square bracket ( ``]`` ) should be preceded by a space. +Suppose, While you are compiling the student marklist, Anne walks up to you and +wants to know her marks. You, being a kind person that you are, oblige. +But you do not wish to her to see the marks that others have scored. What +do you do? Here, the ``grep`` command comes to your rescue. -Let's create something interesting using the 'if-else' clause. Let's write a -script, that greets the user, based on the time. +``grep`` is a command line text search utility. You can use it to search +for Anne and show her, what she scored. ``grep`` allows us to search for a +search string in files. But we could, like any other command, pipe the +output of other commands to it. So, we shall use the previous combination +of cut and paste that we had, to get the marks of students along with their +names and search for Anne in that. -.. L10 +.. L9 +:: -{{{ Open the file clause.sh and show }}} -{{{ Highlight the required content wherever necessary, while narrating }}} + cut -d " " -f 2- marks1.txt | paste -d " " students.txt - | grep Anne .. R10 -There are a couple of new things in this script. ``$LOGNAME`` is another -'environment variable', which has the login name of the user. The variables, -``hour`` and ``now`` are actually taking the output of the commands that -are placed in the back quotes. +This will give us only the line containing the word Anne as the output. +The grep command is by default case-sensitive. So, we wouldn't have got +the result if we had searched for anne, with a small a, instead of +Anne, with a capital a. But, what if we didn't know, whether the name was +capitalized or not? ``grep`` allows you to do case-insensitive searches +by using the ``-i`` option. -Now, let us see how to run loops in bash. We shall look at the ``for`` and -the ``while`` loops. - -.. L11 +.. L10 +:: -{{{ Show slide, 'for' }}} + cut -d " " -f 2- marks1.txt | paste -d " " students.txt - | grep -i Anne .. R11 -Suppose we have a set of files, whose file-names contain numbers before the -text, say ``08 - Society.mp3``. We would like to rename these files by -removing the numbers before the text. How would we go about doing that? +Now, in another scenario, if we wished to print all the lines, which do +not contain the word Anne, we could use the ``-v`` option. -It is clear from the problem statement that we could loop over the list of -files and rename each of them. +.. L11 +:: + + cut -d " " -f 2- marks1.txt | paste -d " " students.txt - | grep -iv Anne .. R12 -First, let us look at a simple ``for`` loop, to understand how it works. +grep allows us to do more complex searches, for instance, searching for +sentences starting or ending with a particular pattern and regular +expression based searches. + +{{{ Show slide with, tr }}} + +``tr`` is a command that takes two sets of characters as parameters, and +replaces occurrences of the characters in the first set with the +corresponding elements from the other set. It reads from the standard +output and writes to the standard output. + +For instance, if we wish to replace all the lower case letters in the +students file with upper case, we can do it as, .. L12 -{{{ Switch to terminal }}} +{{{ Switch to the terminal }}} :: - for animal in rat cat dog man - do - echo $animal - done + cat students.txt | tr a-z A-Z .. R13 -We just wrote a list of animals, each name separated by a space -and then printed each name on a separate line. The variable ``animal`` is a -'dummy' or a 'loop variable'. It can then be used to refer to the element of -the list that is currently being dealt with. We could, obviously, use -something as lame as ``i`` in place of ``animal``. +A common task is to remove empty newlines from a file. The ``-s`` flag +causes ``tr`` to compress sequences of identical adjacent characters in its +output to a single token. For example, .. L13 +:: + + tr -s '\n' '\n' .. R14 -To generate a range of numbers and iterate over them, we do the following. +Hit enter 2-3 times and see that every time we hit enter we get a newline. .. L14 +:: -{{{ Open the script ``for-1.sh`` and show }}} + <Enter> + <Enter> .. R15 -Now, let us run the script and see what we get, +It replaces sequences of one or more newline characters with a single newline. -.. L15 +The ``-d`` flag causes ``tr`` to delete all tokens of the specified set of +characters from its input. In this case, only a single character set +argument is used. The following command removes carriage return characters, +thereby converting a file in DOS/Windows format to the Unix format. -{{{ Switch to terminal }}} +.. L15 :: - sh for-1.sh + cat foo.txt | tr -d '\r' > bar.txt .. R16 -Now, we use a ``for`` loop to list the files that we are interested in. +The ``-c`` flag complements the first set of characters. .. L16 - -{{{ Open the script ``for-2.sh`` and show }}} -{{{ Switch to terminal }}} :: - sh for-2.sh + tr -cd '[:alnum:]' .. R17 - -If the file-names contain spaces, ``for`` assumes, each word separated by a -space,to be a single item in the list and prints it in a separate line. We -could modify the script slightly to overcome this problem. -.. L17 +It therefore removes all non-alphanumeric characters. -{{{ Open the script ``for-3.sh`` and show }}} -{{{ Switch to terminal }}} +Let us consider one more scenario.Suppose we have a list of items, say books, +and we wish to obtain a list which names of all the books only once, without +any duplicates. To achieve this, we use the ``uniq`` command. Let us first +have a look at our file + +.. L17 :: - sh for-3.sh + cat items.txt .. R18 -Now, we have each file name printed on a separate line. The file names are -in the form ``dd - Name.mp3`` and it has to be changed to the format -``Name.mp3``. Also, if the name has spaces, we wish to replace it with -hyphens. +Now, let us try and get rid of the duplicate lines from this file using +the ``uniq`` command. .. L18 - -{{{ Open the script ``for-4.sh`` and show }}} -{{{ Switch to terminal }}} :: - sh for-4.sh + uniq items.txt .. R19 -Now, we simply replace the echo command with a ``mv`` command. +Nothing happens! Why? The ``uniq`` command removes duplicate lines only when +they are next to each other. So, henceforth, we get a sorted file from the +original file and work with that file. .. L19 - -{{{ Open the script ``for-5.sh`` and show }}} -{{{ Switch to terminal }}} :: - sh for-5.sh + sort items.txt | uniq .. R20 -We see that we get our required output. All the files have been renamed and -the spaces are removed. -Now let us move ahead with ``while`` loop. -The ``while`` command allows us to continuously execute a block of commands -until the command that is controlling the loop is executing successfully. +``uniq -u`` command gives the lines which are unique and do not have any +duplicates in the file. ``uniq -d`` outputs only those lines which +have duplicates. + +.. L20 +:: -.. L20 + uniq -u items-sorted.txt .. R21 -Let's start with the lamest example of a ''while'' loop. +The ``-c`` option displays the number of times each line occurs in the file. .. L21 - -{{{ Open the script ``while-1.sh`` and show }}} -{{{ Switch to terminal }}} :: - sh while-1.sh - -.. R22 - -This, as you can see, is an infinite loop that prints ``True``. - -Say, we wish to write a simple program that takes input from the user -and prints it back, until the input is ``quit``, which then quits the program. + uniq -dc items-sorted.txt .. L22 -{{{ Open the script ``while-2.sh`` and show }}} -{{{ Switch to terminal }}} -:: +{{{ Show summary slide }}} - sh while-2.sh +.. R22 + +This brings us to the end of the end of this tutorial. +In this tutorial, we have learnt to, + + 1. Use the ``sort`` command to sort lines of text files. + #. Use the ``grep`` command to search text pattern. + #. Use the ``tr`` command to translate and/or delete characters. + #. Use the ``uniq`` command to omit repeated lines in a text. .. L23 -{{{ Show slide, Environment Variables }}} +{{{ Show self assessment questions slide }}} .. R23 -'Environment variables' are a way of passing information from the shell to the -programs that are run in it. Standard UNIX variables are split into two -categories,'Environment variables' and 'Shell variables'. In broad terms, -'Shell variables' apply only to the current instance of the shell and are -used to set short-term working conditions; 'Environment variables' have a -farther reaching significance, and are set at login, valid for the duration of -the session. By convention, 'Environment variables' have UPPER CASE and 'Shell -variables' have lower case names. +Here are some self assessment questions for you to solve -You can see an example of environment variables in the slide. +1. To obtain patterns; one per line, which of the following command is used ? + + - grep -f + - grep -i + - grep -v + - grep -e -.. R24 +2. Translate the word 'linux' to upper-case. -To see all the variables and their values, we could use any of the -following, +3. Sort the output of the ``ls -al`` command. .. L24 -{{{ Switch to terminal }}} -:: - - printenv | less - env - -.. R25 - -We have looked at the 'PATH' variable, in the previous tutorial. We shall now -use the ``export`` command to change it's value. +{{{ Solution of self assessment questions on slide }}} -.. L25 -:: +.. R24 - export PATH=$PATH:$HOME/bin +And the answers, -.. R26 +1. In order to obtain patterns one per line, we use the ``grep`` command + alongwith the -f option. -Observe the difference in the value of 'PATH' variable before and after -modifying it. +2. We use the tr command to change the word into uppercase +:: -``export`` command is used to export a variable to the environment of all -the processes that are started from that shell. + echo 'linux' | tr a-z A-Z + -.. L26 +3. We use the sort command as, +:: + + ls -al | sort -n -k5 +The -n means "sort numerically", and the -k5 option means to key off of +column five. -.. L27 +.. L25 -{{{ Switch to 'Summary' slide }}} +{{{ Show the SDES & FOSSEE slide }}} -.. R27 +.. R25 -This brings us to the end of this tutorial. -In this tutorial, we have learnt to, - - 1. Prepare scripts using control structures like ``if``, ``if-else``, - ``for`` and ``while``. - 2. Use 'environment variables'. - 3. Export a variable to the environment of all the processes, using - the ``export`` command. +Software Development techniques for Engineers and Scientists - SDES, is an +initiative by FOSSEE. For more information, please visit the given link. -.. L28 +Free and Open-source Software for Science and Engineering Education - FOSSEE, is +based at IIT Bombay which is funded by MHRD as part of National Mission on +Education through ICT. -{{{ Show self assessment questions slide }}} +.. L26 -.. R28 +{{{ Show the ``About the Spoken Tutorial Project'' slide }}} -Here are some self assessment questions for you to solve: +.. R26 - 1. Print the text ``dog man`` in such a way that the prompt - continues after the text. +Watch the video available at the following link. It summarises the Spoken +Tutorial project.If you do not have good bandwidth, you can download and +watch it. - 2. How can you add a new path variable ``/data/myscripts`` to $PATH variable ? +.. L27 -.. L30 +{{{ Show the `` Spoken Tutorial Workshops'' slide }}} -{{{ Solutions of self assessment questions on slide }}} +.. R27 -.. R30 +The Spoken Tutorial Project Team conducts workshops using spoken tutorials, +gives certificates to those who pass an online test. -And the answers, +For more details, contact contact@spoken-tutorial.org - 1. We print the given text using the ``echo`` command by using an additional - option -n as, -:: +.. L28 - $echo -n dog man +{{{ Show the ``Acknowledgements'' slide }}} - 2. We can add a new path variable by using the export command as, - -:: +.. R28 - $export PATH=$PATH://data/myscripts +Spoken Tutorial Project is a part of the "Talk to a Teacher" project. +It is supported by the National Mission on Education through ICT, MHRD, +Government of India. More information on this mission is available at the +given link. -.. L31 +.. L29 {{{ Show the Thank you slide }}} -.. R31 +.. R29 Hope you have enjoyed this tutorial and found it useful. Thank you! - |