path: root/lecture_notes/using_linux_tools/handout.rst

Introducing Linux
=================

Linux (sometimes called GNU/Linux) is a Free and Open Source Operating
System that is inspired by Unix and runs on a variety of hardware
platforms.

Free 
    Free as in Freedom or Free Speech, not Free Beer. 

Open-source 
    licensed to permit modifications and redistribution of its source code.

Linux is a modular operating system, with it's basic design based on the
principles established in Unix. It consists of an important and central
piece called the Linux kernel, which, manages system resources like process
control, networking, peripherals and file system access. This is
complemented by the application software, written on top of the kernel that
give the higher level functionality that facilitate the user to carry out
various tasks.

Why Linux?
----------

Free as in Free Beer 
  GNU/Linux can be downloaded in its entirety from the Internet completely
  for free. No registration fees, no costs per user, free updates, and
  freely available source code in case you want to change the behavior of
  your system.

Secure & versatile 
  The security model used in Linux is based on the UNIX idea of security,
  which is known to be robust and of proven quality. Also, there are no
  viruses in the GNU/Linux world.

Why Linux for Scientific Computing?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Free as in Free Speech 
  You can share your operating system and the libraries that you are using
  with your co-workers, without any headaches of licensing. Also, you can
  study and improve the implementation of various libraries that you may
  use for your work

Tools for Scientific computing
  There are a host of tools and libraries, written by various (groups of)
  people that are useful in day-to-day scientific computing work. You have
  the benefit of standing on the shoulders of giants.

Keeps on running 
  GNU/Linux distributions are very stable and known for their up-time. You
  don't have the fear of losing your computational work, due to system
  crashes.

Parallel & Distributed computing 
  It is pretty easy to build your own cluster with GNU/Linux and there are
  host of libraries for parallel and distributed computing that work with
  GNU/Linux.


Reading Exercises
-----------------

  1. In the Beginning was the Command Line -- Neal Stephenson
  #. Linux -- Wikipedia
  #. GNU/Linux naming controversy -- Wikipedia

Getting Started
===============


Logging in
----------

Let's begin with logging into our system. The GNU/Linux OS supports
multiple users and each user logs in with his/her user-name and password.
After the machine boots up, the OS prompts you for a user-name and
password. You can log-in once you provide your authentication details.

It is a popular misconception that GNU/Linux doesn't have a GUI (Graphical
user interface). It does have a fully functional GUI, but for the purpose
of this course we shall start with using the CLI (Command line interface).
Once your system has booted up, hit ``Ctrl + Alt + F1`` to switch to the
command line interface.

You can log out using the ``logout`` command. 

Where am I?
-----------

Now that we have logged in, where are we? Where did we get in? 

To find out the present working directory, we use the ``pwd`` command. 

::

  $ pwd
  /home/user

What is in there?
-----------------

To see what is in the current directory, we use the ``ls`` command. It
gives us a list of all the files in our present working directory.

::

    $ ls
    jeeves.rst psmith.html blandings.html Music

``ls`` command takes the directory, in which we want to see the list of
files present, as an argument. To see all the files present in the
``Music`` directory, we say

::

    $ ls Music
    one.mp3 two.mp3 three.mp3 

Note that everything in GNU/Linux and the Unix world is case sensitive. For
example if we had said ``ls music`` instead of ``ls Music``, we would get
an error ``No such file or directory``.

New folders
-----------

As you can see, our home folder has two html files one rst file and a
directory for Music. What if we wanted the files to be more organized? Say,
we would like to put all our work during this course in a separate
directory. Let us now create a directory ``sees`` by saying

::

    $ mkdir sees

Again, note that we are using all small case letters. ``sees`` is different
from ``Sees`` or ``SEES``. Type ``ls`` to see that a new directory has been
created. 

::

    $ ls

Also, note that special characters need to be escaped. For example if we
wanted to create a directory with the name ``software engineering``, we do
it either as

::

    $ mkdir software\ engineering

or as

::

    $ mkdir "software engineering"

But it is generally a practice to use hyphens or underscores instead of
spaces in filenames and directory names.

In modern GNU/Linux filesystems all characters except the forward slash are
allowed.

Moving around
-------------

Now that we have created our directory ``sees``, let us make it our present
working directory by moving into it. We use the ``cd`` command for this
purpose.

::

    $ cd sees
    $ pwd 
    /home/user/sees/

This could alternately have been written as ``cd ./sees``. The dot in the
beginning specifies that we are specifying the path, relative to the
present working-directory.

To go up the directory structure, we use ``..``. Typing

::

    $ cd ..

in the ``sees`` directory will take us back to the home directory.

What will happen if we type ``cd ..`` in the home folder? We go to the
``/home`` directory.

All this while, we have been using what are called relative paths, to
specify the path. We could alternatively use the absolute path, which give
the whole path, starting with a /. The absolute path of the ``sees``
directory is, ``/home/user/sees/``.

New files
---------

Now that we have seen how to create a new empty directory and navigate into
it, let us create a new blank file. We use the ``touch`` command for this.

::

    $ pwd
    /home/user
    $ cd sees
    $ touch first

This creates a file named touch in our present working directory. Use the
``ls`` command to see that the file has been created.

::

    $ ls 
    first


Getting Help
============

What does a command do?
-----------------------

To get a quick description of the command, we could use the ``whatis``
command. It gives a short one-line description of the command that is
passed as an argument to it. For instance let's see what is the ``touch``
command that we just saw.

::

    $ whatis touch
    touch (1)            - change file timestamps

Now, what does it mean by change file timestamps? We used it to create a
file, just a while ago. To get a more detailed description of the command,
we use the ``man`` command.

::

    $ man touch

This shows the ``man`` (short for "manual pages") page of the command. This
page gives a detailed description of the command. We can see that the
``touch`` command has a whole host of options that can be passed to it.
Every command in Linux has such a list of options that can be passed to the
command to do specific tasks. Hit the ``q`` key to quit the ``man`` page.

To see the manual on man itself do

::

    $ man man

Using additional options
------------------------

As you may have observed, often the ``man`` page is a bit too much for
quickly cross checking what option to use for a specific task. For this
kind of quick look-up, most of the commands come with a -h or --help
option. This gives a brief description of the options available for that
command.

Let us look at using a couple of useful options that we can pass to
commands that we have already see.

::

    $ ls -R

This lists out all the files in the sub-tree of the current directory,
recursively.

When you wish to create a new directory deep inside a directory structure,
using a ``-p`` option with the ``mkdir`` command would be useful. For
example, if we wish to create a folder ``scripts`` inside the directory
``linux-tools`` inside the directory ``sees``, we could simply say,

::

    $ pwd
    /home/user/
    $ mkdir -p sees/linux-tools/scripts

This will create the scripts directory, inside the required directory
structure, creating any other new directory required, to maintain the tree
structure.

Searching for a command
-----------------------

Let's now say, we wish to remove a directory or a file. How do we find out
what command to use? We use the ``apropos`` command to search for commands
based on their descriptions. To search for the command to remove a
file/directory say,

::

    $ apropos remove

This gives us a whole list of commands that have the word ``remove``, in
their description. Looking through the list tells us that ``rm`` or
``rmdir`` is the command to use.


Basic File Handling
===================

Removing files
--------------

``rm``   is used to delete files. 

Here's example to remove a file named "foo" from a directory, 

::

    $ rm foo

Note that, as such, ``rm`` works only for files and not for directories.
For instance, if you try to remove a directory named ``bar``, 

::

    $ rm bar

we get an error saying, cannot remove `bar`: Is a directory. But ``rm``
takes additional arguments which can be used to remove a directory and all
of it's content, including sub-directories. 

::

    $ rm -r bar

removes the directory ``bar`` and all of it's content including
sub-directories, recursively. The ``-r`` stands for recursive. 

A function called ``rmdir`` is also available, to remove directories, but
we shall not look into it. 

Copying Files
-------------

Let's say we wish to copy a file, ``foo`` from ``sees/linux-tools/scripts`` to
``sees/linux-tools``, how would we do it? 

::

    $ pwd 
    /home/user/sees/

    $ cp linux-tools/scripts/foo linux-tools/

In general, 

::

    $ cp SourceFile TargetLocation

Note, that we haven't changed the name of the file name at the target
location. We could have done that by specifying a new filename at the
target location. 

::

    $ cp linux-tools/scripts/foo linux-tools/bar

This copies the file ``foo`` to the new location, but with the new name,
``bar``. 

So, ``cp`` is the command to copy a file from one place to another. The
original file remains unchanged, and the new file may have the same or a
different name.

But, what would have happened if we had a file named ``bar`` already at the
new location? Let's try doing the copy again, and see what happens. 

::

    $ cp linux-tools/scripts/foo linux-tools/bar

We get no error message, what happened? ``cp`` actually overwrites files.
In this case, it's not a problem since, we just re-copied the same content,
but in general it could be a problem, and we could lose data. To prevent
this, we use the ``-i`` flag with ``cp``. 

::

    $ cp -i linux-tools/scripts/foo linux-tools/bar
    cp: overwrite `bar'? 

We are now prompted, whether the file should be over-written. To over-write
say ``y``, else say ``n``. 

Now, let's try to copy the directory ``sees`` to a new directory called
``course``. How do we do it?

::

    $ cd /home/user
    $ cp -i sees course
    cp: omitting directory `sees/'

``cp`` refuses to copy the directory ``sees``. We use the option ``-r``
(recursive) to copy the directory and all it's content. 

::

    $ cd /home/user
    $ cp -ir sees course


Moving Files
------------

What if we want to move files, instead of copying them? One way to go about
it, would be to ``cp`` the file to the new location and ``rm`` the old
file. 

But, there's a command that does this for you, ``mv`` (short for move). It
can move files or directories. It also takes the ``-i`` option to prompt
before overwriting. 

::

    $ cd /home/user
    $ mv -i sees/ course/

What happened? Why didn't we get any prompt? Did course get overwritten? 

::

    $ ls course

We can see that the ``sees`` directory has been inserted as sub-directory
of the ``course`` directory. The move command doesn't over-write
directories, but the ``-i`` option is useful when moving files around.

A common way to rename files (or directories), is to copy a file (or a
directory) to the same location, with a new name. 

::

    $ mv sees/linux-tools sees/linux

will rename the ``linux-tools`` directory to just ``linux``. 


Linux File Hierarchy & Permissions and ownership
================================================

While moving around our files and directories, we have been careful to stay
within the ``/home/`` directory, but if you were curious, you may have
ventured out and seen that there are a lot of other directories. Let us
take this opportunity to understand a few things about the linux file
hierarchy and file permissions. 

::

    $ cd /

The ``/`` directory is called the root directory. All the files and
directories, (even if they are on different physical devices) appear as
sub-directories of the root directory. 

::

    $ ls 

You can see the various directories present at the top most level. Below is
a table that briefly describes, what is present in each of these
directories and what their function is. 

+---------------+------------------------------------------------+
|   Directory   |             Description                        |
+===============+================================================+
| /             | Primary hierarchy root and root directory of   |
|               | the entire file system hierarchy.              |
+---------------+------------------------------------------------+
| /bin/         | Essential command binaries that need to be     |
|               | available in single user mode; for all users,  |
|               | e.g., *cat*, *ls*, *cp*.                       |
+---------------+------------------------------------------------+
| /boot/        | Boot loader files, e.g., *kernels*, *initrd*;  |
|               | often a separate partition.                    |
+---------------+------------------------------------------------+
| /dev/         | Essential devices, e.g., /dev/null             |
+---------------+------------------------------------------------+
| /etc/         | Host-specific system-wide configuration files  |
|               | (the name comes from *et cetera*)              |
+---------------+------------------------------------------------+
| /home/        | User's home directories, containing saved      |
|               | files, personal settings, etc.; often a        |
|               | separate partition.                            |
+---------------+------------------------------------------------+
| /lib/         | Libraries essential for the binaries in        |
|               | */bin/* and */sbin/*                           |
+---------------+------------------------------------------------+
| /media/       | Mount points for removable media such as       |
|               | CD-ROMs, external hard disks, USB sticks, etc. |
+---------------+------------------------------------------------+
| /mnt/         | Temporarily mounted file systems               |
+---------------+------------------------------------------------+
| /opt/         | Optional application software packages         |
+---------------+------------------------------------------------+
| /proc/        | Virtual filesystem documenting kernel and      |
|               | process status as text files; e.g., uptime,    |
|               | network. In Linux, corresponds to a *Procfs*   |
|               | mount.                                         |
+---------------+------------------------------------------------+
| /root/        | Home directory for the root user               |
+---------------+------------------------------------------------+
| /sbin/        | Essential system binaries; e.g., *init*,       |
|               | *route*, *mount*.                              |
+---------------+------------------------------------------------+
| /srv/         | Site-specific data which is served by the      |
|               | system.                                        |
+---------------+------------------------------------------------+
| /tmp/         | Temporary files. Often not preserved between   |
|               | system reboots.                                |
+---------------+------------------------------------------------+
| /usr/         | Secondary hierarchy for read-only user data;   |
|               | contains the majority of (multi-)user          |
|               | utilities and applications.                    |
+---------------+------------------------------------------------+
| /var/         | Variable files - files whose content is        |
|               | expected to continually change during normal   |
|               | operation of the system - such as logs, spool  |
|               | files, and temporary e-mail files.             |
|               | Sometimes a separate partition.                |
+---------------+------------------------------------------------+


Note that some of these directories may or may not be present on your Unix
system depending on whether certain subsystems, such as the X Window
System, are installed.

For more information, it is recommended that you look at the ``man`` page
of ``hier``. 

::

    $ man hier

Permissions and Access control
------------------------------

Let us now look at file permissions. Linux is a multi-user environment and
allows users to set permissions to their files to allow only a set of
people to read or write it. Similarly, it is not "safe" to allow system
files to be edited by any user. All this access control is possible in
Linux. 

To start, in the root directory, say,

::

    $ ls -l

You again get a list of all the sub-directories, but this time with a lot
of additional information. Let us try and understand what this output says. 

::

    drwxr-xr-x   5 root users  4096 Jan 21 20:07 home

The first column denotes the type and the access permissions of the file.
The second is the number of links. The third and fourth are the owner and
group of the file. The next field is the size of the file in bytes. The
next field is the date and time of modification and the last column is the
file name. 

We shall look at the permissions of the file now, ie., the first column of
the output. 

The first character in the first column specifies, whether the item is a
file or a directory. Files have a ``-`` as the first character and
directories have a ``d``. 

The next 9 characters define the access permissions of the file. Before
looking at it, we need to briefly study groups and users and ownership. 

Each file in the Linux filesystem is associated with a user and a group.
The user and the group of the file can be seen in the third and the fourth
columns of the output of ``ls -l`` command. The third column is the user,
and is usually the person who has created the file. A group is simply a
group of users. Users can be added or removed from groups, but doing that
is out of the scope of this course. This brief introduction to users and
groups is enough to go ahead and understand access permissions. 

We already know what the first character in the first column (in the output
of ``ls -l``) is for. The rest of the 9 characters are actually sets of 3
characters of each. The first set of 3 characters defines the permissions
of the user, the next 3 is for the group and the last three is for others.
Based on the values of these characters, access is provided or denied to
files, to each of the users. 

So, what does each of the three characters stand for? Let's suppose we are
looking at the set, corresponding to the permissions of the user. In the
three characters, the first character can either be an ``r`` or a ``-``.
Which means, the user can either have the permissions to read the file or
not. If the character is ``r``, then the user has the permissions to read
the file, else not. Similarly, ``w`` stands for write permissions and
decides whether the user is allowed to write to the file. ``x`` stands for
execute permissions. You cannot execute a file, if you do not have the
permissions to execute it. 

Similarly, the next set of characters decides the same permissions for the
members of the group, that the file is associated with. The last set of
characters defines these permissions for the users, who are neither owners
of the file nor in the group, with which the file is associated. 

Changing the permissions
------------------------

Now, it's not as if these permissions are set in stone. If you are the
owner of a file, you can change the permissions of a file, using the
``chmod`` command. 

Let's say, we wish to give the execute permissions for a file, to both the
user and the group, how do we go about doing it? To be more explicit, given
a file ``foo.sh``, with the permissions flags as ``-rw-r--r--``, change it
to ``-rwxr-xr--``. 

The following command does it for us, 

::

    $ chmod ug+x foo.sh
    $ ls -l foo.sh

As you can see, the permissions have been set to the required value. But
what did we exactly do? Let us try and understand. 

Symbolic modes
~~~~~~~~~~~~~~

In the command above, the parameter ``ug+x`` is the mode parameter to the
``chmod`` command. It specifies the changes that need to be made to the
permissions of the file ``foo.sh``. 

The ``u`` and ``g`` stand for the user and group, respectively. The ``x``
stands for the execute permission and the ``+`` stands for adding the
specified permission. So, essentially, we are asking ``chmod`` command to
add the execute permission for the user and group. The permission of others
will remain unchanged. 

The following three tables give the details of the class, the operator and
the permissions. 

+--------------+--------+---------------------------------------------+
| Reference    | Class  |                Description                  |
+==============+========+=============================================+
|      u       | user   | the owner of the file                       |
+--------------+--------+---------------------------------------------+
|      g       | group  | users who are members of the file's group   |
+--------------+--------+---------------------------------------------+
|      o       | others | users who are not hte owner of the file or  |
|              |        | members of the group                        |
+--------------+--------+---------------------------------------------+
|      a       | all    | all three of the above; is the same as *ugo*|
+--------------+--------+---------------------------------------------+

+--------------+------------------------------------------------------+
| Operator     |                      Description                     |
+==============+======================================================+
| +            | adds the specified modes to the specified classes    |
+--------------+------------------------------------------------------+
| -            | removes the specified modes from the specified       |
|              | classes                                              |
+--------------+------------------------------------------------------+
| =            | the modes specified are to be made the exact modes   |
|              | for the specified classes                            |
+--------------+------------------------------------------------------+

+-----+--------------+------------------------------------------------+
|Mode |    Name      |                 Description                    |
+=====+==============+================================================+
| r   | read         | read a file or list a directory's contents     |
+-----+--------------+------------------------------------------------+
| w   | write        | write to a file or directory                   |   
+-----+--------------+------------------------------------------------+
| x   | execute      | execute a file or recurse a directory tree     |
+-----+--------------+------------------------------------------------+

So, if we wished to add the execute permission to all the users, instead of
adding it to just the user and group, we would have instead said 

::

    $ chmod a+x foo.sh 

or 

::

    $ chmod ugo+x foo.sh


To change the permissions of a directory along with all of its
sub-directories and files, recursively, we use the ``-R`` option. 

For instance if we wished to remove the read permissions of a file from all
users except the owner of the file, we would say, 


::

    $ chmod go-r bar.txt

It is important to note that the permissions of a file can only be changed
by a user who is the owner of a file or the superuser. (We shall talk about
the superuser in the next section)


Changing Ownership of Files
---------------------------

What if we wish to change the ownership of a file? The ``chown`` command is
used to change the owner and group. 

By default, the owner of a file (or directory) object is the user that
created it. The group is a set of users that share the same access
permissions (i.e., read, write and execute). 

For instance, to change the user and the group of the file
``wonderland.txt`` to ``alice`` and ``users``, respectively, we say.

    $ chown alice:users wonderland.txt

What does it say? We get an error saying, the operation is not permitted.
We have attempted to change the ownership of a file that we own, to a
different user. Logically, this shouldn't be possible, because, this can
lead to problems, in a multi-user system. 

Only the superuser is allowed to change the ownership of a file from one
user to another. The superuser or the ``root`` user is the only user
empowered to a certain set of tasks and hence is called the superuser. The
command above would have worked, if you did login as the superuser and
then changed the ownership of the file. 

We shall end our discussion of the Linux hierarchy and file permissions
here. Let us look at working with text, files and the role of the command
shell in the next section. 

Looking at files
================

cat
---

The ``cat`` command is the most commonly used command to display the
contents of files. To view the contents of a file, say, ``foo.txt``, we
simply say, 

::

    $ cat foo.txt

The contents of the file are shown on the terminal. 

The cat command could also be used to concatenate the text of multiple
files. (It's name actually comes from there). Say, we have two files,
``foo.txt`` and ``bar.txt``, 

::

    $ cat foo.txt bar.txt

shows the output of both the files concatenated on the standard output. 

But if we had a long file, like ``wonderland.txt``, the ouptut of ``cat``
command is not convenient to read. Let's look at the ``less`` command which
turns out to be more useful in such a case. 


less
----

``less `` allows you to view the contents of a text file one screen at a
time. 

::

    $ less wonderland.txt

will give show us the file, one screen at a time. 

``less`` has a list of commands that it allows you to use, once you have
started viewing a file. A few of the common ones have been listed below. 

    * q: Quit.

    * [Arrows]/[Page Up]/[Page Down]/[Home]/[End]: Navigation.

    * ng: Jump to line number n. Default is the start of the file.

    * /pattern: Search for pattern. Regular expressions can be used.

    * h: Help.

wc
--

Often we just would like to get some statistical information about the
file, rather than viewing the contents of the file. The ``wc`` command
prints these details for a file. 

::

    $ wc wonderland.txt

The first number is the number of lines, the second is the number of words
and the third is the number of characters in the file. 

head & tail
-----------

Let us now look at a couple of commands that let you see parts of files,
instead of the whole file. ``head`` and ``tail`` let you see parts of
files, as their names suggest, the start and the end of a file,
respectively. 

::

    $ head wonderland.txt

will print the first 10 lines of the file. Similarly tail will print the
last 10 lines of the file. If we wish to change the number of lines that we
wish to view, we use the option ``-n``. 

::

    $ head -n 1 wonderland.txt

will print only the first line of the file. Similarly, we could print only
the last line of the file. 

The most common use of the tail command is to monitor a continuously
changing file, for instance a log file. Say you have a process running,
which is continuously logging it's information to a file, for instance the
logs of the system messages. 

::

	$ tail -f /var/log/dmesg

This will show the last 10 lines of the file as expected, but along with
that, start monitoring the file. Any new lines added at the end of the
file, will be shown. To interrupt, tail while it is monitoring, hit
``Ctrl-C``. Ctrl-C is used to stop any process that is running from your
current shell. 

cut & paste
-----------

We looked at a couple of functions that allow you to view a part of files,
line-wise. We shall now look at a couple of commands that allow you to look
at only certain sections of each line of a file and merge those parts.

Let's take the ``/etc/passwd`` file as our example file. It contains
information about each user of the system. 

::

    root:x:0:0:root:/root:/bin/bash
    bin:x:1:1:bin:/bin:/bin/false
    daemon:x:2:2:daemon:/sbin:/bin/false
    mail:x:8:12:mail:/var/spool/mail:/bin/false
    ftp:x:14:11:ftp:/srv/ftp:/bin/false
    http:x:33:33:http:/srv/http:/bin/false

Let us look at only the first, fifth, sixth and the last columns. The first
column is the user name, the fifth column is the user info, the sixth
column is the home folder and the last column is the path of the shell
program that the user uses. 

Let's say we wish to look at only the user names of all the users in the
file, how do we do it?

::
    
    $ cut -d : -f 1 /etc/passwd

gives us the required output. But what are we doing here? 

The first option ``-d`` specifies the delimiter between the various fields in
the file, in this case it is the semicolon. If no delimiter is specified,
the TAB character is assumed to be the delimiter. The ``-f`` option specifies,
the field number that we want to choose. 

You can print multiple fields, by separating the field numbers with a
comma. 

::
    
    $ cut -d : -f 1,5,7 /etc/passwd

prints only the first, fifth and the seventh fields. 

Instead of choosing by fields, ``cut`` also allows us to choose on the
basis of characters or bytes. For instance, we could get the first 4
characters of all the entries of the file, ``/etc/passwd`` 

::

    $ cut -c 1-4 /etc/passwd 

The end limits of the ranges can take sensible default values, if they are
left out. For example, 

::

    $ cut -c -4 /etc/passwd 

gives the same output as before. If the start position has not been
specified, it is assumed to be the start of the line. Similarly if the end
position is not specified, it is assumed to be the end of the line. 

::

    $ cut -c 10- /etc/passwd 

will print all the characters from the 10th character up to the end of the
line. 

Let us now solve the inverse problem. Let's say we have two columns of data
in two different files, and we wish to view them side by side. 

For instance, given a file containing the names of students in a file, and
another file with the marks of the students, we wish to view the contents,
side by side. ``paste`` command allows us to do that. 

Contents of students.txt

::
     
     Hussain
     Dilbert
     Anne
     Raul
     Sven    

Contents of marks.txt

::

     89 92 85
     98 47 67
     67 82 76
     78 97 60
     67 68 69

::

    $ paste students.txt marks.txt

    $ paste -s students.txt marks.txt


The first command gives us the output of the two files, next to each other
and the second command gives us the output one below the other. 

Now, this problem is a bit unrealistic because, we wouldn't have the marks
of students in a file, without any information about the student to which
they belong. Let's say our marks file had the first column as the roll
number of the student, followed by the marks of the students. What would we
then do, to get the same output that we got before? 

Essentially we need to use both, the ``cut`` and ``paste`` commands, but
how do we do that? That brings us to the topic of Redirection and Piping. 

The Command Shell
=================

Redirection and Piping
----------------------

Let's say the contents of ``marks1.txt`` are as follows, 

::

     5 89 92 85
     4 98 47 67
     1 67 82 76
     2 78 97 60
     3 67 68 69

The solution would be as below

::

    $ cut -d " " -f 2- marks1.txt | paste -d " " students.txt -

or 

::

    $ cut -d " " -f 2- marks1.txt > /tmp/m_tmp.txt
    $ paste -d " " students.txt m_tmp.txt


Let's first try to understand the second solution, which is a two step
solution. Later, we shall look at the first solution. 

Redirecting
~~~~~~~~~~~

The standard output (stdout), in general, streams (or goes) to the display.
Hence, the output of the commands that we type, come out to the display.
This may not always be what we require. 

For instance, in the solution above, we use the cut command and get only
the required columns of the file and write the output to a new temporary
file. The ``>`` character is used to state that we wish to redirect the
output, and it is followed by the location to which we wish to redirect. 

::

    $ command > file1

In general, this creates a new file at the specified location, to which the
output is written. But, if we wish to append the output to an existing
file, we use ``>>``.

Similarly, the standard input (stdin) is assumed to be from the keyboard.
Instead we could redirect the input from a file. 

::

    $ command < file1

The input and the output redirection could be combined in a single command. 

::

    $ command < infile > outfile


There is actually a third kind of standard stream, called the Standard
error (stderr). Any error messages that you get, are coming through this
stream. Like ``stdout``, ``stderr`` also streams to the display, by default
but it could be redirected to a file, as well. 

For instance, let's introduce an error into the ``cut`` command used
before. We change the ``-f`` option to ``-c`` 

::

    $ cut -d " " -c 2- marks1.txt > /tmp/m_tmp.txt

This prints an error that says the delimiter option should be used with the
fields option only, and you can verify that the ``m_tmp.txt`` file is
empty.  We can now, redirect the ``stderr`` also to a file, instead of
showing it on the display. 

::

    $ cut -d " " -f 2- marks1.txt 1> /tmp/m_tmp.txt 2> /tmp/m_err.txt

The above command redirects all the errors to the ``m_err.txt`` file
and the output to the ``m_tmp.txt`` file. When redirecting, 1 stands
for ``stdout`` and 2 stands for ``stderr``. That brings us to the end of
the discussion on redirecting. 

The second command in the solution of the problem is trivial to understand. 
::

    $ paste -d " " students.txt m_tmp.txt

So, in two steps we solved the problem of getting rid of the roll numbers
from the marks file and displaying the marks along with the names of the
students. Now, that we know how to redirect output, we could choose to
write the output to a file, instead of showing on the display. 

Piping
~~~~~~

Let us now look at the first solution. 

::

    $ cut -d " " -f 2- marks1.txt | paste -d " " students.txt -

First of all, the hyphen at the end is to ask the paste command to read the
standard input, instead of looking for a FILE. The ``man`` page of ``paste``
command gives us this information. 

Now, what is happening with the ``cut`` command. It is a normal ``cut`` 
command, if we looked at the command only up to the ``|`` character. So,
the ``|`` seems to be joining the commands in some way. 

Essentially, what we are doing is, to redirect the output of the first
command to the ``stdin`` and the second command takes input from the
``stdin``. 

More generally, 

::

    $ command1 | command2

executes ``command1`` and sends it's output to the ``stdin``, which is then
used as the input for the ``command2``. This activity is commonly called
piping, and the character ``|`` is called a pipe. 

This is roughly equivalent to using two redirects and a temporary file 

::

    $ command1 > tempfile
    $ command2 < tempfile
    $ rm tempfile

Also, given that a pipe is just a way to send the output of the command to
the ``stdin``, it should be obvious, to you that we can use a chain of
pipes. Any number of commands can be piped together and you need not be
restricted to two commands. 

Using piping and redirection, we can do a whole bunch of complex tasks
combined with the commands we have already looked at, and other commands
that we are going to look at. 

Features of the Shell
---------------------

The Bash shell has some nice features, that make our job of using the shell
easier and much more pleasant. We shall look at a few of them, here. 

Tab-completion
~~~~~~~~~~~~~~

Bash provides the feature of tab completion. What does tab completion mean?
When you are trying to type a word, bash can complete the word for you,
if you have entered enough portion of the word (to complete it
unambiguously) and then hit the tab key. 

If on hitting the tab key, the word doesn't get completed, either the word
doesn't exist or the word cannot be decided unambiguously. If the case is
the latter one, hitting the tab key a second time, will list the
possibilities. 

Bash provides tab completion for the following. 

  1. File Names
  2. Directory Names
  3. Executable Names
  4. User Names (when they are prefixed with a ~)
  5. Host Names (when they are prefixed with a @)
  6. Variable Names (when they are prefixed with a $) 

For example, 

::

    $ pas<TAB>
    $ $PA<TAB>
    $ ~/<TAB><TAB>

History
~~~~~~~

Bash also saves the history of the commands you have typed. So, you can go
back to a previously typed command. Use the up and down arrow keys to
navigate in your bash history. 

::

    $ <UP-ARROW>

You can also search incrementally, for commands in your bash history.
``Ctrl-r`` search for the commands that you have typed before. But, note
that the number of commands saved in the history is limited, generally upto
a 1000 commands. 

::

   $ <Ctrl-r> pas


Shell Meta Characters
~~~~~~~~~~~~~~~~~~~~~

Unix recognizes certain special characters, called "meta characters," as
command directives. The shell meta characters are recognized anywhere they
appear in the command line, even if they are not surrounded by blank space.
For that reason, it is safest to only use the characters A-Z, a-z, 0-9, and
the period, dash, and underscore characters when naming files and
directories on Unix. If your file or directory has a shell meta character
in the name, you will find it difficult to use the name in a shell command.

The shell meta characters include:

\ / < > ! $ % ^ & * | { } [ ] " ' ` ~ ; 


As an example,

::

    $ ls file.*

run on a directory containing the files file, file.c, file.lst, and myfile
would list the files file.c and file.lst. However,

::

    $ ls file.?

run on the same directory would only list file.c because the ? only matches
one character, no more, no less. This can save you a great deal of typing
time. 

For example, if there is a file called
california_cornish_hens_with_wild_rice and no other files whose names begin
with 'c', you could view the file without typing the whole name by typing
this

::

    $ more c*

because the c* matches that long file name.

File-names containing metacharacters can pose many problems and should
never be intentionally created.

More text processing
====================

``sort``
--------

Let's continue with the previous problem of the students and their marks,
that we had. Let's say we wish to sort the output in the alphabetical order
of the names of the files. We can use the ``sort`` command for this
purpose.

We just pipe the previous output to the ``sort`` command. 

::

    $ cut -d " " -f 2- marks1.txt | paste -d " " students.txt -| sort

Let's say we wished to sort the names, based on the marks in the first
subject (first column after the name). ``sort`` command also allows us to
specify the delimiter between the fields and sort the data on a particular
field. ``-t`` option is used to specify the delimiter and the ``-k`` option
is used to specify the field. 

::

    $ cut -d " " -f 2- marks1.txt | paste -d " " students.txt -| sort -t " " -k 2

The above command give us a sorted output as required. But, it would be
nicer to have the output sorted in the reverse order. ``-r`` option allows
the output to be sorted in the reverse order and the ``-n`` option is used
to choose a numerical sorting. 

::

    $ cut -d " " -f 2- marks1.txt | paste -d " " students.txt -| sort -t " " -k 2 -rn    

``grep``
--------

While you are compiling the student marklist, Anne walks up to you and
wants to know her marks. You, being the kind person that you are, oblige.
But you do not wish to her to see the marks that others have scored. What
do you do? The ``grep`` command comes to your rescue. 

``grep`` is a command line text search utility. You can use it to search
for Anne and show her, what she scored. ``grep`` allows you to search for a
search string in files. But you could, like any other command, pipe the
output of other commands to it. So, we shall use the previous combination
of cut and paste that we had, to get the marks of students along with their
names and search for Anne in that. 

::

    $ cut -d " " -f 2- marks1.txt | paste -d " " students.txt - | grep Anne 

This will give you only the line containing the word Anne as the output.
The grep command is by default case-sensitive. So, you wouldn't have got
the result if you had searched for anne instead of Anne. But, what if you
didn't know, whether the name was capitalized or not? ``grep`` allows you
to do case-insensitive searches by using the ``-i`` option. 

::

    $ cut -d " " -f 2- marks1.txt | paste -d " " students.txt - | grep -i Anne 

Now, in another scenario, if you wished to print all the lines, which do
not contain the word Anne, you could use the ``-v`` option. 

::

    $ cut -d " " -f 2- marks1.txt | paste -d " " students.txt - | grep -iv Anne

Grep allows you to do more complex searches, for instance searching for
sentences starting or ending with a particular pattern and regular
expression based searches. You shall learn about these, as a part of your
lab exercises. 

``tr``
------

``tr`` is a command that takes as parameters two sets of characters, and
replaces occurrences of the characters in the first set with the
corresponding elements from the other set. It reads from the standard
output and writes to the standard output. 

For instance if you wished to replace all the lower case letters in the
students file with upper case, 

::

    $ cat students.txt | tr a-z A-Z

A common task is to remove empty newlines from a file. The ``-s`` flag
causes ``tr`` to compress sequences of identical adjacent characters in its
output to a single token. For example,

::

    $ tr -s '\n' '\n'

replaces sequences of one or more newline characters with a single newline.

The ``-d`` flag causes ``tr`` to delete all tokens of the specified set of
characters from its input. In this case, only a single character set
argument is used. The following command removes carriage return characters,
thereby converting a file in DOS/Windows format to the Unix format. 

::

    $ cat foo.txt | tr -d '\r' > bar.txt

The ``-c`` flag complements the first set of characters.

::

    $ tr -cd '[:alnum:]' 

therefore removes all non-alphanumeric characters.

``uniq``
--------

Suppose we have a list of items, say books, and we wish to obtain a list which names of all the books only once, without any duplicates. We use the ``uniq`` command to achieve this. 

::

  Programming Pearls
  The C Programming Language
  The Mythical Man Month: Essays on Software Engineering 
  Programming Pearls
  The C Programming Language
  Structure and Interpretation of Computer Programs
  Programming Pearls
  Compilers: Principles, Techniques, and Tools
  The C Programming Language
  The Art of UNIX Programming
  Programming Pearls
  The Art of Computer Programming
  Introduction to Algorithms
  The Art of UNIX Programming
  The Pragmatic Programmer: From Journeyman to Master
  Programming Pearls
  Unix Power Tools
  The Art of UNIX Programming

Let us try and get rid of the duplicate lines from this file using the ``uniq`` command. 

::

  $ uniq items.txt 
  Programming Pearls
  The C Programming Language
  The Mythical Man Month: Essays on Software Engineering 
  Programming Pearls
  The C Programming Language
  Structure and Interpretation of Computer Programs
  Programming Pearls
  Compilers: Principles, Techniques, and Tools
  The C Programming Language
  The Art of UNIX Programming
  Programming Pearls
  The Art of Computer Programming
  Introduction to Algorithms
  The Art of UNIX Programming
  The Pragmatic Programmer: From Journeyman to Master
  Programming Pearls
  Unix Power Tools
  The Art of UNIX Programming

Nothing happens! Why? The ``uniq`` command removes duplicate lines only when they are next to each other. So, we get a sorted file from the original file and work with that file, henceforth. 

::

  $ sort items.txt | uniq
  Compilers: Principles, Techniques, and Tools
  Introduction to Algorithms
  Programming Pearls
  Structure and Interpretation of Computer Programs
  The Art of Computer Programming
  The Art of UNIX Programming
  The C Programming Language
  The Mythical Man Month: Essays on Software Engineering 
  The Pragmatic Programmer: From Journeyman to Master
  Unix Power Tools

``uniq -u`` command gives the lines which are unique and do not have any duplicates in the file. ``uniq -d`` outputs only those lines which have duplicates. The ``-c`` option displays the number of times each line occurs in the file. 

::

  $ uniq -u items-sorted.txt 
  Compilers: Principles, Techniques, and Tools
  Introduction to Algorithms
  Structure and Interpretation of Computer Programs
  The Art of Computer Programming
  The Mythical Man Month: Essays on Software Engineering 
  The Pragmatic Programmer: From Journeyman to Master
  Unix Power Tools

  $ uniq -dc items-sorted.txt      
  5 Programming Pearls
  3 The Art of UNIX Programming
  3 The C Programming Language

That brings us to the end of our discussion on text processing. Text
processing is an art and there is a lot more to it, than could have been
covered in this short introduction. But, we hope that the tools you learned
to use here, will help you solve a great deal of problems. 

Basic editing and editors
=========================

vim
---

Vim is a very powerful editor. It has a lot of commands, and all of them
cannot be explained here. We shall try and look at a few, so that you can
find your way around in vim.

To open a file in vim, we pass the filename as a parameter to the ``vim``
command. If a file with that filename does not exist, a new file is
created. 

::

    $ vim first.txt

To start inserting text into the new file that we have opened, we need to
press the ``i`` key. This will take us into the *insert* mode from the
*command* mode. Hitting the ``esc`` key, will bring us back to the
*command* mode. There is also another mode of vim, called the *visual* mode
which will be discussed later in the course.

In general, it is good to spend as little time as possible in the insert
mode and extensively use the command mode to achieve various tasks.

To save the file, use ``:w`` in the command mode. From here on, it is
understood that we are in the command mode, whenever we are issuing any
command to vim.

To save a file and continue editing, use ``:w FILENAME`` The file name is
optional. If you do not specify a filename, it is saved in the same file
that you opened. If a file name different from the one you opened is
specified, the text is saved with the new name, but you continue editing
the file that you opened. The next time you save it without specifying a
name, it gets saved with the name of the file that you initially opened.

To save file with a new name and continue editing the new file, use ``:saveas FILENAME``

To save and quit, use ``:wq``

To quit, use ``:q``

To quit without saving, use ``:q!``

Moving around
~~~~~~~~~~~~~

While you are typing in a file, it is in-convenient to keep moving your
fingers from the standard position for typing to the arrow keys. Vim,
therefore, provides alternate keys for moving in the document. Note again
that, you should be in the command mode, when issuing any commands to vim.

The basic cursor movement can be achieved using the keys, ``h`` (left),
``l`` (right), ``k`` (up) and ``j`` (down).

::
 
             ^
             k              
       < h       l >        
             j              
             v

Note: Most commands can be prefixed with a number, to repeat the command.
For instance, ``10j`` will move the cursor down 10 lines.

Moving within a line
++++++++++++++++++++

+----------------------------------------+---------+
| Cursor Movement                        | Command | 
+========================================+=========+
| Beginning of line                      | ``0``   |
+----------------------------------------+---------+
| First non-space character of line      | ``^``   |
+----------------------------------------+---------+
| End of line                            | ``$``   |
+----------------------------------------+---------+
| Last non-space character of line       | ``g_``  |
+----------------------------------------+---------+

Moving by words and sentences
+++++++++++++++++++++++++++++

+------------------------------+---------+
| Cursor Movement              | Command |
+==============================+=========+
| Forward, word beginning      | ``w``   |
+------------------------------+---------+
| Backward, word beginning     | ``b``   |
+------------------------------+---------+
| Forward, word end            | ``e``   |
+------------------------------+---------+
| Backward, word end           | ``ge``  |
+------------------------------+---------+
| Forward, sentence beginning  | ``)``   |
+------------------------------+---------+
| Backward, sentence beginning | ``(``   |
+------------------------------+---------+
| Forward, paragraph beginning | ``}``   |
+------------------------------+---------+
| Backward, paragraph beginning| ``{``   |
+------------------------------+---------+

More movement commands
++++++++++++++++++++++

+---------------------------------+------------+
| Cursor Movement                 | Command    |
+=================================+============+
| Forward by a screenful of text  | ``C-f``    |
+---------------------------------+------------+
| Backward by a screenful of text | ``C-b``    |
+---------------------------------+------------+
| Beginning of the screen         | ``H``      |
+---------------------------------+------------+
| Middle of the screen            | ``M``      |
+---------------------------------+------------+
| End of the screen               | ``L``      |
+---------------------------------+------------+
| End of file                     | ``G``      |
+---------------------------------+------------+
| Line number ``num``             | ``[num]G`` |
+---------------------------------+------------+
| Beginning of file               | ``gg``     |
+---------------------------------+------------+
| Next occurrence of the text     | ``*``      |
| under the cursor                |            |
+---------------------------------+------------+
| Previous occurrence of the text | ``#``      |
| under the cursor                |            |
+---------------------------------+------------+

Note: ``C-x`` is ``Ctrl`` + ``x``

The visual mode
~~~~~~~~~~~~~~~

The visual mode is a special mode that is not present in the original vi
editor. It allows us to highlight text and perform actions on it. All the
movement commands that have been discussed till now work in the visual mode
also. The editing commands that will be discussed in the future work on the
visual blocks selected, too.

Editing commands
~~~~~~~~~~~~~~~~

The editing commands usually take the movements as arguments. A movement is
equivalent to a selection in the visual mode. The cursor is assumed to have
moved over the text in between the initial and the final points of the
movement. The motion or the visual block that's been highlighted can be
passed as arguments to the editing commands.

+-------------------------+---------+
| Editing effect          | Command |
+=========================+=========+
| Cutting text            | ``d``   |
+-------------------------+---------+
| Copying/Yanking text    | ``y``   |
+-------------------------+---------+
| Pasting copied/cut text | ``p``   |
+-------------------------+---------+

The cut and copy commands take the motions or visual blocks as arguments
and act on them. For instance, if you wish to delete the text from the
current text position to the beginning of the next word, type ``dw``. If
you wish to copy the text from the current position to the end of this
sentence, type ``y)``.

Apart from the above commands, that take any motion or visual block as an
argument, there are additional special commands.

+----------------------------------------+---------+
| Editing effect                         | Command | 
+========================================+=========+
| Cut the character under the cursor     | ``x``   |
+----------------------------------------+---------+
| Replace the character under the        | ``ra``  |
| cursor with ``a``                      |         |
+----------------------------------------+---------+
| Cut an entire line                     | ``dd``  |
+----------------------------------------+---------+
| Copy/yank an entire line               | ``yy``  |
+----------------------------------------+---------+

Note: You can prefix numbers to any of the commands, to repeat them.

Undo and Redo
~~~~~~~~~~~~~
You can undo almost anything using ``u``. 

To undo the undo command type ``C-r``

Searching and Replacing
~~~~~~~~~~~~~~~~~~~~~~~

+-----------------------------------------+---------+
| Finding                                 | Command |
+=========================================+=========+
| Next occurrence of ``text``, forward    |``\text``|
+-----------------------------------------+---------+
| Next occurrence of ``text``, backward   |``?text``|
+-----------------------------------------+---------+
| Search again in the same direction      | ``n``   |
+-----------------------------------------+---------+
| Search again in the opposite direction  | ``N``   |
+-----------------------------------------+---------+
| Next occurrence of ``x`` in the line    | ``fx``  |
+-----------------------------------------+---------+
| Previous occurrence of ``x`` in the line| ``Fx``  |
+-----------------------------------------+---------+

+---------------------------------------+------------------+
| Finding and Replacing                 |  Command         |
+=======================================+==================+
| Replace the first instance of ``old`` |``:s/old/new``    |
| with ``new`` in the current line.     |                  |
+---------------------------------------+------------------+
| Replace all instances of ``old``      |``:s/old/new/g``  |
| with ``new`` in the current line.     |                  |
+---------------------------------------+------------------+
| Replace all instances of ``old``      |``:s/old/new/gc`` |
| with ``new`` in the current line,     |                  |
| but ask for confirmation each time.   |                  |
+---------------------------------------+------------------+
| Replace the first instance of ``old`` |``:%s/old/new``   |
| with ``new`` in the entire file.      |                  |
+---------------------------------------+------------------+
| Replace all instances of ``old``      |``:%s/old/new/g`` |
| with ``new`` in the entire file.      |                  |
+---------------------------------------+------------------+
| Replace all instances of ``old`` with |``:%s/old/new/gc``|
| ``new`` in the entire file but ask    |                  |
| for confirmation each time.           |                  |
+---------------------------------------+------------------+

SciTE
-----

SciTE is a *source code* editor, that has a feel similar to the commonly
used GUI text editors. It has a wide range of features that are extremely
useful for a programmer, editing code. Also it aims to keep configuration
simple, and the user needs to edit a text file to configure SciTE to
his/her liking.

Opening, Saving, Editing files with SciTE is extremely simple and trivial.
Knowledge of using a text editor will suffice.

SciTE can syntax highlight code in various languages. It also has
auto-indentation, code-folding and other such features which are useful
when editing code.

SciTE also gives you the option to (compile and) run your code, from within
the editor.

Simple Shell Scripts
====================

A shell script is simply a sequence of commands, that are put into a file,
instead of entering them one by one onto the shell. The script can then be
run, to run the sequence of commands in a single shot instead of manually
running, each of the individual commands. 

For instance, let's say we wish to create a directory called ``marks`` in the
home folder and save the results of the students into a file
``results.txt``. 

We open our editor and save the following text to ``results.sh``

::

    #!/bin/bash
    mkdir ~/marks
    cut -d " " -f 2- marks1.txt | paste -d " " students.txt - | sort > ~/marks/results.txt

We can now run the script, 

::

    $ ./results.sh

We get an error saying, Permission denied! Why? Can you think of the
reason? (Hint: ``ls -l``). Yes, the file doesn't have execute permissions.
We make the file executable and then run it. 

::

    $ chmod u+x results.sh
    $ ./results.sh

We get back the prompt. We can check the contents of the file
``results.txt`` to see if the script has run. 

So, here, we have our first shell script. We understand almost all of it,
except for the first line of the file. The first line is used to specify
the interpreter or shell which should be used to execute the script. In
this case, we are asking it to use the bash shell. 

Once, the script has run, we got back the prompt. We had to manually check,
if the contents of the file are correct, to see if the script has run. It
would be useful to have our script print out messages. For this, we can use
the ``echo`` command. We can edit our ``results.sh`` script, as follows. 

::

    #!/bin/bash
    mkdir ~/marks
    cut -d " " -f 2- marks1.txt | paste -d " " students.txt - | sort > ~/marks/results.txt
    echo "Results generated."

Now, on running the script, we get a message on the screen informing us,
when the script has run. 

Let's now say, that we wish to let the user decide the file to which the
results should be written to. The results file, should be specifiable by an
argument in the command line. We can do so, by editing the file, as below. 

::

    #!/bin/bash
    mkdir ~/marks
    cut -d " " -f 2- marks1.txt | paste -d " " students.txt - | sort > ~/marks/$1
    echo "Results generated."


The ``$1`` above, corresponds to the first command line argument to the
script. So, we can run the script as shown below, to save the results to
``grades.txt``. 

::

    $ ./results.sh grades.txt    

When we run the ``results.sh`` file, we are specifying the location of the
script by using ``./``. But for any of the other commands (even if they may
not be shell scripts), we didn't have to specify their locations. Why? The
shell has a set of locations where it searches, for the command that we are
trying to run. These set of locations are saved in an "environment"
variable called PATH. We shall look at environment variables, again, later.
But, let us look at what the value of the PATH variable is. To view the
values of variables, we can use the echo command.

::

    $ echo $PATH

So, these are all the paths that are searched, when looking to execute a
command. If we put the results.sh script in one of these locations, we
could simply run it, without using the ``./`` at the beginning. 

Variables
---------

As expected, it is possible to define our own variables inside our shell
scripts. For example,

::

    name="FOSSEE"

creates a new variable ``name`` whose value is ``FOSSEE``. To refer to this
variable, inside our shell script, we would refer to it, as ``$name``.
**NOTE** that there is no space around the ``=`` sign. 

::

    ls $name*

It is possible to store the output of a command in a variable, by enclosing
the command in back-quotes. 

::

    count=`wc -l wonderland.txt`

saves the number of lines in the file ``wonderland.txt`` in the variable
count. 

Comments
--------

The ``#`` character is used to comment out content from a shell script.
Anything that appears after the ``#`` character in a line, is ignored by
the bash shell. 

Control structures and Operators
================================

We can have if-else constructs, for and while loops in bash. Let us look at
how to write them, in this section. 

To write an if, or an if-else construct, we need to check or test for a
condition. ``test`` command allows us to test for conditions. ``test`` has
a whole range of tests that can be performed. The man page of ``test``
gives a listing of various types of tests that can be performed with it. 

Let's write a simple script with an ``if`` condition that tests whether a
directory with a particular name, is present or not.

``if``
------

Let's save the following code to the script ``dir-test.sh``

::

    #!/bin/bash
    if test -d $1
    then
      echo "Yes, the directory" $1 "is present"
    fi

When the script is run with an argument, it prints a message, if a
directory with that name exists in the current working directory. 

``if`` - ``else``
-----------------

Let's write a simple script which returns back whether the argument passed
is negative or not

::

   #!/bin/bash
   if test $1 -lt 0
   then
     echo "number is negative"
   else
     echo "number is non-negative"
   fi

We can run the file with a set of different inputs and see if it works. 

::

   $ ./sign.sh -11

Instead of using the ``test`` command, square brackets can also be used. 

::

   #!/bin/bash
   if [ $1 -lt 0 ]
   then
     echo "number is negative"
   else
     echo "number is non-negative"
   fi

Note that the spacing is important, when using the square brackets. ``[``
should be followed by a space and ``]`` should be preceded by a space. 

Let's create something interesting using the if-else clause. Let's write a
script, that greets the user, based on the time. 

::

   #!/bin/sh
   # Script to greet the user according to time of day
   hour=`date | cut -c12-13`
   now=`date +"%A, %d of %B, %Y (%r)"`
   if [ $hour -lt 12 ]
   then
     mess="Good Morning $LOGNAME, Have a nice day!"
   fi

   if [ $hour -gt 12 -a $hour -le 16 ]
   then
     mess="Good Afternoon $LOGNAME"
   fi

   if [ $hour -gt 16 -a $hour -le 18 ]
   then
     mess="Good Evening $LOGNAME"
   fi
   echo -e "$mess\nIt is $now"

There a couple of new things, in this script. ``$LOGNAME`` is another
environment variable, which has the login name of the user. The variables
``hour`` and ``now`` are actually taking the output of the commands that
are placed in the back quotes. 

Let us now see how to run loops in bash. We shall look at the ``for`` and
the ``while`` loops. 

``for``
-------

Suppose we have a set of files, that have names beginning with numbers
followed by their names - ``08 - Society.mp3``. We would like to rename
these files to remove the numbering. How would we go about doing that? 

It is clear from the problem statement that we could loop over the list of
files and rename each of the files. 

Let's first look at a simple ``for`` loop, to understand how it works. 

::

  for animal in rat cat dog man
  do 
    echo $animal
  done

We just wrote a list of animals, each animal's name separated by a space
and printed each name on a separate line. The variable ``animal`` is a
dummy or a loop variable. It can then be used to refer to the element of
the list that is currently being dealt with. We could, obviously, use
something as lame as ``i`` in place of ``animal``.

To generate a range of numbers and iterate over them, we do the following. 

::

    for i in {5..10}
    do
    echo $i
    done

Now, we use a ``for`` loop to list the files that we are interested in.

::

  for i in `ls *.mp3`
  do
    echo "$i"
  done

If the file-names contain spaces, ``for`` assumes each space separated word
to be a single item in the list and prints it in a separate line. We could
change the script slightly to overcome this problem.

::

  for i in *.mp3
  do
    echo "$i"
  done

Now, we have each file name printed on a separate line. The file names are
in the form ``dd - Name.mp3`` and it has to be changed to the format
``Name.mp3``. Also, if the name has spaces, we wish to replace it with
hyphens. 

::

  for i in *.mp3
  do 
    echo $f|tr -s " " "-"|cut -d - -f 2-
  done

Now we just replace the echo command with a ``mv``  command. 

::

  for i in *.mp3
  do 
    mv $i `echo $f|tr -s " " "-"|cut -d - -f 2-`
  done

``while``
---------

The ``while`` command allows us to continuously execute a block of commands
until the command that is controlling the loop is executing successfully.

Let's start with the lamest example of a while loop.

::

  while true
  do
    echo "True"
  done

This, as you can see, is an infinite loop that prints the ``True``. 

Say we wish to write a simple program that takes user input and prints it
back, until the input is ``quit``, which quits the program. 

::

  while [ "$variable" != "quit" ]
  do
    read variable
    echo "Input - $variable"
  done
  exit 0

Environment Variables
---------------------

Environment variables are way of passing information from the shell to the
programs that are run in it. Programs are often made to look "in the
environment" for particular variables and behave differently based on what
their values are. 

Standard UNIX variables are split into two categories, environment
variables and shell variables. In broad terms, shell variables apply only
to the current instance of the shell and are used to set short-term working
conditions; environment variables have a farther reaching significance, and
those set at login are valid for the duration of the session. By
convention, environment variables have UPPER CASE and shell variables have
lower case names.

Here are a few examples of environment variables, 

::

   $ echo $OSTYPE 
   linux-gnu
   $ echo $HOME
   /home/user 

To see all the variables and their values, we could use any of the
following,  

::

   $ printenv | less
   $ env

We have looked at the PATH variable, in the previous section. We shall now
use the ``export`` command to change it's value.  

::

   $ export PATH=$PATH:$HOME/bin

See the difference in value of PATH variable before and after modifying it.

``export`` command is used to export a variable to the environment of all
the processes that are started from that shell. 

Miscellaneous Tools
===================

Finally, here are a bunch of tools, that will prove to be handy in your day
to day work. These tools will help you quickly perform tasks like searching
for files, comparing files and checking if they are the same, viewing the
exact differences between them. 

find
----

The ``find`` command lets you find files in a directory hierarchy. It
offers a very complex feature set allowing you to search for files with a
wide range of restrictions. We shall only look at some of the most
frequently used ones. You should look at the man page, for more. 

To find all files, which end with an extension, ``.pdf``, in the current
folder and all it's subfolders, 

::

    $ find . -name "*.pdf"

To list all the directory and sub-directory names, 

::

    $ find . -type d 

``find`` allows you to set limits on file-size, modification time and whole
lot of other things. 

``cmp``
-------

To compare two files, whether they are identical or not, we can use the
``cmp`` command. Let us consider some situation, we run ``find`` to locate
some file, and it turns out that we have a file with same name in different
location. 

If we are unsure, whether both the files are the same, we can use the
``cmp`` command to check if the files are identical. 

::

   $ find . -name quick.c
   ./Desktop/programs/quick.c
   ./c-folder/quick.c
   $ cmp Desktop/programs/quick.c c-folder/quick.c

If the cmp command doesn't return any output, it means that both files are
exactly identical. If there are any differences in the file, it gives you
the exact byte location at which the first difference occurred. 

Here is the output, after we made a small change to one of the files.

::

   $ cmp Desktop/programs/quick.c c-folder/quick.c
   Desktop/programs/quick.c c-folder/quick.c differ: byte 339, line 24
 

``diff``
--------

Now, we may not be happy with just the knowledge that the files are
different. We may want to see the exact differences between the files.
The ``diff`` command can be used to find the exact differences between the
files. 

::

   $ diff Desktop/programs/quick.c c-folder/quick.c

We get back a line by line difference between the two files. The ``>`` mark
indicates the content that has been added to the second file, and was not
present in the first file. The ``<`` mark indicates the lines that were
present in the first file, but are not existent in the second file. 

``tar``
-------

You would often come across (archive) files which are called *tarballs*. A
tar ball is essentially a collection of files, which may or may not be
compressed. Essentially, it eases the job of storing, backing up and
transporting multiple files, at once. 

Extracting an archive
~~~~~~~~~~~~~~~~~~~~~

The following command extracts the contents of the ``allfiles.tar`` tarball
to the directory extract. 

::

   $ mkdir extract
   $ cp allfiles.tar extract/
   $ cd extract
   $ tar -xvf allfiles.tar 

The option, ``x`` tells ``tar`` to extract the files in the archive file
specified by the ``f`` option. The ``v`` option tells ``tar`` to give out a
verbose output. 

Creating an archive
~~~~~~~~~~~~~~~~~~~

Similarly, if we wish to create a ``tar`` archive, we use the ``c`` option
instead of the ``x`` option. For instance, the command below creates an
archive from all the files with the ``.txt`` extension. 

::

    $ tar -cvf newarchive.tar *.txt


Compressed archives
~~~~~~~~~~~~~~~~~~~

You can also create and extract compressed archives using ``tar``. It
supports a wide variety of compressions like gzip, bzip2, lzma, etc. 

We need to add an additional option to ``tar`` to handle these
compressions. 


+-------------+------------+
| Compression | Option     |
+-------------+------------+
| gzip        | ``-z``     |
| bzip2       | ``-j``     |
| lzma        | ``--lzma`` |
+-------------+------------+


So, if we wished to create a gzip archive in the previous command, we
change it to the following

::

    $ tar -cvzf newarchive.tar.gz *.txt

Customizing your shell
----------------------

What would you do, if you want bash to execute a particular command each
time you start it up? For instance, say you want the current directory to
be your Desktop instead of your home folder, each time bash starts up. How
would you achieve this? Bash reads and executes commands in a whole bunch
of files called start-up files, when it starts up.

When bash starts up as an interactive login shell, it reads the files
``/etc/profile``, ``~/.bash_profile``, ``~/.bash_login``, and
``~/.profile`` in that order.

When it is a shell that is not a login shell, ``~/.bashrc`` is read and the
commands in it are executed. This can be prevented using the ``--norc``
option. To force bash to use another file, instead of the ``~/.bashrc``
file on start-up, the ``--rcfile`` option may be used.

Now, you know what you should do, to change the current directory to you
Desktop. Just put a ``cd ~/Desktop`` into your ``~/.bashrc`` and you are
set!

This example is quite a simple and lame one. The start-up files are used
for a lot more complex things than this. You could set (or unset) aliases
and a whole bunch of environment variables in the ``.bashrc``, like
changing environment variables etc. 

.. 
   Local Variables:
   mode: rst
   indent-tabs-mode: nil
   sentence-end-double-space: nil
   fill-column: 75
   End: