diff options
Diffstat (limited to 'statistics/script.rst')
| -rw-r--r-- | statistics/script.rst | 320 |
1 files changed, 245 insertions, 75 deletions
diff --git a/statistics/script.rst b/statistics/script.rst index cd3dfe9..87f7ba5 100644 --- a/statistics/script.rst +++ b/statistics/script.rst @@ -21,55 +21,83 @@ External Reviewer : Checklist OK? : <put date stamp here, if OK> [2010-10-05] -.. #[punch; add slides, exercises!] +======= +Script +======= -Hello friends and welcome to the tutorial on Statistics using Python +.. L1 -{{{ Show the slide containing title }}} +{{{ Show the first slide containing title, name of the production +team along with the logo of MHRD }}} -{{{ Show the slide containing the outline slide }}} +.. R1 -In this tutorial, we shall learn - * Doing statistical operations in Python - * Summing set of numbers - * Finding there mean - * Finding there Median - * Finding there Standard Deviation +Hello friends and welcome to the tutorial on 'Statistics' using Python. + +.. L2 + +{{{ Show the slide containing the objectives }}} + +.. R2 + +At the end of this tutorial,you will be able to, + 1. Do statistical operations in Python + #. Sum a set of numbers + #. Find their mean,median and standard deviation + +.. L3 +{{{ Show slide with pre-requisite }}} -.. #[punch: since loadtxt is anyway a pre-req, I would recommend you -.. to use a data file and load data from that. that is good, since you -.. would get to deal with arrays, instead of lists. +.. R3 -.. Talking of rows and columns of 2-D lists etc is confusing. Also, -.. converting to float can be avoided. The tutorial will feel more -.. natural, is what I think. +Before beginning this tutorial,we would suggest you to complete the +tutorial on + "Loading Data from files" + "Getting started with Lists" + "Accessing Pieces of Arrays". -.. The idea of separating the main problem and giving toy examples -.. doesn't sound good. Use the same problem to explain stuff. Or use a -.. smaller data-set or something. Using lists doesn't seem natural.] +.. L4 + +Let us invoke our ipython interpreter with pylab loaded. + +.. R4 +:: + + ipython -pylab + +.. L5 + +{{{ Open the file sslc2.txt and show }}} +.. R5 -For this tutorial We will use data file that is at the a path +For this tutorial, we will use data file that is at the path ``/home/fossee/sslc2.txt``. It contains record of students and their performance in one of the State Secondary Board Examination. It has 180,000 lines of record. We are going to read it and process this data. We can see the content of file by double clicking on it. It might take some time to open since it is quite a large file. Please -don't edit the data. This file has a particular structure. +don't edit the data since it has a particular structure. -We can do :: - - cat /home/fossee/sslc2.txt +.. R6 + +To check the contents of the file, we use the cat command. -to check the contents of the file. +.. L6 +:: + + cat /home/fossee/sslc2.txt +.. L7 {{{ Show the data structure on a slide }}} +.. R7 + Each line in the file is a set of 11 fields separated -by semi-colons Consider a sample line from this file. +by semi-colons. Consider a sample line from this file. A;015163;JOSEPH RAJ S;083;042;47;00;72;244;;; The following are the fields in any given line. @@ -77,118 +105,260 @@ The following are the fields in any given line. * Roll Number 015163 * Name JOSEPH RAJ S * Marks of 5 subjects: ** English 083 ** Hindi 042 ** Maths 47 ** -Science 35 ** Social 72 + Science 35 ** Social 72 * Total marks 244 +.. R8 -Lets try and load this data as an array and then run various function on +Lets load this data as an array and then run various functions on it. -To get the data as an array we do. :: +To get the data as an array, we use the loadtxt command + +.. L8 +:: - L=loadtxt('/home/amit/sslc2.txt',usecols=(3,4,5,6,7,),delimiter=';') - L - + L=loadtxt('/home/fossee/sslc2.txt',usecols=(3,4,5,6,7,),delimiter=';') + L +.. R9 + +We get our output in the form of an array. loadtxt function loads data from an external file.Delimiter specifies -the kind of character are the fields of data seperated by. usecols -specifies the columns to be used so (3,4,5,6,7) loads those +the kind of character, that the fields of data seperated by. usecols +specifies the columns to be used. So (3,4,5,6,7) loads those colums. The 'comma' is added because usecols is a sequence. -As we can see L is an array. We can get the shape of this array using:: +As we can see L is an array. We can get the shape of this array using + +.. L9 +:: - L.shape - (185667, 5) + L.shape -Lets start applying statistics operations on these. We will start with +.. R10 + +We get a tuple stating the numbers of rows and columns respectively. +Lets start applying statistical operations on these. We will start with the most basic, summing. How do you find the sum of marks of all subjects for the first student. -As we know from our knowledge of accessing pieces of arrays. To acess -the first row we will do :: +.. L10 + +.. R11 + +As we know from our knowledge of accessing pieces of arrays, to acess +the first row, we will do + +.. L11 +:: - L[0,:] + L[0,:] + +.. R12 -Now to sum this we can say :: +Now to sum this we can say + +.. L12 +:: totalmarks=sum(L[0,:]) totalmarks -To get the mean we can do :: +.. R13 + +Now to get the mean we can divide the totalmarks by the length + +.. L13 +:: + + totalmarks/len(L[0,:]) + +.. R14 + +or simply use the function ``mean``. - totalmarks/len(L[0,:]) +.. L14 +:: -or simply :: + mean(L[0,:]) - mean(L[0,:]) +.. R15 -But we have such a large data set calculating one by one the mean of -each student is impossible. Is there a way to reduce the work. +But we have such a large data set and calculating the mean for each student one by one +is impossible. Is there a way to reduce the work. -For this we will look into the documentation of mean by doing:: +For this we will look into the documentation of mean + +.. L15 +:: mean? +.. R16 + As we know L is a two dimensional array. We can calculate the mean across each of the axis of the array. The axis of rows is referred by -number 0 and columns by 1. So to calculate mean accross all colums we -will pass extra parameter 1 for the axis.:: +number 0 and columns by 1. So to calculate mean accross all columns, we +will pass extra parameter 1 for the axis. + +.. L16 +:: mean(L,1) -L here is the two dimensional array. +.. R17 + +L here, is a two dimensional array. Similarly to calculate average marks scored by all the students for each -subject can be calculated using :: +subject can be calculated using + +.. L17 +:: - mean(L,0) + mean(L,0) -Next lets now calculate the median of English marks for the all the students -We can access English marks of all students using :: +.. R18 - L[:,0] +Next, let us calculate the median of English marks for the all the students. +We can access English marks of all students using + +.. L18 +:: + + L[:,0] + +.. R19 -To get the median we will do :: +To get the median we will simply use the function ``median``. + +.. L19 +:: - median(L[:,0]) + median(L[:,0]) + +.. R20 For all the subjects we can use the same syntax as mean and calculate -median across all rows using :: +median across all rows using median + +.. L20 +:: + + median(L,0) + +.. R21 + +Similarly to calculate standard deviation for English we will use the function ``std`` - median(L,0) +.. L21 +:: + std(L[:,0]) -Similarly to calculate standard deviation for English we can do:: +.. R22 - std(L[:,0]) +and for all rows, we do, -and for all rows:: +.. L22 +:: std(L,0) -Following is an exercise that you must do. +.. R23 + +Pause the video here, try out the following exercise and resume the video. + +.. L23 + +.. L24 + +{{{ Show slide with exercise 1 }}} + +.. R24 -%% %% In the given file football.txt at path /home/fossee/football.txt , one column is player name,second is goals at home and third goals away. +In the given file football.txt at path /home/fossee/football.txt , one column is player name,second is goals at home and third goals away. 1.Find the total goals for each player - 2.Mean home and away goals + 2.Mean of home and away goals 3.Standard deviation of home and away goals +.. L25 + +{{{ Open the file football.txt and keep open for some time }}} + +.. R25 + +This is the required data. + +.. L26 + +{{{ Switch to slide Solution 1 }}} + +.. R26 + +The solution is on your screen. + +.. L27 + {{{ Show summary slide }}} +.. R27 + This brings us to the end of the tutorial. -we have learnt +In this tutorial,we have learnt to, + + 1. Do the standard statistical operations sum , mean + median and standard deviation in Python. + #. Combine text loading and the statistical operation to solve + real world problems. + +.. L28 + +{{{Show self assessment questions slide}}} + +.. R28 + +Here are some self assessment questionss for you to solve + +1. Given a two dimensional list, + two_dimensional_list=[[3,5,8,2,1],[4,3,6,2,1]] + how do we calculate the mean of each row? + + +2. Calcutate the median of the given list? + student_marks=[74,78,56,87,91,82] + + +3. Suppose there is a file with 6 columns but we wish to load text + only in columns 2,3,4,5. How do we specify that? + +.. L29 + +{{{solution of self assessment questions on slide}}} + +.. R29 + +And the answers, + +1. To get the mean of each row, we just pass 1 as the second parameter to the function ``mean``. +:: + + mean(two_dimensional_list,1) + +2. We use the function ``median`` to calculate the median of the list +:: - * How to do the standard statistical operations sum , mean - median and standard deviation in Python. - * Combine text loading and the statistical operation to solve - real world problems. + median(student_marks) -{{{ Show the "sponsored by FOSSEE" slide }}} +3. To specify the particular columns of a file, we use the parameter usecols=(2,3,4,5) +.. L30 -This tutorial was created as a part of FOSSEE project, NME ICT, MHRD India +{{{ Show the Thank you slide }}} -Hope you have enjoyed and found it useful. +.. R30 +Hope you have enjoyed this tutorial and found it useful. Thank you! |