Merge pull request #379 from ankitjavalkar/better-installer-single-mount

Installation and Quickstart script using PyInvoke

11 files changed, 160 insertions, 55 deletions

diff --git a/CHANGELOG.txt b/CHANGELOG.txt
index 56c8910..a227164 100644
--- a/CHANGELOG.txt
+++ b/CHANGELOG.txt
@@ -16,6 +16,7 @@
 * CSV download for quiz attempts enhanced.
 * Updated Courses Page to show Active Courses on top.
 * Added a Datetime Picker to Add Course Page
+* Added invoke script for quickstart and docker deployment
 
 === 0.6.0 (11-05-2017) ===
 
diff --git a/Dockerfile b/Dockerfile
index 72a74f4..6a7d894 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -1,13 +1,11 @@
-FROM debian:8.2
+FROM ubuntu:16.04
 MAINTAINER FOSSEE <pythonsupport@fossee.in>
 
 # Update Packages and Install Python & net-tools
-RUN apt-get update && apt-get install -y python net-tools python-pip && pip install tornado
-
-# Copy the project folder from host into container
-COPY ./yaksh /src/yaksh
-
-WORKDIR /src
-
-# Run Yaksh code server
-CMD ["python", "-m", "yaksh.code_server"]
+RUN apt-get update && \
+apt-get install -y  software-properties-common && \
+echo oracle-java8-installer shared/accepted-oracle-license-v1-1 select true | debconf-set-selections && \
+add-apt-repository -y ppa:webupd8team/java && \
+apt-get update && \
+apt-get install -y oracle-java8-installer && \
+apt-get install -y sudo python net-tools git python3-pip vim libmysqlclient-dev scilab build-essential python3-numpy python3-scipy ipython3 ipython3-notebook python3-pandas python3-nose
diff --git a/README.md b/README.md
index caae485..a0faab2 100644
--- a/README.md
+++ b/README.md
@@ -37,56 +37,52 @@ Quick Start
 #### Installation
 
 1. Install yaksh
-    - For latest stable release
+    - Clone the repository
 
-            $ pip install yaksh
+            $ git clone https://github.com/FOSSEE/online_test.git
 
-    - For the development version
+    - Go to the online_test directory
 
-            $ pip install git+https://github.com/FOSSEE/online_test.git
+            $ cd ./online_test
 
-#### Short instructions
+    - Install the dependencies
+
+        - For Python 2 use:
+
+            $ pip install -r ./requirements/requirements-py2.txt
 
-To see a quick demo after installing yaksh do the following:
+        - For Python 3 (recommended) use:
 
-    $ yaksh create_demo yaksh_demo
-    $ yaksh run yaksh_demo
+            $ pip install -r ./requirements/requirements-py3.txt
+
+#### Short instructions
 
-On another terminal start up the code server that executes the user code safely:
+1. To run the application do the following:
 
-    $ sudo yaksh run_code_server
+        $ invoke serve
 
-Now point your browser to ```http://localhost:8000/exam```.
+    - *Note:* The serve command will run the django application server on the 8000 port
+    and hence this port will be unavailable to other processes.
 
-#### More detailed instructions
+1.  On another terminal start up the code server that executes the user code safely:
 
-1. On the terminal run:
+    - To run the code server in a sandboxed docker environment, run the command:
 
-        $ yaksh create_demo [project_path]
+            $ invoke start
 
-    - `project_path` is the desired directory of the django project the
-      basename of which is also the Django project name. This can be a
-      relative directory.
+    - Make sure that you have Docker installed on your system beforehand.
+    [Docker Installation](https://docs.docker.com/engine/installation/#desktop)
 
-    - In case a `project_path` is not specified, the project is created
-      in a `yaksh_demo` subdirectory of the current directory.
 
-1. The script does the following;
-    1. Creates a new django project with name as the basename of the specified
-       `project_path`
-    1. Creates a new demo database.
-    1. Creates two users, teacher and student.
-    1. Loads demo questions.
-    1. Loads demo quiz.
+    - To run the code server without docker, locally use:
 
-1. To run the server, run:
+            $ invoke start --unsafe
 
-        $ yaksh run relpath/or/abspath/to/demo
+    - Note this command will run the yaksh code server locally on your machine
+      and is susceptible to malicious code. You will have to install the code 
+      server requirements in sudo mode.
 
-1. In a new terminal run the following command which executes user submitted
-   code safely:
 
-        $ sudo yaksh run_code_server
 
 1. Open your browser and open the URL ```http://localhost:8000/exam```
 
diff --git a/README_production.md b/README_production.md
index ed19523..8b79785 100644
--- a/README_production.md
+++ b/README_production.md
@@ -167,19 +167,18 @@ To install this app follow the steps below:
 
 #### Using Dockerized Code Server
 
- 1. Install [Docker] (https://github.com/FOSSEE/online_test/blob/master/README.md)
+ 1. Install [Docker](https://github.com/FOSSEE/online_test/blob/master/README.md)
 
  1. Got to the directory where the project is located
         cd /path/to/online_test
 
  1. Create a docker image. This may take a few minutes
-        docker build -t yaksha:v1 .
+        docker build -t yaksh_code_server .
 
  1. Check if the image has been created using the output of,
         docker images
 
- 1. Run a container using the newly created image.
-        sudo docker run -d -p 53579:53579 -p 8001:8001 yaksha:v1
+ 1. Run the invoke script using the command ```invoke start```
     The command will create and run a new docker container (that is running the code_server.py within it), it will also bind the ports of the host with those of the container
 
 #### Additional commands available
diff --git a/online_test/settings.py b/online_test/settings.py
index 90cce9d..790083e 100644
--- a/online_test/settings.py
+++ b/online_test/settings.py
@@ -15,7 +15,7 @@ BASE_DIR = os.path.dirname(os.path.dirname(__file__))
 # The directory where user data can be saved.  This directory will be
 # world-writable and all user code will be written and saved here by the
 # code server with each user having their own sub-directory.
-OUTPUT_DIR = os.path.join(BASE_DIR, 'output')
+OUTPUT_DIR = os.path.join(BASE_DIR, "yaksh_data", "output")
 
 # Quick-start development settings - unsuitable for production
 # See https://docs.djangoproject.com/en/1.6/howto/deployment/checklist/
@@ -33,8 +33,6 @@ URL_ROOT = ''
 
 # Application definition
 
-FIXTURE_DIRS = os.path.join(BASE_DIR, "yaksh", "fixtures")
-
 INSTALLED_APPS = (
     'django.contrib.admin',
     'django.contrib.auth',
@@ -100,7 +98,7 @@ SOCIAL_AUTH_LOGIN_ERROR_URL = '/exam/login/'
 
 MEDIA_URL = "/data/"
 
-MEDIA_ROOT = os.path.join(BASE_DIR, "yaksh", "data")
+MEDIA_ROOT = os.path.join(BASE_DIR, "yaksh_data", "data")
 
 # Set this varable to <True> if smtp-server is not allowing to send email.
 EMAIL_USE_TLS = False
diff --git a/requirements/requirements-codeserver.txt b/requirements/requirements-codeserver.txt
new file mode 100644
index 0000000..a4f419c
--- /dev/null
+++ b/requirements/requirements-codeserver.txt
@@ -0,0 +1,5 @@
+pytest
+six
+requests
+tornado
+psutil
diff --git a/requirements/requirements-common.txt b/requirements/requirements-common.txt
index 100d693..b4d2e5b 100644
--- a/requirements/requirements-common.txt
+++ b/requirements/requirements-common.txt
@@ -1,9 +1,9 @@
+-r requirements-codeserver.txt
+invoke==0.21.0
 django==1.9.5
 django-taggit==0.18.1
 pytz==2016.4
 python-social-auth==0.2.19
-tornado
 selenium==2.53.6
 coverage
-psutil
-ruamel.yaml==0.15.23
+ruamel.yaml==0.15.23
\ No newline at end of file
diff --git a/tasks.py b/tasks.py
new file mode 100644
index 0000000..eabf8fb
--- /dev/null
+++ b/tasks.py
@@ -0,0 +1,95 @@
+import invoke
+from invoke import task
+import os
+from yaksh.settings import SERVER_POOL_PORT
+
+SCRIPT_DIR = os.path.abspath(os.path.dirname(__file__))
+TARGET_CONTAINER_NAME = 'yaksh_code_server'
+SRC_IMAGE_NAME = 'fossee/yaksh_codeserver'
+
+def create_dir(path):
+    if not os.path.exists(path):
+        os.makedirs(path)
+
+@task
+def setupdb(ctx):
+    print("** Setting up & migrating database **")
+    ctx.run("python manage.py makemigrations")
+    ctx.run("python manage.py migrate")
+    ctx.run("python manage.py loaddata demo_fixtures.json")
+
+@task(setupdb)
+def serve(ctx):
+    print("** Running the Django web server. Press Ctrl-C to Exit **")
+    ctx.run("python manage.py runserver")
+
+@task
+def clean(ctx):
+    print("** Discarding database **")
+    ctx.run("rm -rf {0}".format(os.path.join(SCRIPT_DIR, 'db.sqlite3')))
+
+@task
+def getimage(ctx, image=SRC_IMAGE_NAME):
+    try:
+        result = ctx.run("sudo docker inspect {0}".format(image), hide=True)
+    except invoke.exceptions.Failure:
+        print("The docker image {0} does not exist locally".format(image))
+        print("\n** Pulling latest image <{0}> from docker hub **".format(image))
+        ctx.run("sudo docker pull {0}".format(image))
+
+@task
+def start(ctx, ports=SERVER_POOL_PORT, image=SRC_IMAGE_NAME, unsafe=False,
+    version=3):
+    if unsafe:
+        with ctx.cd(SCRIPT_DIR):
+            print("** Initializing local code server **")
+            ctx.run("sudo python{0} -m yaksh.code_server".format(
+                    version
+                )
+            )
+    else:
+        cmd_params = {'ports': ports,
+            'image': SRC_IMAGE_NAME,
+            'name': TARGET_CONTAINER_NAME,
+            'vol_mount': os.path.join(SCRIPT_DIR, 'yaksh_data'),
+            'command': 'sh {0}'.format(
+                os.path.join(SCRIPT_DIR,
+                'yaksh_data', 'yaksh', 'scripts', 'yaksh_script.sh')
+            )
+        }
+
+        getimage(ctx, image=SRC_IMAGE_NAME)
+
+        print("** Preparing code server **")
+        create_dir(os.path.join(SCRIPT_DIR, 'yaksh_data', 'data'))
+        create_dir(os.path.join(SCRIPT_DIR, 'yaksh_data', 'output'))
+
+        ctx.run('cp -r {0} {1}'.format(
+                os.path.join(SCRIPT_DIR, 'yaksh'),
+                os.path.join(SCRIPT_DIR, 'yaksh_data')
+            )
+        )
+        ctx.run('cp {0} {1}'.format(
+                os.path.join(SCRIPT_DIR, 'requirements', 'requirements-codeserver.txt'),
+                os.path.join(SCRIPT_DIR, 'yaksh_data')
+            )
+        )
+
+        print("** Initializing code server within docker container **")
+        ctx.run(
+            "sudo docker run \
+            -dp {ports}:{ports} --name={name} \
+            -v {vol_mount}:{vol_mount} \
+            -w {vol_mount} \
+            {image} {command}".format(**cmd_params)
+        )
+
+@task
+def stop(ctx, container=TARGET_CONTAINER_NAME, hide=True):
+    result = ctx.run("sudo docker ps -q --filter='name={0}'".format(container))
+    if result.stdout:
+        print ("** Discarding the docker container <{0}>".format(container))
+        ctx.run("sudo docker stop {0}".format(container))
+        ctx.run("sudo docker rm {0}".format(container))
+    else:
+        print("** Docker container <{0}> not found **".format(container))
diff --git a/yaksh/models.py b/yaksh/models.py
index cb9b481..d698232 100644
--- a/yaksh/models.py
+++ b/yaksh/models.py
@@ -82,6 +82,8 @@ test_status = (
                 ('completed', 'Completed'),
               )
 
+FIXTURES_DIR_PATH = os.path.join(settings.BASE_DIR, 'yaksh', 'fixtures')
+
 
 def get_assignment_dir(instance, filename):
     upload_dir = instance.question_paper.quiz.description.replace(" ", "_")
@@ -544,7 +546,7 @@ class Question(models.Model):
 
     def create_demo_questions(self, user):
         zip_file_path = os.path.join(
-            settings.FIXTURE_DIRS, 'demo_questions.zip'
+            FIXTURES_DIR_PATH, 'demo_questions.zip'
         )
         files, extract_path = extract_files(zip_file_path)
         self.read_yaml(extract_path, user, files)
diff --git a/yaksh/scripts/yaksh_script.sh b/yaksh/scripts/yaksh_script.sh
new file mode 100644
index 0000000..f39153e
--- /dev/null
+++ b/yaksh/scripts/yaksh_script.sh
@@ -0,0 +1,11 @@
+#!/bin/bash
+# Basic script to install pip packages and run the yaksh code server command
+
+chown -R nobody output
+chmod -R a+rwX output
+chmod -R a+rX data yaksh
+chmod -R o-w data yaksh
+echo "** Installing python dependencies **"
+pip3 install -r ./requirements-codeserver.txt
+echo "** Running code server **"
+/usr/bin/sudo -su nobody python3 -m yaksh.code_server
diff --git a/yaksh/test_views.py b/yaksh/test_views.py
index e3b0168..652f44c 100644
--- a/yaksh/test_views.py
+++ b/yaksh/test_views.py
@@ -23,7 +23,7 @@ from django.core.files.uploadedfile import SimpleUploadedFile
 from yaksh.models import User, Profile, Question, Quiz, QuestionPaper,\
     QuestionSet, AnswerPaper, Answer, Course, StandardTestCase,\
     AssignmentUpload, FileUpload, McqTestCase, IntegerTestCase, StringTestCase,\
-    FloatTestCase
+    FloatTestCase, FIXTURES_DIR_PATH
 from yaksh.decorators import user_has_profile
 
 
@@ -3201,7 +3201,7 @@ class TestShowQuestions(TestCase):
             username=self.user.username,
             password=self.user_plaintext_pass
         )
-        ques_file = os.path.join(settings.FIXTURE_DIRS, "demo_questions.zip")
+        ques_file = os.path.join(FIXTURES_DIR_PATH, "demo_questions.zip")
         f = open(ques_file, 'rb')
         questions_file = SimpleUploadedFile(ques_file, f.read(), 
                                             content_type="application/zip")