{ "cells": [ { "cell_type": "markdown", "source": [ "# Task : Indian Sign Language Recognition(gesture recognition) and Conversion to speech" ], "metadata": { "id": "bXCEiw_6Q5Y-" }, "id": "bXCEiw_6Q5Y-" }, { "cell_type": "markdown", "source": [ "## Objective:\n", "Develop instructional materials for students to learn machine learning using\n", "Jupyter Notebook and other ML tools. The aim is to accurately recognize\n", "ISL(Indian Sign Language Recognition) gestures, convert them into\n", "corresponding text, and then convert the text into speech." ], "metadata": { "id": "agQMLqQ8RfXX" }, "id": "agQMLqQ8RfXX" }, { "cell_type": "markdown", "source": [ "## Tools and Technologies:\n", " * **Jupyter Notebook:** For data preprocessing, model development, and\n", "training.\n", "\n", " * **Programming Language:** Python - Widely used for its simplicity,\n", "extensive libraries, and ecosystem support." ], "metadata": { "id": "o_cBLhWfRqXs" }, "id": "o_cBLhWfRqXs" }, { "cell_type": "markdown", "source": [ "## Starting the Exciting Journey for this project by following these" ], "metadata": { "id": "syrhOm6nMUPo" }, "id": "syrhOm6nMUPo" }, { "cell_type": "markdown", "source": [ "To make this interesting project we will be following some steps and sub-steps so that our project can be made without any errors. This is the most useful project which we are going to make, which deals with the real life problem of deaf and dumb people.\n", "\n", "So let us start this exciting journey with the following steps:\n", "* **Step1:** Making our system ready for this project, for that you need to download some libraries of python using `pip` command.\n", " * OpenCV ----> `pip install opencv-python` ----> open-source library for computer vision, machine learning, and image processing.\n", " * PIL ----> `pip install pillow` ----> Pillow provides the ImageDraw module that provides simple 2D graphics for Image objects.\n", " * tensorflow ----> `pip install tensorflow` ----> used to build and train deep learning models as it facilitates the creation of computational graphs and efficient execution on various hardware platforms\n", " * sklearn ----> `pip install scikit-learn` used here for spliting data into test and train.\n", " * pyttsx3 ----> `pip install pyttsx3` used to convert text to speech.\n", "\n", " Do not worry you are not supposed to copy these commends and install one by one in your terminal because you will see these commands installed at the time of there use in this notebook file only, so chill....\n", "* **Step2:** Making Hardware ready for this project,for that we will be following some sub steps which are as followed:\n", " * Getting the Arduino code\n", " * Interfacing the Hardware\n", " * Installing the code to the Hardware\n", " * Getting the esp32_url for this project\n", "* **Step3:** Now after sucessfully installing these libraries of python, you need to import these libraries for this project.\n", "* **Step4:** Data collection and processing, this very steps have some sub steps which is needed to be followed:\n", " * **Data Collection:** Capture image form esp32-cam and save them with there class name(in our project it is A to Z gestures).\n", " * **Data storing:** Now after creating your data with there class name you need to store these data into csv file, in our case we are taking the image data of any pixel, resizing it to 40x40, spliting the data into test and train and after this we will be storing these data in to csv files named as `train.csv` and `test.csv`.\n", " * **Data Loading:** We will be loading these csv files in our code.\n", " * **Data Processing:** Now its time to process the data for this project which will be helpful for us to develop the model and test it for the accuracy.\n", " * **Debugging:** Debugging the data is most essential before model development as it debugs our process and alearts us when there is any error.\n", " * **Previewing Data:** After some more data processing now we will be previewing the data in grayscale as this gives us the correct image data for developing our model.\n", "* **Step5:** Developing our ML model.\n", "* **step6:** Deploying this ML model using esp32-cam, for this we will be following some sub steps which are as follows:\n", " * some necessary system commands\n", " * Import some libraries\n", " * Load ML model\n", " * Connect to esp32-cam\n", " * Prediction\n", " * Converting the predicted value to text\n", " * Converting the text to speech\n", "\n", "and yes if you follow these steps, you will be getting your project ready." ], "metadata": { "id": "kVrsKk_fSBF1" }, "id": "kVrsKk_fSBF1" }, { "cell_type": "markdown", "source": [ "## **Step 1:** Getting system ready" ], "metadata": { "id": "hmTvNRyA2s2L" }, "id": "hmTvNRyA2s2L" }, { "cell_type": "code", "source": [ "!pip install opencv-python" ], "metadata": { "id": "6F6xHoAs23yD", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "d3d42533-e033-4d60-bda6-fd1dba7a1c44" }, "id": "6F6xHoAs23yD", "execution_count": 1, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Requirement already satisfied: opencv-python in /usr/local/lib/python3.10/dist-packages (4.8.0.76)\n", "Requirement already satisfied: numpy>=1.21.2 in /usr/local/lib/python3.10/dist-packages (from opencv-python) (1.25.2)\n" ] } ] }, { "cell_type": "code", "source": [ "!pip install pillow" ], "metadata": { "id": "KrVZnHOS2-w5", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "5e70512e-4c60-4276-cff5-60688216b155" }, "id": "KrVZnHOS2-w5", "execution_count": 2, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Requirement already satisfied: pillow in /usr/local/lib/python3.10/dist-packages (9.4.0)\n" ] } ] }, { "cell_type": "code", "source": [ "!pip install tensorflow" ], "metadata": { "id": "KKA6DNlI2-tW", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "ca87e82b-3a93-4fa0-f65e-86fc1171471f" }, "id": "KKA6DNlI2-tW", "execution_count": 3, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Requirement already satisfied: tensorflow in /usr/local/lib/python3.10/dist-packages (2.15.0)\n", "Requirement already satisfied: absl-py>=1.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.4.0)\n", "Requirement already satisfied: astunparse>=1.6.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.6.3)\n", "Requirement already satisfied: flatbuffers>=23.5.26 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (24.3.25)\n", "Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.6.0)\n", "Requirement already satisfied: google-pasta>=0.1.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.2.0)\n", "Requirement already satisfied: h5py>=2.9.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.9.0)\n", "Requirement already satisfied: libclang>=13.0.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (18.1.1)\n", "Requirement already satisfied: ml-dtypes~=0.2.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.2.0)\n", "Requirement already satisfied: numpy<2.0.0,>=1.23.5 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.25.2)\n", "Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.3.0)\n", "Requirement already satisfied: packaging in /usr/local/lib/python3.10/dist-packages (from tensorflow) (24.1)\n", "Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.20.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (3.20.3)\n", "Requirement already satisfied: setuptools in /usr/local/lib/python3.10/dist-packages (from tensorflow) (67.7.2)\n", "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.16.0)\n", "Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.4.0)\n", "Requirement already satisfied: typing-extensions>=3.6.6 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (4.12.2)\n", "Requirement already satisfied: wrapt<1.15,>=1.11.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.14.1)\n", "Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (0.37.1)\n", "Requirement already satisfied: grpcio<2.0,>=1.24.3 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (1.64.1)\n", "Requirement already satisfied: tensorboard<2.16,>=2.15 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.2)\n", "Requirement already satisfied: tensorflow-estimator<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.0)\n", "Requirement already satisfied: keras<2.16,>=2.15.0 in /usr/local/lib/python3.10/dist-packages (from tensorflow) (2.15.0)\n", "Requirement already satisfied: wheel<1.0,>=0.23.0 in /usr/local/lib/python3.10/dist-packages (from astunparse>=1.6.0->tensorflow) (0.43.0)\n", "Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (2.27.0)\n", "Requirement already satisfied: google-auth-oauthlib<2,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (1.2.1)\n", "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (3.6)\n", "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (2.31.0)\n", "Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (0.7.2)\n", "Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard<2.16,>=2.15->tensorflow) (3.0.3)\n", "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (5.4.0)\n", "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (0.4.0)\n", "Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (4.9)\n", "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow) (1.3.1)\n", "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (3.3.2)\n", "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (3.7)\n", "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (2.0.7)\n", "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests<3,>=2.21.0->tensorboard<2.16,>=2.15->tensorflow) (2024.7.4)\n", "Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard<2.16,>=2.15->tensorflow) (2.1.5)\n", "Requirement already satisfied: pyasn1<0.7.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<2.16,>=2.15->tensorflow) (0.6.0)\n", "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<2,>=0.5->tensorboard<2.16,>=2.15->tensorflow) (3.2.2)\n" ] } ] }, { "cell_type": "code", "source": [ "!pip install scikit-learn" ], "metadata": { "id": "cm0oqRn_2-n9", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "c866a67b-ad81-49f4-9b16-75dbc2a47faa" }, "id": "cm0oqRn_2-n9", "execution_count": 4, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (1.2.2)\n", "Requirement already satisfied: numpy>=1.17.3 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.25.2)\n", "Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.11.4)\n", "Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (1.4.2)\n", "Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn) (3.5.0)\n" ] } ] }, { "cell_type": "code", "source": [ "!pip install pyttsx3" ], "metadata": { "id": "KKM_5JMm2-lo", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "93dc064d-da7e-4874-9fe0-fe061953108d" }, "id": "KKM_5JMm2-lo", "execution_count": 5, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Collecting pyttsx3\n", " Downloading pyttsx3-2.90-py3-none-any.whl (39 kB)\n", "Installing collected packages: pyttsx3\n", "Successfully installed pyttsx3-2.90\n" ] } ] }, { "cell_type": "code", "source": [ "!pip install matplotlib" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "_VDtKp6FTMQL", "outputId": "1c873f6b-bcd6-477a-e432-85a1a1524a13" }, "id": "_VDtKp6FTMQL", "execution_count": 6, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Requirement already satisfied: matplotlib in /usr/local/lib/python3.10/dist-packages (3.7.1)\n", "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.2.1)\n", "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (0.12.1)\n", "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (4.53.1)\n", "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.4.5)\n", "Requirement already satisfied: numpy>=1.20 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (1.25.2)\n", "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (24.1)\n", "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (9.4.0)\n", "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (3.1.2)\n", "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib) (2.8.2)\n", "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib) (1.16.0)\n" ] } ] }, { "cell_type": "code", "source": [ "!pip install seaborn" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "3VeR5EahTMGz", "outputId": "86899791-fdb6-4d43-9ec7-d8748311eaef" }, "id": "3VeR5EahTMGz", "execution_count": 7, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Requirement already satisfied: seaborn in /usr/local/lib/python3.10/dist-packages (0.13.1)\n", "Requirement already satisfied: numpy!=1.24.0,>=1.20 in /usr/local/lib/python3.10/dist-packages (from seaborn) (1.25.2)\n", "Requirement already satisfied: pandas>=1.2 in /usr/local/lib/python3.10/dist-packages (from seaborn) (2.0.3)\n", "Requirement already satisfied: matplotlib!=3.6.1,>=3.4 in /usr/local/lib/python3.10/dist-packages (from seaborn) (3.7.1)\n", "Requirement already satisfied: contourpy>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (1.2.1)\n", "Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (0.12.1)\n", "Requirement already satisfied: fonttools>=4.22.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (4.53.1)\n", "Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (1.4.5)\n", "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (24.1)\n", "Requirement already satisfied: pillow>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (9.4.0)\n", "Requirement already satisfied: pyparsing>=2.3.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (3.1.2)\n", "Requirement already satisfied: python-dateutil>=2.7 in /usr/local/lib/python3.10/dist-packages (from matplotlib!=3.6.1,>=3.4->seaborn) (2.8.2)\n", "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.2->seaborn) (2023.4)\n", "Requirement already satisfied: tzdata>=2022.1 in /usr/local/lib/python3.10/dist-packages (from pandas>=1.2->seaborn) (2024.1)\n", "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.7->matplotlib!=3.6.1,>=3.4->seaborn) (1.16.0)\n" ] } ] }, { "cell_type": "markdown", "source": [ "## **Step 2:** Making Hardware ready\n" ], "metadata": { "id": "-y2FfoR13QpS" }, "id": "-y2FfoR13QpS" }, { "cell_type": "markdown", "source": [ "### Getting the code" ], "metadata": { "id": "Nh_opNZ97fh-" }, "id": "Nh_opNZ97fh-" }, { "cell_type": "markdown", "source": [ "Open Arduino IDE and follow these sub steps:\n", "\n", "Go to `File---> Examples---> Esp32---> Camera---> CameraWebServer`\n", "\n", "if you can not see the Esp32 in Examples then do not worry and follow this\n", "\n", "Go to `Tools---> Board:---> Boards Manager...`\n", " \n", "now find Esp32 in board manager and install it in your IDE by clicking the install button." ], "metadata": { "id": "tLsG7FOh4JKj" }, "id": "tLsG7FOh4JKj" }, { "cell_type": "markdown", "source": [ "### Interfacing the Hardware" ], "metadata": { "id": "NK03Z2SN7l8l" }, "id": "NK03Z2SN7l8l" }, { "cell_type": "markdown", "source": [ "See the beloy diagram and interface the esp32-cam as ot is (Do not forget to connect GPIO0 pin to GND).\n", "\n", "Now connect the FTDI to the computer through the usb cable\n", "* 5V ---> VCC\n", "* GND ---> GND\n", "* U0T ---> Rx\n", "* U0R ---> Tx\n", "* IO0 ---> GND\n", "\n", "pin diagram for your ease." ], "metadata": { "id": "6tC39tjo65-E" }, "id": "6tC39tjo65-E" }, { "cell_type": "markdown", "source": [ "" ], "metadata": { "id": "62QVBe0E6jdu" }, "id": "62QVBe0E6jdu" }, { "cell_type": "markdown", "source": [ "### Installing code to Hardware" ], "metadata": { "id": "mmHNA9It_w08" }, "id": "mmHNA9It_w08" }, { "cell_type": "markdown", "source": [ "Follow these steps to install code to your Hardware:\n", "* **Step 1:** Open the example code and do some necessary changes so that it can be used in our project, which are as follows:\n", " * Comment the line no 17 which is `#define CAMERA_MODEL_ESP_EYE // Has PSRAM`\n", " * uncomment the line no 25 which is `#define CAMERA_MODEL_AI_THINKER // Has PSRAM`\n", " * change the ssid and password\n", " ```\n", " const char *ssid = \"**********\";\n", " const char *password = \"**********\";\n", " ```\n", "\n", "Now Install the code on esp32cam by selecting the correct board and port to which the esp32-cam is connected. To do so\n", "\n", "Go to `Tools---> Board---> esp32---> AI Thinker ESP32-CAM`\n", "\n", "Go to `Tools---> Port---> your port`\n", "\n", "and hit the upload button (it will take some time to verify and upload the code to the esp32-cam).\n", "\n", "if it shows any error the for debugging follow these steps:\n", "* Press the reset button present on the esp32-cam module.\n", "* Check the connection for any loose fitting.\n" ], "metadata": { "id": "a7yrk0sg4SJN" }, "id": "a7yrk0sg4SJN" }, { "cell_type": "markdown", "source": [ "### Getting esp32_url for this project" ], "metadata": { "id": "2wWWs7Y1EyZX" }, "id": "2wWWs7Y1EyZX" }, { "cell_type": "markdown", "source": [ "Your Hardware is ready with the code now, only you need to get your esp32_url to get the camera access for this project. To do so you need to follow some steps which are as follows:\n", "* Open the serial monitor\n", "* Set the baud rate to `115200 baud`\n", "* Remove the jumper wire which is connected to IO0 to GND of esp32-cam module\n", "* Press the reset button present on the esp32-cam module, you will notice some url is printed on the serial monitor.\n", "* Copy this url and paste on the browser of your PC (ensure that your pc is also connected to the same hotspot to which the esp32-cam module is connected).\n", "* Now serach for the start stream button red in colour.\n", "* Click that button for starting the video stream, now video will start streaming on the web page\n", "* Now right click on the video displayed and go to open in new tab and cose the previous tab\n", "* Copy the url of the current page and thats it you got your esp32 url." ], "metadata": { "id": "MHxl9NmWjxzC" }, "id": "MHxl9NmWjxzC" }, { "cell_type": "markdown", "source": [ "## **Step 3:** Necessary Imports\n" ], "metadata": { "id": "FSNpz3TzcS0H" }, "id": "FSNpz3TzcS0H" }, { "cell_type": "code", "execution_count": 72, "id": "a305e00a-1919-4ff3-ae75-fb86a50075b7", "metadata": { "id": "a305e00a-1919-4ff3-ae75-fb86a50075b7" }, "outputs": [], "source": [ "import cv2 # openCV library for image processing\n", "import os # for some system functions\n", "import numpy as np # for data processing\n", "import pandas as pd # for data manupulation\n", "from PIL import Image # for accessing the image to our code as data\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns\n", "import tensorflow as tf # for making deeplearning model\n", "from tensorflow.keras.models import Sequential\n", "from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense, Dropout, BatchNormalization\n", "from tensorflow.keras.optimizers import Adam\n", "from tensorflow.keras.callbacks import ReduceLROnPlateau\n", "from tensorflow.keras.preprocessing.image import ImageDataGenerator\n", "from tensorflow.keras.models import load_model\n", "from sklearn.preprocessing import OneHotEncoder\n", "from sklearn.model_selection import train_test_split # for spliting the data into test and train\n", "from sklearn.metrics import classification_report,confusion_matrix\n", "import time" ] }, { "cell_type": "markdown", "source": [ "### Connecting the drive to to notebook by running this code." ], "metadata": { "id": "vVJPcHat9Bm8" }, "id": "vVJPcHat9Bm8" }, { "cell_type": "code", "source": [ "from google.colab import drive\n", "drive.mount('/content/drive')" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "WP4EFgfqvpkG", "outputId": "abc04ade-8785-45bc-afd7-9f395eb4f919" }, "id": "WP4EFgfqvpkG", "execution_count": 73, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n" ] } ] }, { "cell_type": "markdown", "source": [ "## **Step 4:** Data Collection and Processing\n" ], "metadata": { "id": "YjTEsSBJbZL6" }, "id": "YjTEsSBJbZL6" }, { "cell_type": "markdown", "source": [ "### Data Collection\n" ], "metadata": { "id": "hz9T9Is30LLK" }, "id": "hz9T9Is30LLK" }, { "cell_type": "code", "source": [ "def capture_images_from_esp32(class_name, esp32_url, capture_duration=20, save_dir='data'):\n", " # Create the directory for the class if it doesn't exist\n", " class_dir = os.path.join(save_dir, class_name)\n", " os.makedirs(class_dir, exist_ok=True)\n", "\n", " # Start capturing images from ESP32-CAM\n", " cap = cv2.VideoCapture(esp32_url)\n", " if not cap.isOpened():\n", " print(f\"Failed to open video stream at {esp32_url}\")\n", " return\n", "\n", " start_time = time.time()\n", " img_count = 0\n", "\n", " while int(time.time() - start_time) < capture_duration:\n", " ret, frame = cap.read()\n", " if not ret:\n", " print(\"Failed to capture image from ESP32-CAM\")\n", " break\n", "\n", " # Save the frame as an image file\n", " img_path = os.path.join(class_dir, f\"{img_count}.jpg\")\n", " cv2.imwrite(img_path, frame)\n", " img_count += 1\n", "\n", " # Display the frame (optional)\n", " cv2.imshow('Capturing', frame)\n", "\n", " # Press 'q' to quit early\n", " if cv2.waitKey(1) & 0xFF == ord('q'):\n", " break\n", "\n", " # Release the capture and close any OpenCV windows\n", " cap.release()\n", " cv2.destroyAllWindows()\n", " print(f\"Captured {img_count} images for class '{class_name}'.\")" ], "metadata": { "id": "ga0jFBVqlHtE" }, "id": "ga0jFBVqlHtE", "execution_count": 74, "outputs": [] }, { "cell_type": "code", "source": [ "esp32_url = 'http://192.168.144.123:81/stream' # Replace with your ESP32-CAM stream URL" ], "metadata": { "id": "JArxNAVNl2WH" }, "id": "JArxNAVNl2WH", "execution_count": 75, "outputs": [] }, { "cell_type": "code", "source": [ "if __name__ == \"__main__\":\n", " print(f\"Trying to open video stream at {esp32_url}\")\n", "\n", " while True:\n", " class_name = input(\"Which class are you going to capture? (leave empty to exit) \")\n", " if not class_name:\n", " print(\"Exiting...\")\n", " break\n", "\n", " print(f\"Starting capture for class: {class_name}\")\n", " capture_images_from_esp32(class_name, esp32_url)\n", " print(f\"Finished capturing images for class: {class_name}\")" ], "metadata": { "id": "X0ns-iyNlr9n", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "11cda77d-23f4-4d97-88a4-eaa0eeed0cba" }, "id": "X0ns-iyNlr9n", "execution_count": 76, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Trying to open video stream at http://192.168.144.123:81/stream\n", "Which class are you going to capture? (leave empty to exit) \n", "Exiting...\n" ] } ] }, { "cell_type": "markdown", "source": [ "### Storing collected data into a csv file" ], "metadata": { "id": "_ydKy7e_8cfu" }, "id": "_ydKy7e_8cfu" }, { "cell_type": "code", "source": [ "# this code for store the collected data into a csv file\n", "def image_to_csv(image_folder_path, train_csv_file_path, test_csv_file_path, test_size=0.2, random_state=42):\n", " rows = []\n", " labels = [folder for folder in os.listdir(image_folder_path) if os.path.isdir(os.path.join(image_folder_path, folder))]\n", "\n", " for label in labels:\n", " label_folder_path = os.path.join(image_folder_path, label)\n", "\n", " for image_name in os.listdir(label_folder_path):\n", " if image_name.endswith(('.png', '.jpg', '.jpeg', '.bmp', '.gif')):\n", " image_path = os.path.join(label_folder_path, image_name)\n", " image = Image.open(image_path)\n", " image = image.convert('L')\n", " image = image.resize((40, 40)) # Resize images to 40x40\n", " image_array = np.array(image)\n", " image_flattened = image_array.flatten().tolist()\n", " row = [label] + image_flattened\n", " rows.append(row)\n", "\n", " df = pd.DataFrame(rows)\n", " column_names = ['label'] + ['pixel_' + str(i) for i in range(len(rows[0]) - 1)]\n", " df.columns = column_names\n", " train_df, test_df = train_test_split(df, test_size=test_size, random_state=random_state, stratify=df['label'])\n", " train_df.to_csv(train_csv_file_path, index=False)\n", " test_df.to_csv(test_csv_file_path, index=False)" ], "metadata": { "id": "t8zcaj0FF1FL" }, "id": "t8zcaj0FF1FL", "execution_count": 77, "outputs": [] }, { "cell_type": "code", "source": [ "image_folder_path = '/content/drive/MyDrive/Last_Task/dataset' # change according to your file address" ], "metadata": { "id": "Hg-adVHemNGT" }, "id": "Hg-adVHemNGT", "execution_count": 78, "outputs": [] }, { "cell_type": "code", "source": [ "train_csv_file_path = 'train.csv'\n", "test_csv_file_path = 'test.csv'\n", "image_to_csv(image_folder_path, train_csv_file_path, test_csv_file_path)" ], "metadata": { "id": "S2ONwxNS8wiq" }, "id": "S2ONwxNS8wiq", "execution_count": 79, "outputs": [] }, { "cell_type": "markdown", "source": [ "### Data Loading\n" ], "metadata": { "id": "uOhwyEKbUL5U" }, "id": "uOhwyEKbUL5U" }, { "cell_type": "markdown", "source": [ "Now reading our created csv file from the drive, replace this with your address of csv file." ], "metadata": { "id": "wTnB8tco9T49" }, "id": "wTnB8tco9T49" }, { "cell_type": "code", "execution_count": 141, "id": "d91ebdeb-d482-49ae-a86d-e96dfff6ad73", "metadata": { "id": "d91ebdeb-d482-49ae-a86d-e96dfff6ad73" }, "outputs": [], "source": [ "train_df = pd.read_csv(\"train.csv\")\n", "test_df = pd.read_csv(\"test.csv\")\n", "# you should change these file path according to your file path" ] }, { "cell_type": "code", "execution_count": 142, "id": "413d5d1c-0103-4152-bcbc-11de33f1ddef", "metadata": { "id": "413d5d1c-0103-4152-bcbc-11de33f1ddef" }, "outputs": [], "source": [ "# you should change these file path according to your file path\n", "test = pd.read_csv(\"test.csv\")\n", "y = test['label']" ] }, { "cell_type": "code", "source": [ "train_df.head()" ], "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 255 }, "id": "qzpDfFLvUdoq", "outputId": "469891f7-8625-4b26-b909-a5101d176cd2" }, "id": "qzpDfFLvUdoq", "execution_count": 143, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " label pixel_0 pixel_1 pixel_2 pixel_3 pixel_4 pixel_5 pixel_6 \\\n", "0 d 120 128 136 143 151 162 171 \n", "1 c 97 103 106 108 109 131 159 \n", "2 d 121 129 136 145 151 154 154 \n", "3 d 113 122 130 139 148 161 173 \n", "4 d 109 119 126 135 146 162 174 \n", "\n", " pixel_7 pixel_8 ... pixel_1590 pixel_1591 pixel_1592 pixel_1593 \\\n", "0 164 162 ... 142 73 23 24 \n", "1 165 160 ... 26 22 20 20 \n", "2 166 172 ... 48 21 24 24 \n", "3 172 162 ... 142 138 136 133 \n", "4 173 164 ... 144 140 136 132 \n", "\n", " pixel_1594 pixel_1595 pixel_1596 pixel_1597 pixel_1598 pixel_1599 \n", "0 23 23 20 18 19 29 \n", "1 43 103 82 37 35 34 \n", "2 23 20 20 29 39 40 \n", "3 102 30 19 19 18 17 \n", "4 108 39 19 18 18 17 \n", "\n", "[5 rows x 1601 columns]" ], "text/html": [ "\n", "
\n", " | label | \n", "pixel_0 | \n", "pixel_1 | \n", "pixel_2 | \n", "pixel_3 | \n", "pixel_4 | \n", "pixel_5 | \n", "pixel_6 | \n", "pixel_7 | \n", "pixel_8 | \n", "... | \n", "pixel_1590 | \n", "pixel_1591 | \n", "pixel_1592 | \n", "pixel_1593 | \n", "pixel_1594 | \n", "pixel_1595 | \n", "pixel_1596 | \n", "pixel_1597 | \n", "pixel_1598 | \n", "pixel_1599 | \n", "
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
0 | \n", "d | \n", "120 | \n", "128 | \n", "136 | \n", "143 | \n", "151 | \n", "162 | \n", "171 | \n", "164 | \n", "162 | \n", "... | \n", "142 | \n", "73 | \n", "23 | \n", "24 | \n", "23 | \n", "23 | \n", "20 | \n", "18 | \n", "19 | \n", "29 | \n", "
1 | \n", "c | \n", "97 | \n", "103 | \n", "106 | \n", "108 | \n", "109 | \n", "131 | \n", "159 | \n", "165 | \n", "160 | \n", "... | \n", "26 | \n", "22 | \n", "20 | \n", "20 | \n", "43 | \n", "103 | \n", "82 | \n", "37 | \n", "35 | \n", "34 | \n", "
2 | \n", "d | \n", "121 | \n", "129 | \n", "136 | \n", "145 | \n", "151 | \n", "154 | \n", "154 | \n", "166 | \n", "172 | \n", "... | \n", "48 | \n", "21 | \n", "24 | \n", "24 | \n", "23 | \n", "20 | \n", "20 | \n", "29 | \n", "39 | \n", "40 | \n", "
3 | \n", "d | \n", "113 | \n", "122 | \n", "130 | \n", "139 | \n", "148 | \n", "161 | \n", "173 | \n", "172 | \n", "162 | \n", "... | \n", "142 | \n", "138 | \n", "136 | \n", "133 | \n", "102 | \n", "30 | \n", "19 | \n", "19 | \n", "18 | \n", "17 | \n", "
4 | \n", "d | \n", "109 | \n", "119 | \n", "126 | \n", "135 | \n", "146 | \n", "162 | \n", "174 | \n", "173 | \n", "164 | \n", "... | \n", "144 | \n", "140 | \n", "136 | \n", "132 | \n", "108 | \n", "39 | \n", "19 | \n", "18 | \n", "18 | \n", "17 | \n", "
5 rows × 1601 columns
\n", "