Teaching Machines to Understand Sign Language

This project follows the Engineering Design Process. Confirm with your teacher if this is acceptable for your project, and review the steps before you begin.

Setting Up the Google Colab Environment

1. You will need a Google account. If you do not have one, make one when prompted.
2. Download the sign_language_detection.ipynb file from Science Buddies. This is the code you will need to process your data.
3. Within your Google Drive, click on ‘MyDrive,’ then create a new folder and rename it sign_language_detection. Inside the folder, upload the sign_language_detection.ipynb file.
4. Double-click on the sign_language_detection.ipynb file. This should automatically open in Google Colab. You will need to do the following in the notebook:
   1. Read the Troubleshooting Tips and How to Use This Notebook sections. Follow the instructions you find in that section.

Install Dependencies

1. Run all the code blocks under this section. This will download various dependencies such as Tensorflow, OpenCV, and Mediapipe.
   1. Tensorflow – A tool that helps computers learn from data, often used to build and train machine learning models.
   2. OpenCV – A library that helps with working on images and videos, like reading video files or drawing on frames.
   3. Mediapipe – A tool made by Google that can find things like hands, faces, and body positions in videos or images.

Importing Libraries

1. Run all the code blocks in this section to ensure we can access all the functions needed for this project and the files stored on your Google Drive.

Detecting Keypoints Using MP Holistic

1. (Code Block 2A) This code block sets up tools to find and show body parts like the face, hands, and body in images or videos using MediaPipe. It includes functions to detect these parts and draw them on the image. Run this code block.
2. (Code Block 2B) This step tests whether the code can successfully read and process videos. In your sign_language_detection folder on Google Drive, upload a video of yourself waving your hands. You only need to show your upper body, and your hands should be visible at some point so that the MediaPipe model can detect them. The code will display individual video frames with keypoints drawn on them.  Make sure to:
   1. Replace the filename under the #TODO comment with the exact name of your video (including the file extension).
   2. Re-run the Importing Libraries section if the uploaded video is not being found.

Extracting Keypoint Values

1. (Code Block 3A) This code block takes the points detected by MediaPipe–like your body, face, and hands–into one long list of numbers. If parts are not found in a frame, it fills in zeros so the list always stays the same size. This makes it easier to use the data for machine learning. Run this code block.

Setting Up Folders for Collection

1. (Code Block 4A) This code block sets up folders to store keypoint data for each sign language gesture. Under the #TODO comment, enter the names of the gestures for which you want to collect data. By default, the code includes 'hello', 'thank_you', and 'see_you_later', but you can add as many gestures as you’d like. 
   1. Note: After running this code block, two new folders called MP_Data and videos will be created inside your sign_language_detection folder on Google Drive. Inside both MP_Data and videos, you will see subfolders named after each gesture you listed (e.g., 'hello', 'thank_you', 'see_you_later'). 
2. Now it is time to collect 30 video samples for each gesture. You can film with a phone or a webcam. If you are unfamiliar with sign language, you can look up sign language tutorials on YouTube or find other online resources. Ensure the footage focuses solely on the gesture–avoid including actions like reaching towards the camera to stop the recording. If you are filming alone, trim out these parts. Having a second person press the record button can make this process easier and result in cleaner videos. Try to make each video slightly different to help the model learn better. For example, you can:
   1. Move your hands at different speeds while signing.
   2. Vary your position in the frame (closer to or farther from the camera).
   3. Use different hands if the gesture allows for it (e.g., left vs. right).
   4. Add small body movements like shifting your weight or turning slightly.
   5. Change facial expressions (since face landmarks are also captured).
3. Upload your videos to the correct subfolder inside the videos folder. Then, re-run the “Importing Libraries” section to ensure your notebook can access the newly uploaded videos.

Collecting Keypoint Values for Training and Testing

1. (Code Block 5A) This code block reads each video, grabs 30 frames, extracts keypoints from each, and saves them for training your sign language model. Run this code block.

Preprocessing Data and Creating Labels and Features

1. (Code Block 6A) This code block creates a label map – a dictionary that assigns a unique number to each gesture name in your dataset. We do this because machines understand numbers better than words like 'hello' and 'thank_you'. Run this code block. 
2. (Code Block 6B) This code block reads all your saved .npy files (which contain keypoints for each frame), groups them into 30-frame sequences (representing one video), and stores them in a list called sequences. It also stores the corresponding numeric label for each sequence in a list called labels, so your model knows which gesture each sequence represents. We do this to organize our data before giving it to our model for training. Run this code block.
3. (Code Block 6C) Splitting data into training and testing sets is important in machine learning. It helps to see how well your model works on new data. Watch this video to learn more about why we split datasets. Pay attention to the sizes of X_train, X_test, y_train, and y_test. If you see the X_train size as (85, 30, 1662), that means there are 85 training samples (videos), each sample has 30 frames (we extracted 30 frames per video), and each frame contains 1662 features, which come from flattening all of the detected keypoints (pose, face, left hand, right hand) into one long array.

Building and Training an LSTM Neural Network

1. (Code Block 7A) This code block builds a neural network to classify sign language gestures based on 30-frame video sequences of body keypoints. It uses LSTM (Long-Short Term Memory) layers to learn patterns over time, followed by dense layers to refine the prediction. Run this code block.
   1. LSTM is a type of neural network layer that is great at learning from sequences, like video frames or time series data. It can remember important patterns across time, making it useful for tasks where order and context matter.
2. (Code Block 7B) Compiling a model sets up how the model will learn during training. This step is necessary before training and ensures the model can improve its predictions and evaluate its accuracy. Run this code block.
3. (Code Block 7C) Using the training data, this code block trains the model to recognize sign language gestures. It runs 200 times over the dataset to help the model learn. Run this code block.

Making Predictions

1. (Code Block 8A) This code block uses the trained model to predict the test data. It will output the number of predictions it has made. Run this code block.
2. (Code Block 8B) This code block displays the predicted gesture for a specific test sample–by default, it is set to the 4th sample (index 3). You can change the index to any number between 0 and the total number of predictions shown in the previous code block -1 to view the predicted gesture for different test samples.
3. (Code Block 8C) This code block retrieves the true label (actual gesture) for the specific test sample–by default, it is set to the 4th sample (index 3). If you changed this value in the previous code block, change it to the same index here. Compare whether the predicted value was the same as the actual value.

Saving Weights

1. (Code Block 9A) This code block saves the trained machine learning model (the LSTM model) to a file so you can reuse it later without needing to retrain it. Run this code block.
2. (Code Block 9B) This code block loads a previously saved model from the file specified by model_path. Run this code block to use the model again.

Evaluating Using a Confusion Matrix and Accuracy

1. (Code Block 10A) This code block uses the trained model to make predictions and then compares them to the correct answers. It turns the predictions and actual labels into numbers so they can be easily compared and checked for accuracy. Run this code block.
2. (Code Block 10B) This code block calculates a multilabel confusion matrix, which helps evaluate how well the model performs for each individual class (in this case, each sign or gesture). Instead of creating a single confusion matrix for the entire model, it generates a separate confusion matrix for each class. This allows you to see how often the model currently identifies a specific gesture versus how often it confuses it with others. To learn more about interpreting confusion matrices, click this link here.
3. (Code Block 10C) This code block calculates the accuracy of the model.
   1. Accuracy is a performance metric that measures how often a classification model correctly predicts the labels.
      Accuracy = (Number of Correct Predictions) / (Total Number of Predictions) 
   2. In other words, accuracy tells you what percentage of the total predictions made by the model were correct.
   3. If a model makes 5 predictions and 4 of them are correct:
      Accuracy = 4/5 = 0.8 or 80%.

Testing with Videos

1. Now, it is time to test how well your model recognizes multiple signs in a row. Record a single video of yourself performing the gestures your model was trained on, but do them randomly and use different hands if applicable to add variety.
2. Once you are done recording, upload the video to the test_videos folder in your Google Drive.
3. (Code Block 11A) Next, re-run the “Importing Libraries” section of your notebook so the video becomes accessible in your code. Under the #TODO comment, replace the text defining the test_video variable with the exact name of your video file. Run the code block, and when it finishes, it will generate a new video called output_video.mp4 in the same test_video folder. In this output, you will see:
   1. A blue bar at the top showing the model’s detected gesture.
   2. The probabilities of each possible gesture displayed below the blue bar.
      Watch the video and see if your model could recognize your signs accurately. 

(Optional) Improving Your Model

If the results are not what you desire, here are some suggestions on how you can improve your model:

1. Collect More Diverse Training Data
   1. Include multiple people performing the signs to improve generalization.
   2. Perform signs with both left and right hands, and change the angle of your elbow if applicable.
2. Increase Training Data Quality
   1. Trim gesture clips accurately to avoid noisy data.
   2. Increase the amount of data you have.
   3. Note: If you decide to add more data, you would need to delete the MP_Data folder and increase the variable no_sequences to the number of videos you want to take.