Key Management System (PyKMIP)

Boxing Punch Classification with LSTM

Introduction

This project aims to classify different boxing punches using data collected from an ESP32. The project consists of two Arduino codes for the ESP32, one for data collection and one for streaming data serially through Bluetooth, a Python notebook for preprocessing, building, training and evaluating the model, and a p5.js application for providing a user interface for interacting with the trained model.

Project Details

Hardware Requirements

ESP32

Additional equipment for data collection (e.g. accelerometer)

Software Requirements

Arduino IDE with ESP32 support

Required libraries for ESP32 code

Python 3 with necessary libraries (e.g. TensorFlow, pandas)

p5.js library for the user interface

Project Process

Data Collection

Objective

Collect 3 dimensional accelerometer data using the MPU 6050 and ESP32 board, each sample should contain 60 frames over a 1.5s period.

Challenges:

I want a way to collect data and label it in an efficient way, simply collecting the streamed data, would make it close to impossible to split it into 60 frames, find a where a punch occurred, and label the punch accordingly.

Solution:

I wrote some procedural code to the ESP32, such that it will prompt me with different punch types, and once I send the input associated with the punch the subject intends on preforming, then the ESP32 will stream the accelerometer data for 60 frames result in an time frame of ~ 1.5s

Data sample


@@ 1: JAB | 2: HOOK | 3: UPPERCUT | 4: STANCE
3,-2.00,-2.00,1.80,1
3,-2.00,1.52,1.51,1
3,-1.11,1.88,1.18,1
3,-2.00,-0.59,2.00,1
3,-1.53,-0.52,2.00,1
3,-0.79,0.17,0.87,1
3,0.67,0.73,0.19,1
...

The code is collected in text file, the @@ is used in the preprocessing phase to remove the data collection prompt

Training and Evaluating LSTM model

Model:

The model architecture consists of convolutional and dense layers with ReLU activation functions. The input layer is a 2D convolutional layer that takes the input data with a filter size of (2,2) and 16 filters, and applies a ReLU activation function. A dropout layer is added with a dropout rate of 0.1 to prevent overfitting.

The next layer is another 2D convolutional layer with a filter size of (2,2) and 32 filters, followed by another dropout layer with a dropout rate of 0.2.

Then, the Flatten layer is used to convert the output of the convolutional layers to a 1D array. After that, a dense layer with 64 units and a ReLU activation function is added, followed by another dropout layer with a dropout rate of 0.5.

Finally, the output layer has 4 units with a softmax activation function to classify the input data into one of the four boxing movements.

The model is compiled using the Adam optimizer with a learning rate of 0.001 and a loss function of sparse categorical cross-entropy. The metrics used to evaluate the model during training are accuracy.

Training: The model is trained for 10 epochs on the training data (X_train, y_train) and evaluated on the validation data (X_test, y_test) with the validation_data parameter. The verbose parameter is set to 1 to display progress during training.

Evaluation

User interface (P5.js Application)

I used p5.js a JavaScript library for creative coding, on the canvas i displayed the predicted punch with the confidence score in green.

below the canvas I displayed a single frame of the streamed data from the ESP32 to confirm that the connection is working.

The user can select record to start record a 1.5s frame of the accelerometer data as he/she throws a punch.

the Connect to serial button will list computer ports, the user can select the ESP32 bluetooth through that.

Demo Video

Results and Impact

What I’ve Learned

This project taught me a lot about about Machine learning, LSTM models, Data processing , data collection, model training, and microcontroller bluetooth module.

Future

This project is currently being reused for a similar application, to create a skateboarding game, where the game prompts the user with a skateboarding trick and the evaluates if the user landed the trick using the LSTM classification model that I used in this project.

Conclusion

In conclusion, this project successfully achieved its goal of classifying various boxing punches using data collected from an ESP32. Although some challenges were faced during the data collection process, they were resolved, and the project was completed. The project employed various tools, including two Arduino codes for data collection and streaming, a Python notebook for preprocessing and building the model, and a p5.js application for user interaction. This project serves as a great example of how technology can be leveraged to solve real-world problems,

LSTM Boxing Classifier