Hand Motion Recognizer

Overview

In this project, we'll make a glove that can recognize some basic hand movements, using a MicroBit, and a few sensors. We'll be using the Bluetooth capabilities on the MicroBit, in conjunction with an Android App and an Web Server to train a machine learning model to identify hand movements.

Getting Started

A majority of the effort involved in this project is on the software side, and all the code needed to run this project is available on GitHub. The code base involves 3 components, the code to generate a HEX file for the MicroBit, the Android App codebase which is heavily based on the MicroBit Foundation's MicroBit Blue app, with modifications made for this specific use case, and a web server with code for training a Tensorflow based model to identify hand movements.

We'll see how to build the glove and hook it with the App and Web Server next.

• 1 BBC Microbit
• 1 Battery Holder with 2 AAA batteries
• 1 Glove
• A set of jumper wires, alligator clips
• A flex sensor
• A force sensor
• Velcro
• Electrical Tape
• An Android Phone
• A PC/Laptop

• Start by attaching the battery holder to a piece of velcro as shown in the first image. Use electrical tape to firmly attach the battery holder to the velcro strap.
• Next make a loop with electrical tape such that it is sticky on both sides, and stick it on top of the battery pack.
• Stick the MicroBit onto the loop of tape to firmly attach the MicroBit to the Battery Holder as shown in the second image.

• Follow the circuit diagram shown in the image to connect your flex sensor to Pin 1 of the MicroBit, and force sensor to Pin 0 of the MicroBit.
• Secure the sensors on the Glove using electrical tape as shown in the images.

• Use the ends of the velcro straps to form a loop and slide the loop over the fingers of the glove, as shown in the image.
• You can use wire ties to secure the wires on the glove to prevent them from moving too much.

In the next section we'll look at how to setup the software.

Pairing your phone to your MicroBit

1. To pair your phone, first ensure bluetooth is on on your phone.
   
2. Power up your MicroBit, and press and hold both A and B buttons. At the same time press and release the reset button while still holding A and B buttons. The microbit should now enter pairing mode.
   
3. On your phone, find your MicroBit under the bluetooth devices list where you usually add a new Bluetooth device, and begin pairing. On your MicroBit you'll see an arrow pointing to the A button. When you press this, the MicroBit will display a series of numbers which is the Pairing Code that you must enter on your phone. Once you enter the code on your phone and select pair, a check mark should be displayed on the MicroBit.
   
4. Press the reset button on your MicroBit.

Setting up the software

Follow the ReadMe guides in each sub folder on the GitHub Repository to setup the Android App project in Android Studio, to build and flash the HEX file to your MicroBit, and run the web server for running the Machine Learning Models.

Web Server

• Open a terminal in the Web Server's project directory and run `python server.py` to start the server after following the instructions in the ReadMe to install dependencies.

Android App

1. Build and make an APK for the Android App from Android Studio. Run the app after pairing your phone with the MicroBit (see previous step).
2. In the accelerometer page, you can set the web server url using the settings menu in the top right corner. Make sure you change this to your Web Server's IP.
3. Wait till the accelerometer readings start getting populated from the MicroBit. You will see the readings change with different frequency. To change the frequency press B on the MicroBit. Ideally you can use a frequency value of 10 (which samples readings every 10ms)
4. Once the readings are being populated, name your gesture using the Text Box labeled 'Gesture:', and press the record button. As soon as you press the record button, make your hand movement, repeatedly until the button gets enabled again.
5. Repeat step 3 for recording multiple gestures.
6. Press the train button to initiate the model training on the server. Once the training is done (about 15 seconds), you can proceed to make predictions.
7. Press the predict button and make your movement/gesture. The app will try to match it to one of the trained movements as best as possible.