AI Gesture Smart Wand Tutorial

We've had plenty of experience with machine learning projects on computers. But can we create a project that allows machine learning models to run on hardware, independent of a computer?

Recently, there's an open-source project that uses the STM32 microcontroller to run NNoM. It deploys a convolutional neural network motion classification model on the STM32. By detecting the attitude data from the six-axis gyroscope MPU6050, it returns the correct classification.

The original address of the open-source project is: GitHub - lyg09270/CyberryPotter_ElectromagicWand_Basic_Project: Cyberry Potter electromagic wand. Basic project that can light up LEDs with different (spell) motions, and can also modify the LEDs to other functions as needed.

This tutorial and the open-source project's tutorial will be more detailed, filling in the pitfalls we've encountered, so everyone can successfully replicate it! Also, the open-source project returns results as serial data, which is not very intuitive. Therefore, we've added two Microbits to visualize the returned data, making the project more vivid and interesting.

Hardware:

1. Microcontroller: STM32F103C8T6 Development Board
2. IMU Sensor: MPU6050
3. Button: Sugar Button Module
4. LED: 5mm Through-Hole LED
5. CH340 USB Serial Downloader
6. STlink
7. Microbit V2 x2
8. Iobit
9. Robotbit
10. Sugar String Lights Module
11. 3D Printed Parts
12. Meowbit Battery Pack

Software:

1. ST-LINK Utility
2. Serial Debug Assistant (or other serial debugging software)
3. MakeCode

Tools:

1. Soldering Iron
2. Hot Glue Gun
3. Screwdriver
4. USB Extension Cable
5. Wire Stripper
6. 3D Printer

Other Materials:

1. M3-8 x 3 Screws
2. Self-tapping Screws M1.7-6 x 4
3. 3M Adhesive
4. Female-to-Female Dupont Wires x10

What's NNoM？

NNoM is a lightweight neural network framework designed specifically for microcontrollers. It supports common deep learning functions, such as Convolutional Neural Networks (CNNs) and fully connected layers, and has the following features:

1. Efficient and Lightweight: Suitable for embedded devices with limited resources.
2. Easy to Use: Provides intuitive APIs and a variety of example codes.
3. Open Source: Supports free modification and extension.

In this project, NNoM will be used to run a gesture classification model on the STM32 microcontroller, enabling offline gesture recognition.

Wands (four styles, shared by netizens):https://makerworld.com/zh/models/519053?from=search#profileId-440711

Matching wand casings (designed by KittenBot):https://makerworld.com/zh/models/1034922#profileId-1018455

The core of this project lies in training the model. However, setting up the environment can be quite troublesome and often poses a challenge for beginners. Therefore, this tutorial will only teach you how to download the pre-trained program (which includes 12 standard gestures, generally sufficient for most use cases). Once you have successfully completed this project and want to delve deeper, such as adding or modifying gestures, you can explore the open-source project repository on your own.

The project can be broken down into the following steps:

1. Wiring the STM32 with the MPU6050
2. Connecting the STlink programmer to the STM32 and downloading the program
3. Connecting the USB serial downloader to the STM32 to test the serial return data
4. Writing and testing the 2.4G wireless communication program between Microbits
5. Modifying the Microbit 2.4G wireless communication program to add serial data reception and judgment
6. Connecting the STM32's serial port to the Microbit's serial port and testing
7. Assembling the printed casing to form the wand

Connecting the STM32 with the MPU6050

Please strictly follow the wiring diagram below for connection.

(As shown in Figure 1 above)

You can use the 3D-printed parts I designed for fixing and soldering.

(As shown in Figure 2 above)

Connect the STLink programmer and download the compiled hex file.

Wiring for the programmer:

(As shown in Figure 1 above)

ST-LINK Utility Software Download:

https://www.st.com.cn/zh/development-tools/stsw-link004.html#get-software

Usage:

Connect the main controller

(As shown in Figure 2 above)

Select the programming file

(As shown in Figure 3 above)

Download the provided file "CyberryPotter.hex" and click "Start"

(As shown in Figure 4 above)

Prompt that the download is successful

(As shown in Figure 5 above)

After the download is complete, you can unplug the STLink programmer.

Connect the serial debugger to test whether the program and hardware are running correctly.

Wiring between CH340 and STM32:

(As shown in Figure 1 above)

Physical Connection:

(As shown in Figure 2 above)

Connect using a serial tool:

The baud rate is 115200. After opening the serial port, press the reset button on the STM32 board.

(As shown in Figure 3 above)

The window will receive

"dmp init complete

Hello!"

[!NOTE]

If the serial port does not receive any information, please check:

1. Whether the program has been successfully downloaded
2. Whether the serial wiring is correct

(As shown in Figure 4 above)

When you press the button without making any gestures, the window will receive "Unrecognized," which indicates that your board is working properly. At this point, when you perform the corresponding gestures, it will return the corresponding results.

[!NOTE]

If the serial port does not receive any information, please check:

1. Whether the wiring between the gyroscope and the STM32 is correct

(As shown in Figure 5 above)

For example, perform a →↓ motion. Press the button (just tap it, do not hold it down), and complete the motion within 1.5 seconds.

The serial port will return the result, which also indicates that your MPU6050 and the trained model are working properly.

(As shown in Figure 6 above)

(The effect is as shown in the GIF.)

At this point, the core functionality of the AI Gesture Smart Wand has been successfully implemented.

When the corresponding gesture is performed, the serial port will return the corresponding string. You can use another microcontroller to interface with the serial command, thus enabling various applications.

The gesture results are shown in the figure below:

(As shown in Figure 8 above)

Transmitter

Assume Microbit_A acts as the transmitter, connecting to the STM32's serial port to receive the serial results returned by the gestures.

Here, an IObit is used for interfacing.

Microbit —— STM32

P12————A10

P13————A9

5V————5V

GND————GND

The lithium battery supplies 5V to the IObit.

(As shown in Figure 1 above)

Program address:

https://makecode.microbit.org/_5Pi3AiUAKJv2

(As shown in Figure 2 above)

Receiver

Only one Microbit is needed, which is plugged into the Robotbit with an external cube LED strip module (P1).

(As shown in Figure 3 above)

Program address:

https://makecode.microbit.org/_WvyiVaMzuPze

(As shown in Figure 4 above)

Physical Testing

When gestures 1-6 are triggered, the LED strip turns off.

When gestures 7-12 are triggered, the LED strip lights up.

[!NOTE]

Tip: Press the button and release it, then complete the gesture within 1.5 seconds. Make the gesture more forceful.

Assemble the AI Wand, paying attention to the installation positions of the screws. Use double-sided tape to attach the lithium battery pack to the side of the casing.

(As shown in Figure 5 above)

Assembly Complete

(As shown in Figure 6 above)

Testing

(The effect is as shown in the GIF.)

The core basic interaction has been successfully tested.

The rest is up to your creativity.