How to Do Cardboard Gesture Recognition With Embedded AI

The goal of this tutorial is to show you a way to easily add AI to a project, without any knowledge in this field, using the software NanoEdge AI Studio and its Arduino compatible libraries!

This tutorial guides you through building a cardboard touchpad that relies on vibration analysis and an Embedded AI algorithm running on an Arduino UNO R4. The UNO emulates a USB keyboard device.

Vibration data from the cardboard is captured using a basic accelerometer connected via the Qwiic connector. Within the UNO microcontroller, vibrations are classified using a NanoEdge AI library.

Based on the detected class, the touchpad triggers either a "PageUp" keystroke by tapping on the cardboard or "PageDown" keystroke by swiping on the cardboard. 

Hardware:

• Arduino Uno R4
• LIS3DH triple axis accelerometer
• A micro-USB cable to connect the Arduino board to your desktop machine

Software:

• Arduino IDE: To develop the code.
• NanoEdge AI Studio: To create the AI model and library.

Download and install Arduino IDE. (For Windows users, you may need to use the 1.8.19 desktop version)

We will use Arduino IDE to create the C code for the project:

• First some data logging code to collect data and use NanoEdge to create an AI model
• A demo code to use the created model and play with it

Install NanoEdge AI Studio

Documentation: NanoEdge AI Studio Documentation

NanoEdge is a free machine learning software developed by STMicroelectronics which allow to easily create and integrate AI libraries to any cortex M microcontroller. Essentially, select a project type, import data locally, run a benchmark to find the best model automatically, test the model if you want and get an AI library.

In Nanoedge AI Studio, four kinds of projects are available, each serving a different purpose:

• Anomaly detection (AD): to detect a nominal behavior and an abnormal one. Can be retrained directly on board.
• 1 class classification (1c): Create a model to detect both nominal and abnormal behavior but with only nominal data. (In case you cannot collect abnormal examples)
• N class classification (Nc): Create a model to classify data into multiple classes that you define
• Extrapolation (Ex): Regression in short. To predict a value instead of a class from the input data (a speed or temperature for example). 

Open Arduino IDE and create a new project:

1. Download the .ino code file attached
2. Open the .ino file in Arduino IDE
3. Click on Sketch > Include Library > Adafruit_LiS3DH to install the library.
4. Click compile and flash the code.

Be careful : If your board is emulating a keyboard, you need to double press the reset button to be able to flash it.

Open the serial in Arduino IDE to check that the accelerometer data is correctly coming to your computer. (don't forget to close the serial after that) 

Now that we are able to collect data via serial, we will use NanoEdge to create the dataset

The goal is to create an AI model able to recognize two classes:

• Tapping to trigger a "Page up" keystroke.
• Swiping to trigger a "Page down" keystroke.

To do so, open NanoEdge AI Studio and create a new " N-Class Classification " project.

In project settings:

• Set target to "UNO R4 Wifi"
• Sensor to 3-axes accelerometer 
• Click Next

Here we will collect both tapping and swiping gesture with the accelerometer.

In the Signal step:

• Make sure the board is connected to the pc with the data logger code flashed on it.
• Click ADD SIGNAL > FROM SERIAL
• Make sure to select the right come port.
• Click START/STOP to collect data (100 buffers per signal should be enough)
• Collect one kind of gesture per dataset!
• Once finished click CONTINUE and then IMPORT. 

If everything is correct, you should see a new dataset added with plots and information.

Create the AI model:

Once you have all the classses that you want to recognize, go to the Benchmark step.

• Click RUN NEW BENCHMARK.
• Select all your datasets and click START.

NanoEdge AI Studio will take your data and look for a model that is able to classify them. (it also applies pretreatment on its own to your data)

You get the accuracy of the model and its RAM and Flash requirements.

You should reach an accuracy of around 99% pretty fast if you collected good data. You can stop the benchmark when it happens. 

Once satisfied with the model found on the benchmark, we can test it in the Emulator step with new serial data:

• Click INITIALIZE EMULATOR
• Click FROM SERIAL
• Test your model with new real time data. 

If you are not satisfied with the performance of the model you may want to try another model. To do so, go to the Validation step and select another model.

You can also try to log more data, making sure to have a lot of different examples covering all the possible situations.

The output of NanoEdge is an AI library containing the model but alos the preprocessing used and functions to interact witht the model (here make classification)

The last step in NanoEdge is to get the AI library that we will use in Arduino IDE:

1. Click COMPILE LIBRARY
2. Get and extract the .zip file
3. The library is the .zip file in the folder Arduin , we will use it below. 

Now that we have the model, we will create a new code to both log data, classify the gesture detected and activate the key activation:

• Download the attached file below
• Open it in Arduino IDE
• Click on Sketch > Include Library > Adafruit_LIS3DH to be able to use the accelerometer.
• Add the Nanoedge AI Library like in the associated picture (select the previously extracted zip)
• Compile the code.
• Flash the code.

It is finished, you can play with it. 

The code is commented to understand how it works.

For the usage of NanoEdge AI functions, you can take a look here.

Through this example, this tutorial shows how to add AI to an Arduino project using NanoEdge AI Studio. It is now up to you to think about other use case and do them on your own!