In this guide, I will tell you a story. We will see how to implement an AI based project using STmicroelectronics tools and the STEVAL-STWINKT1B board.

1. Evaluation board: STEVAL-STWINKT1B
2. Programmer: STLINK-V3MINI
3. Tool for programming STM32 products: STM32CubeProgrammer
4. C/C++ development platform for STM32 microcontrollers: STM32CubeIDE
5. Configuration and project creation functionalities: STM32CubeMX
6. Development of libraries optimized for AI Project based: NanoEdgeAIStudio
7. Serial viewer: Tera Term
8. Breadbord and cables
9. Small LED
10. SD card

Hello everyone! First of all, I introduce myself. It's been a long time since I've posted my projects here.... I love this site, but unfortunately, it takes a lot of passion and motivation to create something beautiful. My name is Andrea, I am 29 years old and I study prototype or industrial solutions based on microcontroller logic or similar. I have been a robotics and automation engineer. Recently, I have been passionately studying Machine Learning development for my world. I think it is absolutely great to create a Tiny Machine learning algorithm that can work at such low resources. A simple, isolated task with little energy required, kind of like our brain does, isn't it? Passion and motivation gave me the energy to tell you about this simple project at a difficult time.

A child long ago played happily, loved technology and tried to build small things. Like all young makers, the nursery was his workshop. In the mess of a messy little room, the child could learn new things and rest happily. A happy story of a lucky child.

The warm attic room was an ideal place to create and experiment with new projects. The sound of rain on the roof isolated him from everything, the ideal space to create. As the boy grew up, he learned to observe and create. curiosity was necessary to try to understand the world-what would be the next project?

the boy continued year after year doing what he loved, until a bad accident happened. Severe pain forced him to stay in bed, his health was not good and he was in danger of damaging his memory, his remembrances. In those long rainy days many thoughts returned, is if he had lost something? the world became simpler and details became important. Even that little bedroom provided so many insights, so much information. The senses were working hard to capture as many stimuli as possible, the brain was working not to lose information, the colors, the images... the sounds.

As I hope you have guessed, in the story, the boy tries to “encapsulate” a memory, a piece of intelligence from his past. The idea is to try to use a microphone to record sounds and create an ML model that can recognize among all the sounds just the sound of rain on the roof of his bedroom. The very sound that, in the past, had helped and stimulated his imagination. You guys can modify it any way you like!

TinyML, or Tiny Machine Learning, is a subfield of machine learning that focuses on deploying and running machine learning models on low-power, resource-constrained devices. These devices typically include microcontrollers and other edge hardware with limited memory, processing power, and battery capacity. TinyML enables devices like sensors, wearables, and IoT devices to perform tasks such as anomaly detection, voice recognition, or gesture recognition, all while consuming minimal energy.

What is the difference with classic ML models?

TinyML bridges the gap between traditional machine learning and embedded systems, enabling sophisticated AI functionalities in small, portable devices that were previously not possible due to hardware constraints. Here is a simple illustrative example, just to make a focus on available resources compared with a typical application.

So why work with TinyML?

As you can also see from the chart, the resources are extremely less, but so are the costs! Tiny therefore means:

1. Small and portable.
2. Low power consumption, ability to use a battery.
3. Specific AI applications, optimized for a specific task.

Can't I define precise rules in my code to solve the task?

The question also contains the answer, the advantage of working with AI based projects is just that, not adopting precise rules for your code. Let the machine identify the key features. If you think about it, that's exactly what our brains do with “intelligence.” Let the machine find the solution in an “artificial” way. At this point you should know that this process has a cost, the calculator will find the solution but you will be the one to provide the solution. You can learn more about the main techniques used for training in this article.

So in our case we will have to select which sounds belong to rain and which represent other, that's it! What you still need to know is that for our project we will use an N-class classification technique to distinguish different sounds, I give you a simple working diagram here.

How to do a project based on TinyML?

Conceptually the life cycle of an AI based project is always the same, I try to show it here in a diagram: In the next steps I will show you how to do this, step by step. What do you think now? Let's try to realize our project!

Let's get into the core of the project, and, stay alert because this is the most important step of the whole project!

Yes because over time you will understand that, there is no model or computing power sufficient to solve a problem that arises from a bad dataset.

In our specific case then, since it is an embedded solution it is even more important! Which is why I try here to list some of the most useful tips.

If you use for example STEVAL-STWINKT1B board:

1. If you already know the exact environment in which you will use your project record audio from there.
2. Use the same device (and microphone) that you will use in your project.
3. Pay attention to recording all scenarios, I'll give you an example: record sound from several different angles and distances.
4. Always make recordings of the same duration.
5. Make sure that your collection of sounds (e.g., the sound of rain) does not have a specific case more recorded, for example: 50% of the recordings with heavy rain and 50% of the recordings with light rain.
6. Record a lot! And listen again to all the sounds.

All this may seem long, and it will, but I assure you it will make a difference in your project. Before I show you how to do it though, I also want to explain why this is essential.

Once you generate the model you cannot easily know how it will work, your guarantee will be just a good job done in creating the dataset. In this way you will avoid unexpected biases. You can learn more about this here.

In this section I will show you two different methods of using your STEVAL-STWINKT1B board as a datalogger. The reason is simple: the first method is more powerful; the second is easier to use. We will see the second method in the NanoEdge AI studio chapter.

Create your dataset - High Speed Datalog function pack

Connect your STEVAL-STWINKT1B board like as you can see here:

For this part you need to install python v3.10/v3.11, so install it and at startup remember to add it to path.

Open STM32CubeProgrammer, download this tool and open the folder until you get to the “HSDatalog.bin”.

Take the STLINK-V3MINI programmer, and connect it as in the picture. Insert the microSD. High Speed Datalog function pack is a ready to install tool to use your board with your smartphone!!

Open your powershell and use this basic comands:

Now you can directly use the instructions in the video, consider that our goal is to get an audio file in binary format to convert to a .csv file.

Study and validate your dataset

As I mentioned at the beginning of the chapter, creating the dataset will be the most important thing to do. Normally a data scientist works precisely to analyze the raw dataset and figure out what correlations exist with reality.

Convert your binary file in a .wav file with:

If you find obvious patterns or features, the computer can do that as well. For this reason, I create a simple script for our example that generated an FFT with a .wav file.

The Fast Fourier Transform (FFT) is a powerful algorithm that converts a signal from the time domain to the frequency domain efficiently, reducing computational complexity It allows rapid analysis of frequency components, making it ideal for extracting features from signals in resource-constrained environments, such as ML applications on embedded devices.

This the output:

As you can see from this simple example, physics suggests that there is a difference, and our sensor registers it. The ML model will learn, for example, these differences to recognize classes.

In this section we look at how to use NanoEdgeAIStudio to generate the perfect model for your STEVAL-STWINKT1B board. What you need to know is that ST provides a complete environment for all steps from uploading data to downloading your model! Normally it is not that easy and these steps also require very specific skills. I will purposely not go into more detail here, but I want to point you to other ways to do the same thing:

1. EdgeImpulse: is a web interface that you can interact with to generate your model from scratch.
2. TensorFlow Lite for Microcontrollers and ExecuTorch: these are very powerful code-based environments, they allow you to get similar results, and they come from their full-power version.

I can try to summarize all the capabilities of NanoEdgeAIStudio with this image:

In this section, we will see how to import the data generated in the previous chapter or do it directly from serial port into NanoEdgeAIStudio. So if you failed to complete the previous step, don't worry, you can follow this part directly, in the video I will show you how to do it. All you have to do now is to keep the STEVAL-STWINKT1B connected to STLINK-V3MINI programmer and both devices connected to the PC. Download this tool from the ST site and open the folder STWIN-DTMF-Classifier-wiki-package_v2.0, follow the path:

And proceed to download it in your board again using STM32CubeProgrammer.

Again, the binary you just uploaded will do the work for you. Now follow the video and let's see how to use the NanoEdgeAIStudio development platform to generate what we want...almost! In fact, with the steps I will show you we will basically generate “text files”, as we will see it will be necessary to integrate these .h .c and .lib files into our project created in STM32CubeMX and STM32CubeIDE.

One step at a time! Now let's see how to collect data, label it, and use it for training. As you will see the computer will use a lot of time to generate the model, this will essentially depend on the amount of data and the computing power of the PC. In the video I will give a basic example to introduce you to the platform, in your case you could wait even several hours!

Output considerations from NanoEdge AI Studio

As you have been able to see, this tool seems clearly very powerful. It is, of course, a simplified environment that proves to work in a few minutes. Especially interesting is the ability to optimize the output for the specific HW by setting the performance first. I also find very convenient the possibility of being able to import live data from serial, both in dataset creation and inference. I want to quickly see two aspects that deserve further investigation. Having chosen the most suitable model, you have seen that it is possible to observe performance during the training.

Cross validation is a technique used to evaluate the performance of a machine learning model by splitting the data into multiple subsets. The model is trained and tested on different combinations of these subsets to ensure that it generalizes well to unseen data. The most common form is k-fold cross validation, where the data is divided into k folds, and the model is trained k times, each time using a different fold for testing.

Confusion matrix is a table used to evaluate the performance of a classification model. It shows the number of true positives, true negatives, false positives, and false negatives, helping to visualize how well the model distinguishes between classes.

Confusion Matrix and Cross Validation are some of the main tools used to evaluate your result. You can explore some details on how to use these tables here. Great! Our model is perfect! ..Are you sure? We are probably looking at a case of Overfitting.

Overfitting is when a machine learning model learns the training data too well, capturing noise and specific patterns that do not generalize to new, unseen data.

This results in high accuracy on training data but poor performance on validation or test data, indicating that the model has failed to learn the underlying general patterns. As always, I'll leave it here just to explore. I just feel like repeating that the best way to not get this problem is to create your dataset with the criteria I was pointing you to in the previous step. When you see these incredible results, don't believe it! It is very likely to fall into the case of overfitting.

Now let's see how to use our output!

If you are here now, you have almost reached your goal. I also imagine that, you will not be a complete beginner in this world. You probably have experience with Arduino IDE, what you need to know is that now we are not going to do anything different. STmicroelectronics provides two environments, in the first (STM32CubeMX) you configure your STM32 and in the second (STM32CubeIDE) you write your own code.

Let us follow these steps and see how to do it. Attached you will find the complete project; for simplicity's sake I show you only the procedure by which to use one of the GPIOs available on STEVAL-STWINKT1B to turn our LED ON and OFF. Download the attached project and open it, you will see this.

What you see on the right is exactly the pinout of your ST core! Incidentally this is an STM32L4R9ZIJx and you can choose pin by pin the behavior. Obviously though, they are already connected to precise peripherals on your board, so no one is stopping you from changing anything, but if that core pin is physically connected for example to the microphone you won't be able to configure it as a digital output! it simply wouldn't work. So you have to look at the schematic and figure out “what you can and cannot do” For example this is the schematic of our core, let's see the microphone where it is connected. (Schematic available here).

As seen in the diagram above, the output of our IMP34DT05 digital microphone directly reaches PIN PB6 of the core. Therefore, the configuration of PB6 will have to be correct.

In the same way as the PB6 PIN was configured we now see how to manage one of the free GPIOs to drive an LED, on schematic we understand that the CN2 connector provides unmanaged pins, in particular here you also find a GND reference, necessary for our case. following the schematic below we choose to use pin 20 and pin 5 or 16 for GND. Also in the picture you can see the corresponding footprint. We configure for the same reason pin PG12 as a digital output.

At this point you can edit the configuration again as you wish, just press the blue button in the upper right corner “GENERATE CODE” to edit or create the project for STM32CubeIDE. Again, I have not been so detailed in my explanation of STM32CubeMX, partly because very specific technical skills are required and I don't think I can be complete in my explanation anyway. I recommend that you study other cases by watching these excellent videos.

We are now ready to enter our code into the STEVAL-STWINKT1B board. As you may have noticed, pressing “GENERATE CODE” will give you a new project automatically. In a very simple way, STM32CubeMX will generate for you the code needed to use the microcontroller! It will remain for you to edit the “main.c” file to code the behavior you want. But Andrea the files generated with NanoEdgeAI studio!?

Is now time to use them! Go to the library folder,

1. Copy the file libneai.a Go to the project folder ..\STWIN_DTMF_Classifier\Core\Src, paste the file here, and click on Replace the file in the destination in the pop-up window.
2. Go back to the library folder, copy the files knowledge.h and NanoEdgeAI.h. Go back to the project folder ..\STWIN_DTMF_Classifier\Core\Inc, paste the files here, and click on Replace the files in the destination in the pop-up window.

Import project into STM32CubeIDE workspace Open STM32CubeIDE on your PC, create or choose a workspace and click Launch to launch your workspace.

In the menu bar, click File -> Import... In the pop-up window, select Existing Projects into Workspace, and click Next, as shown below: If everything worked now you will have the files generated by NanoEdgeAI studio visible like this:

Let's also quickly see what the newly added files are:

1. knowledge.h contains a large array of float constants, probably used as a kind of knowledge base for the generated model.
2. NanoEdgeAI.h provides an interface for using STMicroelectronics' NanoEdge AI library, defines the API to interface with the classifier. It uses a buffered approach to process the input data.
3. libneai.a contains the machine learning model generated by NanoEdge AI Studio, along with all the code needed to perform inferences, and is used as a library in your embedded project.

Attached to this section you will be able to download the entire project for the STM32CubeIDE workspace. I would just like to report here the main loop and describe the behavior.

Press CTRL + S to save the change. Make sure the STWIN and STLINK_V3MINI are connected to PC via USB cables, and then click on the "Run" button in the Toolbar to build and program the project. You can see the log information in the Console window. When the project is successfully built and programmed on the STWIN, you should see the following information:

The while(1) loop is an infinite loop that runs continuously. In embedded systems, such loops are common and typically run until the device is powered off or reset. The condition audioDataReady != 0 checks whether new data from the microphone has been acquired and is ready for processing. Once the microphone data is ready (controlled by interrupts or DMA), audioDataReady is set to a non-zero value. If data is ready, the code calculates the bufferIndex (which determines the starting point of the new data in the buffer) and resets audioDataReady to 0. The code copies the acquired and processed microphone data into the input_user_buffer (used as input for the NanoEdge AI classification algorithm). A for loop iterates through the audioBufferProcessed array and copies its data into the input_user_buffer. This prepares the data for the AI classification. Calls the neai_classification() function, which processes the input_user_buffer and fills output_class_buffer with the probability for each class. It also sets newClass to the class ID with the highest probability. The classification function uses the data in input_user_buffer to perform machine learning inference and assigns a class ID to newClass. If the identified class (newClass) is 2, it increments a counter (consecutiveClassCounter). If this counter reaches a threshold (CONSECUTIVE_THRESHOLD), it turns on the LED connected to a GPIO pin. Otherwise, the LED remains off. The classification result is printed every second, regardless of class changes. Checks if the identified class (newClass) is different from the previous class (oldClass), and updates oldClass if necessary. You could add specific actions to be performed when a class change occurs (e.g., logging, triggering events).

The use of the variable (CONSECUTIVE_THRESHOLD) is simply an attempt to intervene in the behavior of the model once it has now been created! This allows you to tune the behavior in the field, imagine for example managing this threshold through an external command such as a trimmer or a value from a serial port. Now open Tera Term, you can view the behavior from here too!

The boy is now out of danger. The boy was afraid he might forget important things, or even lose brain abilities. I hope this useless project will be useful to someone, at least as it was for me.

Good things are sometimes simply like that.