Add Connectivity to Doorbell With AI

The goal of this tutorial is to add functionality to a non connected doorbell. We will detect when the doorbell is ringing to send a message for example.

We will do it in a total non intrusive way, meaning that we will only use the sound made by the doorbell. We will not disassemble anything.

We will use NanoEdge AI Studio (a free tool) to automatically create an AI model able to classify if the doorbell is ringing or if it is just background noise.

Do not worry, you don't need knowledge in AI to follow this tutorial :)

Here is the plan:

1. Setup
2. Collect microphone data
3. Create the classification model
4. Add the model in our Arduino code

Hardware:

• Arduino Uno R4 WIFI
• Max4466 microphone
• A micro-USB cable to connect the Arduino board to your desktop machine

You also need a doorbell, In my case, I use the one that I had:

https://www.amazon.de/-/en/Wireless-Doorbell-Waterproof-Ringtones-Flashing/dp/B0B2P3PKHH

Software:

• Arduino IDE: To develop the code
• NanoEdge AI Studio: To detect the sound class automatically (click on the link to get the software)

First, we need to connect the microphone to the Arduino board.

Use jumper wires to connect:

• OUT (mic) to A0 (board)
• GND to one of the GND on the board
• VCC to 3.3v

Make sure that you have a USB data cable connecting the board to the pc.

Then we need to put the microphone close to the part of the doorbell that make sound!

In Arduino IDE:

Make sure you selected the right COM port: Tools > Port and select the right one.

Select the right board:

• Tools > Boards > Arduino Renesas UNO R4 boards > Arduino UNO R4 WIFI
• If you don't find it, click on Tools > Boards > Boards Manager..., look for the UNO R4 and install the package

We use a digital microphone that has a very high data rate.

We will collect chucks of the music by collecting buffers of values from the microphone and also reduce the data rate by keeping only 1 value every 32 collected.

We collect buffers of music and note single notes to classify them. Even for a human it is impossible to recognize a song with one random note taken from the song.

To accomplish this:

• Define the AMP_PIN to A0 as our microphone use the A0 pin to send data
• We define a buffer called neai_buffer to stock the value collected
• In our case, the buffer is of size 1024 (SENSOR_SAMPLE)
• We initialize the serial in the setup()
• We create a get_microphone_data() to collect buffers of data from the microphone. We get only 1/32 values
• We call the function only if a sound > 400 is detected, to avoid anything random noise that cannot be a doorbell sound. Depending on your equipment, you may need to lower it.
• We print the buffer to send it via serial.

The code:

/* Defines  ----------------------------------------------------------*/
#define SENSOR_SAMPLES    1024 //buffer size
#define AXIS              1    //microphone is 1 axis
#define DOWNSAMPLE        32   //microphone as a very high data rate, we downsample it


/* Prototypes ----------------------------------------------------------*/
void get_microphone_data(); //function to collect buffer of sound


/* Global variables ----------------------------------------------------------*/
static uint16_t neai_ptr = 0; //pointers to fill for sound buffer
static float neai_buffer[SENSOR_SAMPLES * AXIS] = {0.0}; //souhnd buffer
int const AMP_PIN = A0;       // Preamp output pin connected to A0


/* Setup function ----------------------------------------------------------*/
void setup() {
  Serial.begin(115200);
  delay(10);
}


/* Infinite loop ----------------------------------------------------------*/
void loop() {
  if (analogRead(A0)> 400){
    get_microphone_data();
  }
}


/* Functions declaration ----------------------------------------------------------*/
void get_microphone_data()
{
  static uint16_t temp = 0; //stock values
  int sub = 0; //increment to downsample
  //while the buffer is not full
  while (neai_ptr < SENSOR_SAMPLES) {
    //we only get a value every DOWNSAMPLE (32 in this case)
    if (sub > DOWNSAMPLE) {
      /* Fill neai buffer with new accel data */
      neai_buffer[neai_ptr] = analogRead(AMP_PIN);
      /* Increment neai pointer */
      neai_ptr++;
      sub = 0; //reset increment
    }
    else {
      //we read the sample even if we don't use it
      //else it is instantaneous and we don't downsample
      temp = analogRead(AMP_PIN);
    }
    sub ++;
  }
  //print the buffer values to send them via serial
  for (uint16_t i = 0; i < SENSOR_SAMPLES; i++) {
    Serial.print(neai_buffer[i]);
    Serial.print(" ");
  }
  Serial.print("\n");
  neai_ptr = 0; //reset the beginning position
}

To use this code, copy and paste it in Arduino IDE. If you have followed the setup part, you only need to click on UPLOAD (little arrow on the top).

In the next step, we will use this code to collect data in NanoEdge AI Studio and create an AI library to classify songs.

The goal here is to use NanoEdge to create an Anomaly detection project. We want to detect the doorbell as "nominal" and everything else as an "abnormal" behavior. To do that:

1. Open NanoEdge
2. Create a Anomaly Detection project
3. Select the Arduino R4 WIFI board as target (other boards are compatible)
4. Select Microphone 1axis as sensor
5. Click Next

Then we will collect data for every music. In the NOMINAL STEP:

1. Click ADD SIGNAL
2. then FROM SERIAL (USB)
3. Then click START/STOP to collect data (make sure the right COM port is selected)
4. Record multiple times the doorbell ringing (x100). Avoid empty buffers (pause if you need)
5. Click CONTINUE then IMPORT
6. Rename the file if you want

For the ABNORMAL STEP, you want to collect everything else that can happen at your house (sounds of kids playing, dog, piano, works, vacuming etc). You can let record what is happening every minute for example (add a delay in the datalogging code after the get_microphone_data() function).

What I did is that I used multiple "house background noise" videos found on Youtube.

Once you have everything that you want, go to the BENCHMARK STEP.

The more song you have, the harder it will get, so start simple.

1. Click on NEW BENCHMARK
2. Select all the song and click START

The benchmark will look for the best model and preprocessing of your data to find a model that is able to classify the songs.

I got a good benchmark score and it was working as expected in the emulator.

In the EMULATOR, when asked to gives new learning signals, it means playing nominal sounds (the doorbell) so that the model can be retrained.

Then go to the COMPILATION STEP:

• Check the box "Include knowledge from benchmark" at the bottom of the page
• Click compile, fill the little form and save your AI library (.zip).

Here we use what was learned during the benchmark. You could choose to not include the knowledge and instead retrain the model directly on the microcontroller. More on that here:

https://wiki.st.com/stm32mcu/wiki/AI:NanoEdge_AI_Library_for_anomaly_detection_(AD)

Now that we have the anomaly detection library, we need to add it to our Arduino code:

• Open the .zip obtained, there is an Arduino folder containing another zip
• Import the library in Arduino IDE: Sketch > Include library > Add .ZIP library... and select the .zip in the Arduino folder

IF YOU ALREADY USE A NANOEDGE AI LIBRARY IN ARDUINO IDE:

go to document/arduino/library and delete the nanoedge one. Then follow the instruction above to import the new library.

IMPORTANT:

If you get an error because of RAM, it may be because of the library in NanoEdge. Go back to the VALIDATION STEP in NanoEdge and select a smaller library (click on the crown on the right), then compile it and replace it in Arduino IDE.

To use NanoEdge, it is pretty easy:

1. Use neai_anomalydetection_init() to check if everything is fine
2. include the knowledge for the model to be "trained"
3. run detection with neai_anomalydetection_detect

if the similarity is 100 it means that we have a signal 100% similar to the nominal data (doorbell sound) if it is 0, we have 0% chance of being nominal data.

So we put a threshold at 90 to print that we are detecting a doorbell sound.

/* Setup function ----------------------------------------------------------*/
void setup() {
  Serial.begin(115200);
  delay(10);


  /* Initialize NanoEdgeAI AI */
  neai_code = neai_anomalydetection_init();
  if (neai_code != NEAI_OK) {
    Serial.print("Not supported board.\n");
  } else {
    neai_anomalydetection_knowledge(knowledge);
  }
}


/* Infinite loop ----------------------------------------------------------*/
void loop() {
  if (analogRead(A0) > 400) {
    get_microphone_data();
    neai_anomalydetection_detect(neai_buffer, &similarity);
    Serial.println(similarity);
    if (similarity > 90){
      Serial.println("Doorbell ringing!");
      //do the stuff you want
    }
  }
}

The complete code it attached

Now that you can detect a doorbell ringing, it is up to you to continue, here are some ideas:

• Use the WiFi of the board to send a message or an email when a detection is made: https://www.youtube.com/watch?v=KwFyCGYuV6Q
• Create a dashboard hosted on your server and send update using the WiFi: https://www.youtube.com/watch?v=32VcKyI0dio

Thank you for reading :)