How to Play Flappy Bird Like Game With Your Voice!

Welcome to this tutorial on creating a Flappy Bird-like game with an Arduino R4 WIFI board. In this guide, we'll cover hardware setup and everything related to the code so that you can reproduce it.

Here are the main parts of this project:

• Use the LED matrix
• Use a microphone
• Automatically classify sounds to play
• Handle multiple loop()
• Handling a player moving up and down
• Handling a wall with a hole moving toward the player and collision
• Keep track of the score
• Augment the difficulty as the game progress

The steps 1, 7 and 8 are required to reproduce the game. The other steps are here to explain each part of the code.

Hardware:

• Arduino Uno R4 WIFI
• Max4466 microphone
• A micro-USB cable to connect the Arduino board to your desktop machine

Software:

• Arduino IDE: To develop the code
• NanoEdge AI Studio: To detect the sound class automatically

First, we need to connect the microphone to the Arduino board.

Use jumper wires to connect:

• OUT (mic) to A0 (board)
• GND to one of the GND on the board
• VCC to 3.3v

Make sure that you have a USB data cable connecting the board to the pc.

In Arduino IDE:

Make sure you selected the right COM port: Tools > Port and select the right one.

Select the right board:

• Tools > Boards > Arduino Renesas UNO R4 boards > Arduino UNO R4 WIFI
• If you don't find it, click on Tools > Boards > Boards Manager..., look for the UNO R4 and install the package

Select the needed libraries:

• Sketch > Include Library, we need to include the following libraries:
• ArduinoGraphics
• Arduino_LED_Matrix
• Wire
• Scheduler
• You can also download libraries that you don't have: Sketch > Include Library > Manage libraries...
• Later in this tutorial (Step 8) we will also need to add the NanoEdge AI Library to process the sound on its own instead of doing it manually.

To get sound from the microphone, we don't need much:

We need to define which pin we use and call the function analogRead()

int const AMP_PIN = A0;       // Preamp output pin connected to A0

void setup() {
  ...
}
void loop() {
  ...
  /* Get a sound sample */
  static uint16_t sample = 0; //stock values
  sample = analogRead(AMP_PIN);
  ...
}

This example showcase how to get a single value from the microphone, but in this project, we will use buffers of samples. More on that in step 7.

To play the game, we use the LED matrix on the board.

Basically we have an array representing the matrix where we put 0 (LED off) and 1 (LED on).

Here is a simple example of what is required to use it:

/* Libraries needed */
#include "ArduinoGraphics.h"
#include "Arduino_LED_Matrix.h"

/* Defines */
#define HEIGHT            8
#define WIDTH             12

/* Object */
ArduinoLEDMatrix matrix;

/* Declare matrix to display */
byte frame[8][12] = {
  { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 },
  { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 },
  { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 },
  { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 },
  { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 },
  { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 },
  { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 },
  { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 }
};

void setup() {
  ...
  /* requiered to use the LED matrix */
  matrix.begin();
  ...
}

void loop() {  
   ...
   /* Change value of the LED matrix */
   frame[some x][some y] = 1
   /* Update display when needed */
    matrix.renderBitmap(frame, HEIGHT, WIDTH);
   ...
}

Depending on what we want to display, there are multiple ways to do it.

In this case, we calculate the position of the player and the wall and update the matrix above in a while loop.

Here is the documentation: https://docs.arduino.cc/tutorials/uno-r4-wifi/led-matrix/

We also use the matrix to display a text at the beginning of the game, here is how we achieved it (read the documentation for more info):

void introduction_message()
{
  matrix.beginDraw();
  matrix.stroke(0xFFFFFFFF);
  matrix.textScrollSpeed(50);
  // add the text
  const char text[] = " Flappy bird! ";
  matrix.textFont(Font_5x7);
  matrix.beginText(0, 1, 0xFFFFFF);
  matrix.println(text);
  matrix.endText(SCROLL_LEFT);
  matrix.endDraw();
}

To have a better experience we use multiple loops:

• One that handles the generation of the wall, its movement, the collision with the player and the score
• A second one that handles the player position and the detection of sound to make it move

Two loops allow us to move the player almost independently from the wall, we can move it multiple times while the wall moves only once.

The documentation: https://www.arduino.cc/reference/en/libraries/scheduler/

IMPORTANT: if later you get an error saying that loop2 is declared, it may be because you don't have enough Flash memory left, you may need to get a smaller NanoEdge library.

To use that we only need to import the scheduler library and create a second loop in the setup:

/* Libraries needed */
#include <Scheduler.h>

void setup() {
  ...
  /* start a second loop */
  Scheduler.startLoop(loop2);
  ...
}

void loop() {  
   ...
   /* Code related to the wall */
   ...
}

void loop2() {  
   ...
   /* Code related the player */
   ...
}

Here is the code in the Wall loop:

void loop() {
  if (game_ongoing) {
    /* Clean the last column of the matrix */
    for (uint8_t y = 0; y < HEIGHT; y++) {
      frame[y][WIDTH - 1] = 0;
      frame[y][WIDTH - 2] = 0;
    }

    /* Set wall position on the matrix using random */
    wall_start_pix = random(0, 5);

    /* Move the wall through matrix */
    do {
      if (wall_move) {
        for (uint8_t y = 0; y < HEIGHT; y++) {
          frame[y][wall_pos_x] = (y >= wall_start_pix && y < wall_start_pix + wall_size) ?  0 : 1;
          if (wall_pos_x > 1) {
            frame[y][wall_pos_x - 2] = 0;
          }
        }
        wall_pos_x++;
      }
      wall_move = !wall_move;

      /* Update display */
      matrix.renderBitmap(frame, HEIGHT, WIDTH);

      /* Adapt level */
      adapt_game_level(); // function in the full code

      /* Check if player touch the wall */
      if (frame[player_y][player_x] == frame[player_y][player_x - 1]) {
        game_ongoing = 0; //stop the game
        reset_global_variables(); //function in the full code
        return;
      }
    } while (wall_pos_x < WIDTH);
    /* Increment score counter */
    score++;
    /* Reset wall position */
    wall_pos_x = 0;
  }
}

The code is pretty self-explanatory but here is what is done:

1. The wall is moving from left to right. Because we are looping, we need to delete the wall when reaching the edge before making a new one (could be done at the end of the loop, but we chose to to it here)
2. We determine randomly the position of the hole in the wall. The wall is of size 3 and the height of the matrix is 8 so we have 5 possible starting points.
3. We move the wall to the right end of the screen. It means putting 1 in the new position and 0 in the last position of the wall
4. Update the matrix
5. Increase the difficulty based on the score
6. Check for collisions and end the game + display the score if needed

Here is the loop of the player:

}void loop2() {
  if (game_ongoing) {
    /* make a classification only if we detect a sound */
    if (analogRead(AMP_PIN) > 400) {
      get_microphone_data(); //function in the full code
      neai_classification(neai_buffer, output_class_buffer, &id_class);
      /* Player next movement based on class detected */
      if (id_class == 2) {
        player_move = -1; //up
      }
      else if (id_class == 3) {
        player_move = 1; //down
      }
    }
    else {
      //We don't want to repeat the same movement until we detect something else
      id_class = 0;
      player_move = 0;
    }
    //move the player
    move_player();

    /* Clean neai buffer */
    memset(neai_buffer, 0.0, AXIS * SENSOR_SAMPLES * sizeof(float));
  }
  delay(10);
}

What we do here is also pretty simple:

1. If sound is detected (400 is the value I get when nothing is happening), we do something
2. We collect sound from the microphone
3. We classify that sound: either meaning the move the player up or down (see the next step)
4. Based on the class, we move the player accordingly

To play the game, we make two sounds, one for up, one for down.

To classify the sounds, we could have done it manually. I am not sure how, but if we compute the FFT and set thresholds it should be possible.

Instead, we used NanoEdge AI Studio to do it automatically and include a very small AI model in our code.

• Install NanoEdge AI Studio: https://stm32ai.st.com/download-nanoedgeai/
• Create N class classification project
• Select microphone 1 axis as a sensor
• Select Arduino UNO R4 WIFI as target
• Collect data:
• Flash code attached in this step with Arduino IDE
• In NanoEdge, in signals tab, click add Signal, then serial
• First collect multiple examples of sound to play "up" (around 100 examples)
• Then multiple examples of another sound to play "down"
• Launch a benchmark and obtain a good accuracy
• Compile the library

Step-by-step tutorial for more details on how to use NanoEdge AI Studio and Arduino: https://wiki.st.com/stm32mcu/wiki/AI:How_to_create_Arduino_Rock-Paper-Scissors_game_using_NanoEdge_AI_Studio

Note:

The code attached only contains a function to collect downsampled buffers of sound and send them via serial.

We get a downsampled buffer because the base frequency of acquisition is pretty fast and so we would get a buffer representing few milliseconds. By downsampling it, it represents a little bit more time in real life.

To do it we simply read all the values that the microphone sends, but we only keep 1/8 values.

Now that we have the classification library, we need to add it to our Arduino code:

• Open the .zip obtained, there is an Arduino folder containing another zip
• Import the library in Arduino IDE: Sketch > Include library > Add .ZIP library... and select the .zip in the Arduino folder

Then in our code, to use the NanoEdge classification, we simply need some code like in this example:

• We include the NanoEdge library
• We initialize the library
• We collect data from the microphone
• We do the classification

#include "NanoEdgeAI.h"
#include "knowledge.h"

/* NanoEdgeAI variables part */
uint8_t neai_code = 0;
uint16_t id_class = 0; // Point to id class (see argument of neai_classification fct)
float output_class_buffer[CLASS_NUMBER]; // Buffer of class probabilities
const char *id2class[CLASS_NUMBER + 1] = { // Buffer for mapping class id to class name
  "unknown",
  "up",
  "down",
};
static float neai_buffer[SENSOR_SAMPLES * AXIS] = {0.0};

void setup() {
  ...
  /* initialize NanoEdge Library */
  neai_code = neai_classification_init(knowledge);
  if (neai_code != NEAI_OK) {
    Serial.print("Not supported board.\n");
  }
  ...
}

void loop() {  
   ...
   /* make a classification */
   get_microphone_data();
   neai_classification(neai_buffer, output_class_buffer, &id_class);
   /* then depending on the class in id_class, we play up, down or wathever */
   ...
}

Warning:

You need to check that the id2class variable in the Arduino code is the same than in NanoEdgeAI.h file from the library we imported earlier:

• in document/arduino/libraries/nanoedge/src/NanoEdgeAI.h (at the end of the file)

const char *id2class[CLASS_NUMBER + 1] = { 
  "unknown",
  "up",
  "down",
};

The entire code for the project is attached.

You need to follow the step 1 and the steps 7 & 8 to make it work!

Thank you for reading :)