Melodically - Play With Music!

Melodically is an engaging activity for children that combines colors, music and creativity.

The idea is to allow children to create a melody by placing colored circles on the floor, each corresponding to a specific note.

The project tries to address the need to provide children with interactive and stimulating learning experiences.

It’s a creative and innovative way for kids to learn about the connection between visuals and sound.

How does it work?

The idea is to have a Raspberry placed on the ceiling with a camera pointing down to the floor. In the demo you'll see that for practicality I placed it on top of a microphone stand.

The user will place some colored circles on the floor in different lines under the camera view. Each color is associated to a specific note in the notation Do Re Mi Fa Sol La Si (see last image above).

The user has the possibility to start the experience by pushing a button or using the web interface. Once the experience has started, the program will detect and play the notes according to the line and order they belong to. The distance between the circles defines the pause between the notes. This will form a melody based on the notes that the user placed.

Guide to this instructable

This instructable describe the development and the set up processes of the project.

Steps 1-3 and 8-12 focus on the installation of the project, and it's aimed at those who simply want to reproduce the project..

Steps 4-7 focus on the development, and it's aimed at those who want to go into the details of implementation or improve the project.

Hardware requirements:

• Raspberry Pi 3B+
• 16/32GB Micro SD card
• 5V 3A micro-USB charger
• 1 Raspberry Pi camera (or the official one)
• USB Speakers with jack audio
• 2 long Wires (or one ethernet cable), better with female pin connectors on one side
• 1 Push Button

Source code is available in the following GitHub repo:

https://github.com/loris-bruno/melodically

The first step is to install the Raspberry operating system on the raspberry. For this, we use the Raspberry pi imager.

In the imager we can select the Raspberry Pi OS (32-BIT) operating system, choose the micro-SD storage and write the OS.

The SD-card is now ready to be placed in the Raspberry slot.

The project has been developen in python.

The required libraries are installed through the terminal using pip.

First of all we have to make sure everything is up to date, by running the following command:

sudo apt-get update

Here are the commands that need to be executed in order to install the corresponding libraries:

OpenCV (following this guide)

This is the main computer vision library.

Prerequisites:

sudo apt-get install build-essential cmake pkg-config libjpeg-dev libtiff5-dev libjasper-dev libpng-dev libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libx264-dev libfontconfig1-dev libcairo2-dev libgdk-pixbuf2.0-dev libpango1.0-dev libgtk2.0-dev libgtk-3-dev libatlas-base-dev gfortran libhdf5-dev libhdf5-serial-dev libhdf5-103 python3-pyqt5 python3-dev -y

Library:

pip install opencv-python==4.5.3.56

Imutils

This library is used to get the detected contours.

pip install imutils

Pygame

This is a library to develop games in python, but in this project we will only use it to play the melody.

pip install pygame

Flask and flask-socketio

This libraries are used to develop the web server.

pip install flask
pip install flask-socketio

By default, the raspberry camera interface is not enabled.

We can activate it by typing the following command in the terminal:

sudo raspi-config

The interface shown in the attached image appears. From here we can go to Interface Options and press enter.

Finally we can enable the legacy camera.

The camera module is developed in the file camModule.py.

The purpose of this module is to offer the capture_camera(imageName) function that allows to take a picture using the picamera library and save it with the given name.

In this module you can choose the resolution of the picture.

The musicModule.py file contains the main program, which allows to detect and identify the notes inside the image captured from the camera and play the melody.

The module offers a function make_song(imageName) that does all the process and plays the melody and a function setSounds(newSounds) that allows to change the set of sounds (instruments) to play the melody with.

The flow of the module is the following:

1. Define the color range for each note
2. Detect the color inside the image and form a mask (example of red color mask in the first image above)
3. Detect the circles in the mask
4. Assign the note name to the detected circles based on their color
5. Sort the notes according to their position
6. Compute the delay between notes based on the distance between their center points
7. Play the melody using the previously defined set of sounds
8. Save the result image showing the detected notes (result.jpg)

Colors

The most challenging part of this module is to define the color range in the HSV color model. In the module you will find the noteColors list containing the seven notes with the corresponding color in the HSV model:

# for each color to detect, define its spectrum in OpenCV HSV space [0...179]
noteColors = []
noteColors.append(NoteColor('do', 'red', [0, 100, 20], [10, 255, 255], [160,100,20], [179, 255, 255]))
noteColors.append(NoteColor('re', 'orange', [11, 100, 100], [20, 255, 255]))
noteColors.append(NoteColor('mi', 'yellow', [21, 100, 100], [40, 255, 255]))
noteColors.append(NoteColor('fa', 'green', [60, 100, 20], [80, 255, 255]))
noteColors.append(NoteColor('sol', 'cyan', [78, 100, 100], [100, 255, 255]))
noteColors.append(NoteColor('la', 'blue', [100, 100, 10], [120, 255, 255]))
noteColors.append(NoteColor('si', 'pink', [120, 10, 100], [170, 255, 255]))

If you want to set your own colors for the notes, you can convert the RGB values to HSV by following this opencv guide in the section "How to find HSV values to track?".

⚠ The range of a color is influenced by the ambient lighting, the camera quality and other factors so the detection might not be accurate in some situations.

Performances

Each color is processed on a separate thread, speeding up the computer vision process. In fact, without threads on the Raspberry Pi 3B+ the program was taking about 4.1 seconds to detect the notes, while with threads the detection is done in parallel and it lowered the times down to 2.4 seconds.

Sounds

The sets of sounds are located in the sounds folder. For each set there has to be a folder with the name of the instrument (e.g.: sounds/piano/) and inside it the audio files in the wav format. The audio files have to be the following:

• do.wav
• re.wav
• mi.wav
• fa.wav
• sol.wav
• la.wav
• si.wav

If you want to add more instruments, just place another folder in sounds, as described before. It will automatically be added.

The gpio.py file contains the logic for the push button. Since we want to connect the button through the GPIO pins the module uses the RPi.GPIO library, to continuously check the button state. The chosen GPIO pin is the number 16, and it's set with the integrated pull up resistor.

The program is running on a thread, so that is can run in parallel with the web server. The thread takes as parameter the function to be called in order to start the experience.

The webServer.py file is the starting module. It contains the web server and the initialization of the other modules.

The web server is done using Flask, and the routes are the following:

• / : get the index.html page which is stored in the templates folder
• /setsounds : set the name of the instrument you want to play the melody with
• /start : start the experience

The server will be running on port 5000.

Sockets are used to update the web interface in real time, so that the user can have direct feedback on the experience status (running or not) and the detected notes on the last experience.

When the server gets the start request, the camera module will take a picture and the music module will process it and play the melody.

⚠ keep debug=False or else picamera won't work (see https://raspberrypi.stackexchange.com/a/67256)

To print the notes I created a pdf file with the colored circles based on the definition in the introduction. You can print this file many times to have many notes to play with.

Button

The button needs to be plugged in through the GPIO pins. Since I didn't have any long single wires, i cut open the ends of an ethernet cable and connected two wires to the button pins and the other ends to the Raspberry GPIO pins.

As shown in the pictures, the wires are plugged in the pins 36 which corresponds to GPIO 16, and 39 which corresponds to the Ground.

Since it's just a button, it doesn't matter which end is plugged to which GPIO pin.

Camera

The camera needs to be plugged in the dedicated Raspberry slot and put under the raspberry to make it facing down to the floor (as in the 4th image above).

Speakers

The speakers need to be plugged with the audio jack and the USB port for power.

On the Raspberry, clone the GitHub repository by running the following command in the terminal:

git clone https://github.com/loris-bruno/melodically.git

Once you cloned the source code, you can enter the melodically folder (>cd melodically) and run the following command to start the program:

python webServer.py

The web interface is available at port 5000 of the raspberry. To connect from a device in the same network, access the web page using the Raspberry's ip address: http://<ip>:5000/.

To get the Raspberry's ip address, we can run the following command:

ifconfig

As shown in the image, we will find the ip address near the keyword "inet".

From this interface you can select the instrument you want to play the melody with and start the experience.

You will also see the list of the detected notes once the experience has ended.

The project is now fully set up! You can now experiment different melodies by putting the colored circles on the floor and listen to your creations!