KeyGlove: Play Piano Anywhere With Smart Gloves

In this guide, you'll learn how to create a pair of gloves that transform hand movements into piano sounds. Using different sensors, these gloves can distinguish between black and white keys, simulating piano playing without the need for an actual instrument. All you need is a flat surface to play on.

Whether you're a music enthusiast, a tech hobbyist, or an educator looking for innovative ways to teach music, this project offers a unique blend of technology and creativity. By following this guide, you'll gain hands-on experience with sensor technology, microcontroller programming, and wearable electronics, culminating in a portable piano you can play anywhere.

Check out my GitHub repository to access the code required to complete this project!

Materials:

• Raspberry Pi (+SD card and power supply)
• Raspberry Pi t-cobbler
• ESP32
• 4x MCP3008 ADC
• 10x pressure sensors (these will go on the fingertips, so choose an appropiate size)
• 10x flex sensors (these will go along the back of eacht finger, so choose them appropriately as well)
• speaker (I used a 1W 4Ohm speaker, but a simple piezo buzzer will also work fine and won't need an amplifier)
• LM386 integrated amplifier module
• LCD1602-display
• 3.3V / 5V power supply
• 1x button
• 2x potentiometer
• general electronics: resistors, jumper cables, breadboards, ...
• soldering wire
• pair of gloves
• fabric and thread for securing sensors to gloves
• 600x450 mm MDF wood sheet (or any other material of your choosing, this will be used to create the housing of the central module)

Tools:

• soldering iron
• wire cutters/strippers
• sewing needle and thread
• lasercutter
• hot air gun
• computer for programming
• software:
• Raspberri Pi Imager or Win32DiskImager
• PuTTy
• MySQL Workbench
• Visual Studio Code
• Arduino IDE

Raspberry Pi:

Set Up Raspberry Pi: Connect your Raspberry Pi to the T-cobbler and place it onto a breadboard. On the opposite side of the breadboard, connect your 3.3V/5V power supply. Ensure the +/- pins are correctly aligned with the breadboard's power rails.

Connect the Button: Attach a button to one of the GPIO pins using a 470Ohm resistor and connect the other end to the ground.

Sensors:

Flex and Pressure Sensors: Each glove has two groups of five sensors (flex sensors and pressure sensors), making a total of four groups, each with its own MCP3008 ADC.

Connect Sensors to ADC: For each sensor, create a voltage divider: use a 100kOhm resistor for each flex sensor and a 10kOhm resistor for each pressure sensor.

ADC Connections: Connect the appropriate pins of the ADC to the ground and 3.3V power supply. Connect the MOSI, MISO, and CLK pins, and then connect these along with the CS/CE pins to the T-cobbler.

Pro Tip: Trim your resistor wires so they fit snugly onto the breadboard to prevent accidental shorts when the final product is moved. Use stiff wires where possible to avoid visual clutter and ensure sturdiness. (image)

ESP32:

Connect all eight outgoing pins together, ensuring each runs through a 470Ohm resistor to prevent shorting your ESP32. Connect this to a potentiometer (useful for tuning the speaker during development), then connect the potentiometer's output to the amplifier.

Attach the speaker to the amplifier and connect all necessary ground and VCC pins.

Connect the ESP32 to the 5V power source and ground and link the RX/TX pins to the TX/RX pins on the T-cobbler.

LCD Display:

LCD Connections: Connect the LCD pins as follows:

• VSS, RW, and LED- to GND
• VDD and LED+ to +5V
• V0 to a potentiometer output
• RS, E, and DB0-7 to GPIO pins on the Raspberry Pi

Ensure all components are connected according to the provided example. You are now ready to get started on the software!

Install Raspbian OS:

1. Download Raspbian: Obtain the latest Raspbian image from the official Raspberry Pi website.
2. Write the Image: Use a tool like Raspberry Pi Imager or Win32DiskImager to write the Raspbian image to your microSD card.
3. Configure SSH: Ensure that SSH is enabled in the configuration settings. This can typically be done within the imaging software.
4. Insert and Boot: Insert the microSD card into your Raspberry Pi, power it on, and let it boot up.

Connect via SSH:

Using an SSH client like PuTTY, connect to your Raspberry Pi with the hostname. Complete the initial setup and install the necessary packages:

sudo apt install mariadb-server
sudo apt install python3-pip
pip3 install mysql-connector-python

Network configuration:

Set Up Static Ethernet Connection:

• Configure your Raspberry Pi to have a static IP address over Ethernet.
• Ensure your workstation's Ethernet settings are compatible with the Raspberry Pi's network settings.

Configure Wi-Fi:

• Connect to your home Wi-Fi or the network where you'll be developing.
• Edit the wpa_supplicant.conf file:

bash
Copy code
sudo nano /etc/wpa_supplicant/wpa_supplicant.conf

• Add your network details:

plaintext
Copy code
network={
    ssid="your_SSID"
    psk="your_password"
}

• Save and exit the file, then restart the Wi-Fi interface:

bash
Copy code
sudo wpa_cli -i wlan0 reconfigure

You are all set to start coding now!

Open MySQL Workbench and connect to your Raspberry Pi using the static IP address you've configured. Create and the database that will store the data for this project and fill it with dummy-data. You can find the SQL script to create the database in my GitHub repository. Feel free to make any desired adjustments!

Setup

Connect to your Raspberry Pi:

Use Visual Studio Code to connect to your Raspberry Pi over SSH. This allows you to directly edit your code with a user-friendly GUI and run it conveniently during development.

Prepare the virtual environment:

Open a terminal and set up a virtual environment, and install the required packages:

python3 -m venv <venv_name>
pip install -r ./requirements.txt

Configure Database Connection:

Open the config.py file and add your database credentials to connect app.py to your database:

[connector_python]
user = YOUR_USERNAME_HERE
host = 127.0.0.1
port = 3306
password = YOUR_PASSWORD_HERE
database = YOUR_DATABASE_HERE

[application_config]
driver = 'SQL Server'

app.py

Open app.py, which serves as your backend script. Below is a brief overview of the essential components needed in app.py. For detailed code, please refer to my GitHub repository.

Hardware setup

Initialize hardware components:

• Set up the power-off button.
• Configure all sensors using the MCP3008 class for each group of sensors.
• Define the keymap for the desired piano keys.
• Establish serial communication.
• Set up the display.
• Initialize various other variables needed throughout the script.

Define functions

Write functions to handle interactions with all your hardware components, ensuring smooth operation and data flow.

Thread

Sensor Data Thread:

• Create a thread to read sensor data continuously in the background.
• Ensure it communicates with the ESP32 when key configurations change to play the appropriate sound.
• Manage different situations using global variables to handle state and interactions effectively.

API

Implement API routes for the following events:

• Power down: Safely shut down the system.
• Recording management: Manage recording functionalities:
• Start recording.
• Stop recording.
• Save recording.
• Discard recording.
• Data fetching: Retrieve data from the database:
• Get recordings.
• Get sessions for both 'table' and 'chart' routes.
• 
  

Socketio

Set up SocketIO to manage real-time communication and updates:

• Broadcast key pressing events.
• Update recording statuses.

ESP32

Arduino IDE Setup:

• Write a script in the Arduino IDE for the ESP32 to handle key-indices received over serial communication.
• Utilize 8 separate PWM channels to manipulate the frequency and generate tones. Ensure the PWM channels are bound to the GPIO pins specified during the circuit construction.

By following these steps, you will establish a fully functional backend that seamlessly interacts with the hardware components and supports your KeyGlove project’s features.

Develop the website pages

Create the necessary HTML pages for your application, defining the structure and content for each page.

Apply design with a general stylesheet

Implement your design by writing a general stylesheet using CSS to ensure a consistent and visually appealing layout across all pages.

Write a general script for backend communication and other website functionalities

Use JavaScript to determine which HTML page is currently loaded, enabling page-specific functionalities and optimizations. These include:

• Listen to socket communication: receive information from the backend to display the activated keys or display the current recording status.
• Listen to UI actions: handle menu actions, the poweroff button (-> backend) and recording actions (-> backend).
• Fetch dataa from database if applicable:
• Make asynchronous requests to fetch data from the database to receive appropriate recording- or session-information as needed.Dynamically update the page content based on the retrieved data.

By following these steps, you'll create a robust and dynamic frontend that seamlessly interacts with the backend and provides an engaging user experience.

Deploy your code and set up a service to ensure your application runs on boot.

Display your frontend using Apache2

sudo -i
nano /etc/apache2/sites-available/000-default.conf

Navigate downwards to the line DocumentRoot /var/www/html and change it to DocumentRoot /home/user/<name_of_your_repo>/front

Save the file and restart Apache:

service apache2 restart

Configure the right to the root folder:

nano /etc/apache2/apache2.conf

Navigate downwards untill you find

<Directory />
    Options FollowSymLinks
    AllowOverride All
    Require all denied
</Directory>

and change it to

<Directory />
    Options Indexes FollowSymLinks Includes ExecCGI
    AllowOverride All
    Require all granted
</Directory>

Save the file. Configure permissions and restart Apache

sudo chmod o+x /home/user/<name_of_your_repo>
sudo chmod o+x /home/user/<name_of_your_repo>/front
service apache2 restart

Your website is now accesible thour the wlan ip address of you Raspberry Pi!

Configure automatic startup of your backend

Create the file myproject.service

nano myproject.service

and add the following contents

[Unit]
Description=ProjectOne Project
After=network.target
[Service]
ExecStart=/home/user/<name_of_your_repo>/<name_of_your_venv>/bin/python -u /home/user/<name_of_your_repo>/backend/app.py
WorkingDirectory=/home/user/<name_of_your_repo>/backend
StandardOutput=inherit
StandardError=inherit
Restart=always
User=user
CPUSchedulingPolicy=rr
CPUSchedulingPriority=99
[Install]
WantedBy=multi-user.target

Copy this file to this location:

sudo cp mijnproject.service /etc/systemd/system/mijnproject.service

You can now start and stop your file, check it's status and look at the logs using the following commands:

sudo systemctl start myproject.service
sudo systemctl stop myproject.service
sudo systemctl service  myproject status
sudo journalctl -u myproject

(an example of the logs can be found in the image)

If everything is working correctly, enable the service to start automatically whenever the Raspberry Pi is booted:

sudo systemctl enable myproject.service

Soldering

Sensor Wiring:

• Solder a long wire to one end of each sensor. This wire will connect to the ADC (refer to the yellow thread in the breadboard diagram).
• Solder a shorter wire (10-15cm) to the other end of the sensor.

Ground Connections:

• Connect the shorter wires together using clamps at the wrists. These represent the ground wire.
• With this setup, only one ground wire is needed from each hand, resulting in a total of 11 wires from each hand.

Sewing

Pressure Sensors:

• Sew the pressure sensors onto the fingertips of the gloves.
• Run the wires along the palm of the hand to the wrist.

Flex Sensors:

• Sew the flex sensors onto the back of the fingers. Attach the tip securely, ensuring there is enough room for the sensor to slide over the finger during movement.
• Run the wires along the back of the hand to the wrist.

Clamps and Bundling:

• Sew the clamps onto the gloves at the wrist.
• Bundle the ground cables that will run to the central module for better organization and to prevent tangling.

TIP: Get someone to wear the gloves while you sew to ensure accurate positioning of the pressure sensors.

Create the enclosure box

Design and prepare the box

Use an online tool like MakerCase to design the enclosure box for your components. Ensure the design file is compatible with a laser cutter.

Place the MDF sheet in the laser cutter, load your design file, and start the cutting process.

After cutting, use a pencil to trace the outlines of your components on the box, marking where holes need to be drilled. Provide space for the following components:

• LCD display
• Speaker
• Button
• Sensor wires
• Breadboard power supply
• Raspberry Pi power supply
• Raspberry Pi ethernet port (optional, for easy access to the Pi if needed)

Drill and Shape Holes:

Drill the necessary holes and shape them using files and sandpaper to ensure a perfect fit for each component.

Attach components to the enclosure box

Attach the speaker and LCD display using bolts and nuts. If possible, attach the Raspberry Pi using bolts and nuts as well. If not, secure it with nails or another method.

Run the button’s connector wires through the top part of the enclosure.

Attach the breadboard to the inside of the bottom part of the enclosure using the provided sticky strips. Ensure the breadboard power supply lines up with the drilled hole.

Run the sensor wires through the designated spaces. Secure them to the inside walls of the box using glue, tape, or brackets to prevent them from dislodging components on the breadboard.

Final Assembly

After all components are securely attached, finalize the assembly of the enclosure box.

Congratulations!

Your project is now complete! Plug in the breadboard power cable and the Raspberry Pi power cable to see your project come to life. Feel free to share your thoughts on this project. Whether you plan on making it yourself or have been inspired to create something new, I'd love to hear about it!