Synesthesia Player Piano

This player piano plays music using a color wheel and a camera! Images are captured by the camera, processed, and interpreted as musical notes. Right now it is programmed to play nursery rhymes that can be changed by uploading a new arduino sketch which specifies which notes should be played ( Which colors should be shown to the camera ).

The onboard Linux PC starts a service at bootup that takes a picture with a camera, computes a function of the RGB values in the image, and outputs a sound based on the return value of the function. Before playing the sound it sends a signal telling an Arduino to rotate a servo, presenting the color corresponding to the next note to be played. Then the loop starts again, a picture is taken, the color wheel is rotated, and a sound is played ad infinitum.

This project is a fun way to integrate music, modern Linux systems programming, micro-controller programming, inkscape doodling, laser cutting, and re-purposing some junk cardboard boxes.

All of the code and inkscape SVGs are available on github at:

https://github.com/melvyniandrag/pyMusic

Materials Used:

1. Blank white labels
2. Cardboard box
3. Webcam
4. Crayons
5. Arduino
6. Beagleboard-xM
7. Wires
8. Serial Cable
9. Basic knowledge of image processing and music

On-board you will need a small Linux-capable computer such as a Raspberry Pi. I used an old BeagleBoard-xM because I happened to have it laying around and I already had an operating system on the SD card. You can use whatever, so long as it can handle a webcam, can run OpenCV ( the computer vision library used to take and process images of the color wheel ), and has an audio out jack.

I set up the operating system to run the python code that captures images at bootup following a systemd tutorial. For the script to run you will need to sudo apt-get install python-pyaudio python-opencv to get the indicated libraries. You'll also need to pip install numpy pyserial to get those libraries. pyaudio is used to play sounds, opencv is used for image taking and processing, numpy is also used for image processing, and pyserial is needed for communicating with the Arduino.

The Arduino is used for rotating the servo motor on which the color wheel is mounted. If you had a Raspberry Pi you could make use of the GPIO pins on there and leave out the Arduino, but you'd need to modify the Python code.

In the images you can see the technological guts of the operation and a couple of code snippets.

I made the piano out of cardboard from some old boxes I had. I have the good fortune of living near a makerspace that has a laser cutter. For a whole year I've been wondering what the heck one would want a laser cutter for, and it turns out I finally found a use for it.

I measured the computer, arduino, and servo so I knew how much space I'd need, drew the parts of the piano in inkscape, and cut them out on the laser cutter. ( Edit: I ran out of cardboard to make the enclosure hold the computer and arduino and so I didnt put them in. They are on the outside of the case. Doesn't matter, could have gotten another box from somewhere, but it didn't matter in the end.)

The pieces were put together with epoxy.

Here are some images of the piano being assembled and colored. The printer was out of ink so I couldn't print the colors onto the white labels as I had planned. So i used my daughter's crayons. And I couldnt find an xacto knife to cut the label to the shape of the cardboard, so I just used a steak knife and put something protective on the kitchen counter. Use what you got!

A challenging portion of this project was getting the proper RGB values to be read by the camera. The webcam I have has some incredibly bright LEDs on it that I don't know how to turn off in Linux. I think there is a Windows driver that allows you to toggle them on and off. I experimented with taping the LEDs with electrical tape and leaving them on. I also tested the software for robustness by not only showing it colored paper, but also writing a little website that shows colors.

The sounds produced by the computer sometimes have pops and crackling in them, and this comes from the fact that I have very little experience in Linux audio programming. The pyaudio library spits out a whole bunch of complaints about JACK not being installed on the PC, but by default JACK doesn't want to run headless (without a gui desktop and a monitor). This is a programming task that I thought was a refinement, but wasn't necessary to express the essence of the player piano I had in mind.

I had considered using a hand crank to change the colors, and perhaps to have them on a rolling pin kind of thing, but that introduces its own complications. The servo is a bit noisy, but works fine for now.

The player piano is currently only programmed to play 4 notes. This is because the servo only moves 180 degrees, most nursery rhymes can be played with only 3-4 notes, and I didn't want the color wheel to get cluttered with color regions, possibly confusing the camera. So through a combination of musical luck and technical constraints, 4 notes is fine.