Falangafone - Use Gestures to Control Your Music!

At HackPrinceton 2014, my teammate and I built a robust, hand movement-controlled music system. We used the Leap Motion and its developer API to capture hand and finger data, and we utilized the Python language and the Pyo library for Python to process audio files in a variety of ways with the data received from the Leap Motion sensors. For example, each finger represented a different equalizer; by bending each of his or her fingers downwards and upwards, the user could adjust the values of these filters. We also used intuitive gestures to modify audio volume and speed / pitch. Falangafone was ultimately packaged as a Flask web application that allows users to easily select songs, profiles, and even visualize data from the equalizer!

We took it a step further, however, and used an Electric Imp to interface with Falangafone. Using a toggle switch, we created an play / stop button for the application, and also programmed several LEDs to flash based on the music.

This project was a great opportunity to learn about python, Flask, pyo, and electric imp. Huge thanks to the HackPrinceton team for providing participants with such great facilities to make projects such as this one.

Parts

• Electric Imp SD Card (1) - Used to communicate with the April Board over Wi-Fi
• April Development Board (1) - Used to turn on LEDs and power the switch.
• Mini-USB Cable (1) - Used to connect the April board to our laptop
• Breadboard (1) - Used to arrange the circuit
• Momentary Push Button (1) - Used to send HTTP GET queries to our web application
• LEDs (5) - For decorative effect.
• 330Ω resistors (5) - For the LEDs.
• Leap Motion - To track hand and finger movements.

Tools

• Wire Stripper
• Creativity =)

Software

• Python - The language the application is written in
• Pyo Library - used to perform signal processing operations in Python. Use the universal installer on the website.
• Leap Motion SDK - used to read hand and finger data from the Leap Motion. Download and run the installer on the website.
• Flask- a Python framework for web applications. Install with pip using 'sudo pip install Flask'.
• ElectricImp IDE- used to program the Electric Imp.
• ngrok - used to expose a dev version of our application to the internet and Electric Imp. Install with the link on the website.

Prepare the Electric Imp

To prepare the Electric Imp for use, we inserted the card into the SD card reader of the April board. Plug the Electric Imp into the a laptop via the micro-USB cable. The Electric Imp should blink red, signifying that it is on but not connected. After awhile, it will shut off the blinking LED to save energy. Before using BlinkUp to connect the Imp to WiFi, ensure that the jumpers are on the USB selection (if you are powering the Imp via usb).

Connect the Electric Imp to WiFi

To use BlinkUp, steadily hold the end of the Electric Imp to the phone screen as the flashing occurs. At the end, the LED in the Electric Imp should blink green, indicating that it has successfully connected to the internet.

Assembling the components on the breadboard

Next, insert the Electric Imp into the breadboard; place it at the strip closest to the + and -, to maximize the amount of space allotted to the electrical components.

To connect the switch, we inserted a wire leading to the positive switch terminal to a pin on the April (in this case, pin 9). Although it doesn't matter what pin you use, pin 9 is recommended due to its proximity to the ground (GND); we will save the other pins for LEDs. Proceed to connect a wire linked to the negative terminal of the switch to April's ground (GND).

After connecting the switch, we add the LEDs and their resistors. Connect the positive LED pins (generally the positive side is slightly longer than the negative side) with the rest of the pins on the April (in our case, pins 1, 2, 5, 7, and 8). Insert the negative end across the slit in the middle of the breadboard.

Add the 330 ohm resistors next (it doesn't matter which end goes where with resistors, unlike LEDs). For each LED, connect one leg of the resistor and insert the other resistor leg into ground (blue - row).

Lastly, we connect ground for the entire April board. Use a wire and connect one end to the GND pin (right above the VIN pin next to the USB and Battery jumpers) to the ground terminal (blue - row).

This tutorial assumes you have Python installed; if not, go ahead and install it.

Next, get the Falangafone code from our GitHub repo and get the Electric Imp code from this Gist.

With these two snippets in hand, we will combine the hardware and software sections.

Install prerequisite packages

At this point, make sure that you have installed all the packages and programs specified in Step 1. They will all be used together now.

Test the Falangafone Flask App

Utilizing the terminal (mac and linux) or the Windows Command Prompt, run the Falangafone Flask App by typing and submitting 'python LeapMusic.py'. Using your browser, navigate to 'localhost:5000' (without the single quotes). You should be greeted by the Falangafone index page!

Use ngrok to expose Falangafone to the Internet!

Using the terminal (mac and linux) or the Windows Command Prompt, navigate to the folder where you downloaded ngrok. Run './ngrok 5000' (without the single quotes) in the command prompt. Doing this instructs ngrok to mirror anything at that port and automatically generate a public URL for you (so you don't have to fiddle with firewalls and ports). If the previous command was successful, ngrok will have printed out a URL labeled 'forwarding' directly under the 'version'. This URL will begin with http:// . Save this URL; we'll need to use it to interface with the Electric Imp.

Upload the ElectricImp code
Copy the code from the GitHub Gist's Agent file into the ElectricImp IDE and from the Gist's Device file into the ElectricImp IDE's Device code input area. Make sure you replace the ngrok URL in the agent code with your own, and modify the pins you used if necessary in the device code. Also, replace the ngrok URL in main.js, located in the 'static' folder of the Falangafone project. Check the code, then build and run it. Your Imp should now properly be configured to work with Falangafone.

Test Falangafone!

At this point, navigate to your ngrok URL in your browser; you should be greeted with the same page you saw earlier at 'localhost:5000'. Connect your Leap Motion and ensure that it is reading data. To begin playing a song, place some '.aiff' and '.flac' files into the 'audio' directory of the Falangafone flask application. Then, throw the switch on your Electric Imp! If all went well, music is now playing, and you are interacting with it in ways you have never experienced before. Swipe your hands left and right and watch the binary counter change. Move your hands up and down to change the volume, wiggle your fingers to see the bandpass filters, and turn your hands inwards and outwards to modify the speed.