Converting a Rescued Toy Into a MIDI Controller

In this Instructable, I will walk you through the process of converting a rescued noise-making children's toy into an actually useful musical instrument using MIDI! Take a moment to just glance over the titles of the steps in this Instructable and familiarize yourself with the general process, so you know what to expect when you're complete, and whether or not this Instructable is what you're looking for.

I'll help you pick out a good toy to rescue, and then guide you through the process I used to successfully hack all of the buttons and switches to make something really cool and useful. We'll rip out the old, useless guts of the toy and replace it with a cheap microcontroller that is capable of sending and receiving MIDI messages to a PC, which will do the actual sound synthesis for us. I'll discuss the ins and outs of how to do this using code, and hopefully help you improve your project-making skills along the way!

Example of finished MIDI controller

What is a MIDI controller?
The Musical Instrument Digital Interface (MIDI) protocol was developed by professional audio technicians as a way for their various electronic devices (drum machines, synthesizers, samplers and more) to communicate by sending messages back and forth. Messages like, “play this musical note” and “we’re now in the 3rd beat of the 2nd measure” are transmitted between devices to help keep all of the devices in sync and reduce the amount of work that people need to do.

Every musical device out there today can be classified as a specific type of device, such as a sequencer, synthesizer, sampler, controller or others. In this article, we will be converting a toy into a MIDI controller. A MIDI controller typically has no ability to produce any noise or manage musical notes into a piece of a music. Instead, it’s more like a keyboard or a mouse; they tell a more intelligent device about changes in it’s own systems (i.e. “user pressed the A button”).

You can learn a great deal more about the MIDI using a variety of resources. Since MIDI has been around for quite some time, there is a great deal of information available on the internet, such as:
- http://www.midi.org/
- http://en.wikipedia.org/wiki/MIDI
- http://home.roadrunner.com/~jgglatt/tutr/miditutr.htm
- http://www.zzounds.com/edu--learningmidi
- http://www.academictutorials.com/midi/

For personal reference, I ordered the following book and have been very happy with it: http://www.amazon.com/MIDI-Power-Comprehensive-Robert-Guerin/dp/1598630849/

Truthfully, once you gain the confidence to do so, you can convert nearly any old toy into a MIDI instrument! However, in this article I am going to rely on the assumption that you have never hacked a toy before, and therefore would prefer to start with a simple project. Try to choose a toy that is relatively simple, meaning has less than a dozen buttons. Obviously, the toy I have chosen for my own project is not super simple, but I can't help it! I just fell in love with the toy as soon as I saw it, and knew I wanted to rescue it, bring it home, and give it a new life! However, be advised that less complicated toys = less headache, especially if you've never done this before!

What to look for
- Relatively low number of buttons. Less than a dozen is a good idea.
- Very few or no sliders or knobs. You can utilize these in a MIDI controller, I just won't cover it here
- A big enough enclosure to accommodate a few extra wires and a Teensy microcontroller

Where to find a good toy
Don’t over-think this process too much; if its a toy that has electronics inside of it, you can convert it into a MIDI controller! If you’re flipping through pages of Google results or Amazon products trying to find the toy that is “just right”, stop it. The best place to get fantastic toys from is your local Goodwill, Salvation Army or other local thrift store, and are they only places I like to go to when looking for toys. Buying a brand new retail toy from Wal-Mart will only contribute to wasteful consumerism and put a hole in your wallet.

The toy guitar that I will be using for demonstration in this article was purchased from my local Salvation Army for $1.95! Thousands upon thousands of cheap toys are thrown out all the time by families all over the country, usually for trivial reasons like cracks or a broken button. Believe it or not, the most common cause for toys being thrown out is simply that the batteries are dead!

Introducing my toy guitar
I stumbled across this beauty one day at my local Salvation Army, and knew I had to have it as soon as I found it. In fact, I decided to do this article after I found this guitar, because I thought it would make such a cool MIDI instrument! The guitar features a large variety of pieces that can be pressed, spun, rocked and tilted. It even includes some neat lights and unusual features (like that gray wheel at the bottom – it spins!). When I brought it home, I quickly found out why it was discarded; there was a crack in the plastic of the fretboard, making the whole thing flex in odd ways. Furthermore, the batteries were dead. I applied super-glue to crack and removed the batteries, and the guitar was good as new!

The most important thing to understand about your toy is that it was most likely manufactured overseas using cheap processes. Over the years, these manufacturers have invented some really neat cost-saving techniques to implement many different kinds of interactions without relying on traditional parts. The downside to this is that when you crack open your toy, you may not immediately recognize how exactly the buttons you saw on the outside actually work. By and large, in my experience, these toys rely on a relatively small number of tried-and-true tricks to give the impression that much more is going on.

Buttons made from PCBs and silicone domes
In many bulk-manufactured devices today, buttons look nothing like the buttons you can buy from RadioShack. Instead, interlocking (but disconnected) traces are etched into custom-made printed circuit boards (PCBs) to create a normally-open (NO) type switch. An example of these interlocking patterns is shown in the fifth picture of this step. A silicone dome, housing a small piece of conductive material, is then fitted onto the PCB in such a way that when the dome is pushed down, the conductive material bridges the various interlocking traces on the PCB, thus completing the circuit and closing the switch.

By adjusting the height, size, materials and other properties of the silicone dome, manufacturers can alter the 'feel' of a button with ease. You are likely to find these all over your board, because they are quite cheap and useful.

Leaf spring switches
Another common kind of switch you might find, which provides a different kind of 'feel' and I imagine is a little more durable, is a leaf spring switch, as seen in the seventh picture of this step. If you have a strip of metal, it's thickness and material characteristics will determine how easy it is to bend. The right combination of this properties will result in a strip that feels somewhat 'springy', meaning it will bend a little bit, but will keep bouncing back to it's original form when released. If you place a small piece of metal at the point where the strip bends to, you can create a switch that springs open until you close it.

Understanding the web of traces
Each board should have some wires coming from it, which carry electrical signals to and from the board and other parts of your toy. Generally, you can easily turn the circuit board over and study the traces to figure out which wire goes to which component. It may help you to draw a picture of the circuit board, then make notes about which wires go where, and trace out the path of each signal around the board. Remove the silicone domes temporarily to make sure that you aren't missing any 'ninja' traces underneath them.

Take some time to make diagrams of each of your circuit boards and determine which of the board's wires are for ground and which are for signals, as well as what these signals represent. Generally, you should be able to identify at least one wire that is shared by several components - this is the ground wire. You should also see at least one individual signal wire per component.

We will take a look at what to do with these wires in Step 4, so keep those diagrams nearby!

Highly-specialized sound-making circuitry
In addition to the buttons and switches throughout your toy, you should also have a board which contains a small, round black dot. This is a highly-specialized chip that contains all of the sound samples and logic required to turn the signals from the various buttons into actual noises. Furthermore, you will be able to find a speaker of some sort somewhere in your toy. In the next step, we'll remove these components, so that we can put our own microcontroller in it's place.

If a toy makes noise, it’s a guarantee that there is a circuit board inside of it that all (or most) of the wiring goes to. The easiest way to identify this board is to first find the speaker in the toy. Simply follow the wires of the speaker back to the board they connect to, and you can be reasonably certain that you’ve found the board we’re looking for.

Another key identifying characteristic of the main sound-generation board is a small, oddly shaped black chip. It will look like someone dripped a small amount of black candle wax onto the board and let it form a small circular glob. This component will likely have many small copper traces coming out of it. This little chip actually contains all of the logic to interpret all of the various buttons, sliders and other components throughout the toy and make noise using the speaker. Talk about no-nonsense manufacturing!

So what can you do with this board? Well, not much. You may have some success exploring circuit bending techniques, but for the purposes of this Instructable, it's useless.

Cut each of the wires coming from this circuit board and remove it. Also take care to remove the speaker, as we won't be using it either.

Credit to David Storch for the photo, from https://wiki.brown.edu/confluence/display/mcm0750/Boot+with+Knob+and+Buttons

As discussed in Step 2, the vast majority of the components we will be working with in this Instructable are switches and buttons. However, there may be some confusion about what we will actually be doing with these, which is completely understandable. Recall that each circuit board in your toy should have at least one ground wire and one signal wire per button. We will be connecting each of the grounds together to form a common ground (Step 6), and then connecting each of the signal wires to a microcontroller (Step 11). However, only doing this will overlook one important fact - if the switches are open, will the microcontroller read 0 or 1?

Floating inputs
If you connect a normally-open switch to a digital device like a microcontroller, then attempt to read the state of that switch, you will actually receive a somewhat random value that you cannot predict. A digital device must be connected to either ground (to read "0") or some voltage like +5V (to read "1"), so if it is not connected to either, it cannot be sure what its connected to. This phenomenon is called floating, and is very important in the world of microcontroller circuits. We can solve this problem using a single resistor, either tied to +5V (pull-up) or ground (pull-down). Since we're already connecting one end of the switch to ground, we need to use a pull-up resistor.

What is a pull-up?
Take a look at the picture for this step, and you'll see a schematic of what a pull-up resistor and a switch look like together. The switch, SW1, is a button in your toy, which we are connecting to ground. The resistor, R1, links the other side of the switch to +5V, so the microcontroller will only ever see one or the other. However, using modern microcontrollers, we don't have to add a resistor to every single button in the toy - the microcontroller already has one built into each of it's pins for this exact purpose! Maybe I'm easily amused, but that's pretty cool.

At this point, you should have a sound-making toy that you have opened up, learned about and torn the sound-making circuitry out of. This means that you have completed the destructive phase of the project and are ready for the constructive / creative phase!

Identifying the components you want to use.
Begin by identifying everything that you want to use. I have found that the technique that works best for me is to pick up a piece of paper and draw a very simplified picture of what I see inside the toy, and try to identify which things I’d like to use, and which things I don’t want to use. These sketches are attached to this post, so check them out as you read this step.

Notice that at the bottom of each page, I am keeping track of how many inputs and outputs I have identified in my drawings. This will help you determine what microcontroller to buy later!

An important bit of information that should help you in identifying how your circuit boards work is to remember that every single board will have at least one ground wire and at least one signal wire for each component.

Special note about charlieplexing
For circuit boards that have many buttons on them, you may run into a technique that can be quite confusing for beginners called charlieplexing. Essentially, it’s a technique whereby several buttons share the same signal and/or ground lines. In this case, we must identify more than one ground line and connect one wire to each. Using a microcontroller, we can enable or disable these ground lines as we please, and thus figure out which button is sending information along a signal line. If you have never encountered this technique before, and find a circuit board that you think uses it, you can choose to either ignore that circuit board (and thus it’s buttons), or pick up another cheap toy.

Before soldering signal wires to your components, first solder all of the ground wires together, so that you have a common ground across the toy. There really isn't much to this step, just take all of the ground wires of your switches and buttons you have and solder them all together. The photo for this step shows a close-up of one part of my common ground - look for the thick black wire that travels from part to part.

Note about multiple grounds and charlieplexing
If you have one or more circuit boards that use multiple grounds (charlieplexing) and you DO want to use them, do not connect them together, but rather consider them signal wires, and thus leave them alone for now.

You may find that the wires connecting your various buttons, switches and circuit boards are quite flimsy and small. This is another cost-saving technique that many manufacturers opt to use, though it can make it frustrating for us. Like myself, you may have inadvertently pulled a wire or two out already!

Since these wires are quite frustrating to solder, strip and manipulate, you may decide to completely replace the wires with the hookup wire of your choice. Personally, I love the solid core wires that ethernet cable is made out of - once they are debraided and straightened out, they make for excellent hookup wires, and are quite cheap!

In the photos, you'll notice that I also labeled each of the wires and groups of wires using masking tape - it helps me keep track of which wire is which (since I have so many), and keeps me organized between hacking sessions.

Before things get any more complicated with the addition of a microcontroller, I recommend testing out each of your components to make sure that they behave the way you expect them to. You can do this a number of ways, and I encourage you to use whatever resources you have. I had an extra Teensy microcontroller laying around, so I decided to use it to test each of my wires, but you can use a regular Arduino, or even a simple analog circuit that turns on and off and LED on a breadboard.

This will help you identify simple problems with either your wiring, soldering or mapping without having to worry about whether or not the Arduino code or MIDI functionality is to blame.

Use the Arduino code attached to this sketch to test out each wire from the previous step individually.

1. Upload the sketch to your Arduino
2. Connect the toy's common ground to your Arduino's ground
3. Connect the Arduino pin coded into the sketch to a signal wire of a component
3. Open the Serial Monitor
4. Press the button on your toy and make sure the stream of numbers changes from 1 to 0

Help! One or more buttons do nothing!
First, double-check your wiring. Do the solder joints look good? Are the bare parts of any wires touching? Do any wires have any cuts or nicks in them? Are both ends of the wire firmly connected to their parts? Is your common ground connected to your Arduino? 

If everything checks out but the wire still doesn't work, try moving on to another wire, just to make sure that the code and PC connection are correct. If you can get at least one other wire working, you can be sure that your Arduino environment and USB connection are OK. If you absolutely cannot get it to work, you may have to just disconnect it and move on - sometimes fixing low-grade consumer toys is more trouble than its worth. Feel free to leave a comment on this step with any more tips or troubleshooting hints!

If you have experience in the world of electronics, you will know that microcontrollers are the components that act like small computers for you, carrying out your instructions and doing advanced things for you. You may also know that Arduino has taken the hobby electronics world by storm over the last few years, and been fantastically helpful for beginners and experts alike. Therefore, it shouldn’t be any surprise that we’re going to use the Arduino platform in this project! Of course, you can use whatever you want to get the job done, but that will go beyond the scope of what I can present to you in this article!

Choosing and acquiring your microcontroller
The main Arduino board that you can buy is wonderful for beginners to learn basic concepts about electronics and to prototype their ideas without much hassle. However, it can be quite a challenge to fit an entire Arduino board into your toy! Furthermore, it is presently not as easy as it should be to make your Arduino board act like a genuine MIDI device.

For my project, I chose to use the Arduino-compatible Teensy++ 2.0 from PJRC, and I am going to highly recommend that you use one as well. In addition to being extremely small, the Teensy series boasts the ability to emulate different types of USB devices, like keyboards, mice and (you guessed it) MIDI devices! In my opinion, these factors make Teensy the perfect choice for this kind of project, so go and get yourself one!

Setting up your Arduino environment to work with Teensy
To work with your Teensy in the Arduino IDE, you need to install a special add-on called Teensyduino. This adds a few new menu items to your "Tools" menu, and adds some really useful functionality to your environment. Read the Teensyduino page for installation instructions, and check out the various sections of the menu on the left to learn about all the cool things you can now do.

Planning out how your wires will connect to your microcontroller
Regardless of how complicated your toy is, it would be a good idea to make yet another diagram documenting which wire in your toy should connect to each pin of your microcontroller. PJRC supplies some really helpful diagrams showing all of the pins of the Teensy microcontrollers and their labels in Arduino. Print one out, or use it as reference to draw your own picture, and label the pins according to how you want to connect your wires.

My toy had a lot of inputs and outputs, so I wanted to keep very careful notes about which wire goes to what. Therefore, I created a simple spreadsheet to keep track of every wire, their labels, functions, Teensy connection and MIDI note value. You may not need to go to these lengths if your toy is relatively simply, but I wanted to share my process just in case.

Before soldering up the wires, I think it's a good idea to install a USB cable to connect your Teensy to the outside world. There are several ways to do this, but I chose to use a properly sized hole and a zip tie (see the attached pictures). If you'd like, you can permanently install a socket (like this one, offered by PJRC), though I didn't feel like spending the extra money. If you can work out a better way to do this than what I will show, feel free to do so!

The most important thing about adding a USB cable to your toy is to be very careful about the mechanical stresses it might come under. Always imagine what might happen in a worst-case scenario; maybe you step on the cable while you’re moving around, or it gets caught on something and yanks very hard on your toy. For this reason, I recommend securing placing the Teensy where you want it in the toy, then securing the USB cable to the case to provide enough slack to reach the Teensy. The more you can secure the USB cable to your toy’s enclosure, the less stress it will put on your Teensy. It’s much better to have the USB cable simply become unplugged than for the socket of your Teensy to snap off, or the board to crack!

For now, don't worry about affixing the Teensy to the case - you will be turning it over and moving it around as you solder wires to it in the next step.

Following the diagram or spreadsheet you generated in Step 8, you should be ready to actually solder up each wire to the microcontroller, and test it out with code. I highly recommend that you test each wire or group of wires using the procedure outlined in the next step right after you solder it up. This way, if you do discover a problem, you don't have to undo anything to identify and fix the problem. Just remember:

Start small, test often.

Pick a wire, or group of wires, and find a good way to route it that is not going to make it hard to close the toy back up again. Find empty spaces that don’t have anything in them, and try to avoid posts that are used for screws. Your two best friends in this process will be sticky tack and masking tape. Take a look at my process photos below to get an idea of how I went about laying out my wires and soldering them to the microcontroller.

NOTE: I recommend testing each pin as you solder it using the procedure in the next step!

Just like we tested out each component in Step 7, I recommend that you follow a similar procedure and test out each pin of your microcontroller once each wire is soldered up. This will again help you identify any problems that may have come up during soldering and wiring, and help you catch problems before things get any more complicated.

Download the Arduino sketch attached to this step and open it up. You'll notice it looks a bit more complicated than the code from Step 7, but it is for good reason. Firstly, the code includes a debouncing routine that is capable of checking the state of an arbitrary number of pins. Secondly, the code uses an array to hold all of the pin numbers, so that you only have to edit one variable whenever you want to add or remove pins from your toy, rather than diving into the entire sketch!

Debouncing
Debouncing may be necessary with certain kinds of buttons because of something called mechanical contact chatter, or “bouncing”. This occurs when the two metal pieces of a switch make contact, and then “bounce” off each other for a brief moment. This happens so quickly that you won’t notice it in the Serial Monitor or by touch, you would only really know about it from the problems it would cause, or by looking at the signal on an oscilloscope.

There are many techniques one can use to debouncing inputs, but I am using a technique outlined by Limor "ladyada" Fried here: http://www.adafruit.com/blog/2009/10/20/example-code-for-multi-button-checker-with-debouncing/

Using the sketch
I recommend testing one pin at a time, just to make troubleshooting easier. Update the buttons[] array to contain the pin number of the pin you want to test, then upload the sketch to your Teensy. Open the Serial Monitor and watch for messages sent from your Teensy. The sketch will send a message every time a button is pressed and released, indicating the pin number that recorded the event. Once you know a pin works, you can keep it in the array. When you move on to the next pin, add it to the end of the array, so that by the end, all of your buttons are added to the code.

With all of the wires soldered up and tested, you can feel comfortable about securing your microcontroller to the case, to prevent it from flopping around too much during use. This will help prevent mechanical stresses on the solder joints, as well as keep the USB cable firmly connected.

To secure my Teensy, I simply drilled small holes through my case on either side of the board (leaving a little room for wiring), then slipped two zip ties around the Teensy. Be careful not to excessively tighten the zip ties, they just need to be tight enough to hold the Teensy in place and keep it from flying inside your toy.

If you were able to successfully test out each pin individually using the simple Arduino sketch in the previous step, you are now ready to add some simple MIDI functionality. 

MIDI at a glance
MIDI can be a complex and immensely powerful protocol, and as such, there are many kinds of messages you can send from your Teensy. If you are so inclined, and want take your MIDI toy to the next level, I suggest first learning more about what you can send from your Teensy from this helpful page from PJRC about using MIDI on the Teensy.

To keep things simple, however, we are only going to focus on two specific types of messages in this Instructable: note on and note off. There’s no real mystery to these messages, they do exactly what you expect them to; they tell the device your controller connected to that it should either start or stop playing a specific musical note. You can learn about other types of messages, like control changes (CC) and sysex messages to enable more advanced effects, but we'll just keep it simple for now.

Using the Arduino sketch
The Arduino sketch attached to this step should look familiar, as it is based heavily on the sketch used in Step 11. All of the debouncing code is still in place, but a couple new variables have been added:

midiNotes[] array = array of MIDI note numbers that should be associated with the pins listed in the buttons[] array. For example, if the code sees that you are pressing or releasing buttons[5], it will send appropriate MIDI messages to turn on and off the MIDI note in midiNotes[5].

channel = all MIDI messages must been sent on a certain channel. This allows more than one device to be plugged into a MIDI chain and interact with sequencers and synthesizers.

Don't forget to change the USB Type of your Teensy!!
Your Teensy is capable of sending MIDI messages only when it is configured to be a MIDI device! To do this, just go to the Tools > USB Type menu and select "MIDI". The downside to this is that while in MIDI mode, you cannot send or receive Serial messages. This is why we did all the debugging in previous steps - you should already figured out if all of your pins work, we're just adding to the working system in this step!

So how do you know if your device is actually sending MIDI messages? There are many ways, and I'll share a couple of the easiest here. If you have any professional audio production / music making environment like Ableton Live, Traktor, FL Studio or Reaktor, you are probably already familiar with interfacing with MIDI devices - use them if you have them!

For the rest of us, however, open-source software can help us out a little bit. As I mentioned, there are many ways to look for MIDI messages, including writing Processing code, using a small MIDI utility like MIDI-OX or using popular synthesis programs like PureData or VVVV.  My personal favorite environment is Processing, but that can take a bit of work to just see if MIDI messages are coming through. Therefore, we'll use PureData to get up and running quickly.

Using Puredata to check for MIDI messages
First, head on over to the PureData website and download a copy of the latest version of PD for your operating system. Once you've installed it and opened it up, you can easily check for MIDI messages using a built-in function of the IDE.

Go to Media > MIDI Settings and select the Teensy from the menu (for me, it showed up as "OSS MIDI Device #1")

Next, go to Media > Test Audio and MIDI and watch the boxes in the lower-left part of the window that pops up, labeled MIDI In. Each time you press and release a button on your toy, you should see some changes occur. Numbers should change inside of the two boxes directly under the "stripnote" box (corresponding to MIDI note number and velocity), and the circle at the bottom should turn black and white from time to time (to indicate note on and note off).

Congratulations, you now have your very own, hand-made MIDI controller! You can use your MIDI controller for a number of cool things, not the least of which making music! However, this is where things can get somewhat complicated and go beyond the scope of this Instructable. In this step, I'll try to compile a list of example uses for your new MIDI controller for you to pursue. Feel free to leave a comment and give some more examples!

Music
To create music with your MIDI controller, you need to interface it with a synthesizer of some sort which can translate MIDI messages into actual noises. There are many, many great pieces of software to help you with this, but they can get quite costly. Here is a list of programs worth checking out that can help:
- Ableton Live!
- Reaktor
- Traktor
- GarageBand (Mac, free)
- Reason
- Puredata (free)

Other uses
Since MIDI messages can be read by nearly any software framework capable of serial communication, you can actually use MIDI for much more than music! You can use it in any situation where you would like to control some sort of activity through a physical interface, such as the following:
- Lighting and home automation (like http://www.midi.org/aboutmidi/halloween.php)
- Controlling video games. Re-program the Teensy to use the Keyboard and Mouse objects to send keystrokes like WASD, space and mouse clicks. Ever wanted to play Skyrim on a guitar?

What I've used my controller for
From the get-go, I was intending to use my MIDI controller to make sounds and, hopefully, music. However, I don't have much of a background in this area, so I'm more just messing around at this point. In the video below, I've connected my MIDI controller to Pure Data and created a simple synth patch based on this helpful article from Anthony Mattox.

Sometime in the next couple weeks, I hope to give the controller to a friend who dabbles in professional audio production and let him make something cool with it, but here is what I have for now:

Using the code from Step 14, your MIDI controller should be capable of sending note on and note off messages corresponding to button presses. However, the MIDI protocol (and the Teensy MIDI functionality) is capable of much more!

Pitch bends and after-touch
In MIDI, you can also send messages that alter the properties of notes after they have been sent. Pitch bend messages alter the pitch of a note, while maintaining the same velocity. After-touch messages alter the velocity, and leaves the pitch the same. You can use these messages to create cool kinds of continuous controllers, like violins and cellos.

Control changes (CC) and System Exclusive (SysEx) messages
You can also send a variety of very specific messages to tell your synth what kind of behavior you want your controller to have. You can use these messages to mimic modulation wheels, sustain pedals and more.

Receiving MIDI messages on a Teensy
In addition to sending MIDI messages, the Teensy can also receive MIDI messages, which can be really cool if you have LEDs, motors or other actuators that you want to control from your computer. All you really have to do is implement a method corresponding to the message you want to receive, then use the parameters that are passed into it however you want to control the IO of the Teensy! Learn more here: http://www.pjrc.com/teensy/td_midi.html