Touch-Sensitive Musical Painting

• This musical painting is a graphical score that you touch to play melodies and harmonies. It incorporates visual composition, musical composition, and gestural interaction to create a new form of visual music where the composition and the instrument are combined together.
• This Instructable will show you how to make a touch-sensitive musical painting with a Teensy to send MIDI to a Mac Mini that synthesizes sounds with Ableton Live. The painting itself is made with acrylic paint, conductive paint, colored pencil, and nails on a wood panel.
• To get an overview of how to do capacitive touch sensing in general, check out this Instructable.
• This technique can be easily applied to other materials and touch sensing applications!
• This painting was a collaboration between Sophia Brueckner and Eric Rosenbaum.
• This Instructable is a work in progress. We have been refining our process with each iteration, and feedback is welcome!

• Conductive paint
  • Electromagnetic shielding paint works really well, and you can buy it in large quantities (http://www.lessemf.com/paint.html)
  • Bare Conductive paints are easily available, but they smear a little bit more (http://www.bareconductive.com/shop/)

• Wooden art panel
  • These are available at most art stores
  • They can be round or rectangular
  • Choose a thicker one (We used a 1.5" thick panel) to give yourself room for the electronics
• Gesso primer (available at any art store)
• Acrylic paints
• Brushes, stencils, etc.
• Hammer and nails
  • Use the smallest nails you can as long as they go all the way through the panel and stick out far enough to solder to. Larger nails may split the wood.
• Wire, solder, copper tape (available on Adafruit or Sparkfun)
• Teensy board (available on Adafruit or Sparkfun)
• Computer (We used Ableton Live, but you could use Processing or anything else you like to make sound.)
• External speakers (optional)

We used an unprimed wooden art panel (1.5" thick) because:

• It's sturdy...people will be touching this, so you want a durable material.
• It's a great painting surface, and it works well with the conductive paint.
• There's room in the back to hide the electronics.
• The rigid surface keeps electrical connections from being jostled too much.

These are available at most art stores. You could also make your own, or experiment with other materials (canvas and conductive thread?).

Prime your surface with Gesso. It's easier to paint on a primed surface, and your colors will look better. I usually do 2-3 coats. You can use a cheap, wide paintbrush to do this. Wait for it to dry after each coat. If your surface is streaky, you can sand it when it's fully dry.

Next, paint your background and all the parts of the painting that are not going to be touch sensitive. Do this as if you were painting a regular painting. You will paint the conductive shapes on top of this in the next step.

I used Holbein Acryla gouache and Prismacolor colored pencils, but you could use something else. Just remember that people will be touching the painting, so choose something that won't get ruined if it gets wet or if it is rubbed a lot. Acrylic paints are a great choice. If you are not as comfortable with a paintbrush, you could also use stencils and a stencil brush.

After your painting is fully dry, use the conductive paint to paint your touch sensitive shapes on top. Remember these shapes shouldn't touch. Paint somewhat thickly, and avoid watering down the conductive paint. You can mix in a little water with some conductive paint on a palette if you are struggling to paint tiny details or clean edges and need it to flow a bit more. Let the painting dry completely.

Warning: If you attempt to mix colored paint with the conductive paint to make new colors, your shapes may no longer be conductive enough for touch sensing. It's best to keep the paint as pure as possible.

We wanted this to be very sturdy because we put it in a public space and it gets a lot of use, so we hammered regular nails through each conductive shape. You can touch the shape anywhere, not just on the nail head, to produce sound. The nails create a solid connection between the conductive shapes and the electronics on the back of the painting. I painted over some of the nail heads with more conductive paint for visual interest, but this is not needed for functionality.

Tip: You don't necessarily have to have one shape per sound. On the back of the painting you can wire multiple shapes to play the same sound. This simplifies the electronics while allowing you to create a more complex composition.

Connect the nails to the inputs on your board/s. We used a mixture of soldered wires and copper tape. If your painting is complex, it is helpful to label all the nails with a pencil on the back. Keep in mind how many inputs you have available on your board/s. We often connected multiple shapes together to the same input to reduce the number of inputs we needed.

The following code allows you to program your Teensy board to send MIDI to your computer. You can also download the code on GitHub.

int bounceThreshold = 0; // a press must last this many loop cycles (about 20ms each) before triggering. 
// higher values make it less sensitive, fewer false triggers, higher latency
#define NUM_INPUTS 44
int channel = 1;
// edit this array to change the MIDI notes sent 
//int pitches[NUM_INPUTS] = {
//  79, 84, 75, 67, 65, 63,           // top of board (up, left, down, right, space, click)
//  60, 62, 77, 68, 58, 56,           // left side of female header, w,a,s,d,f,g
//  51, 53, 48, 55, 72, 74            // right side of female header, up,down,left,right,left-click,right-click
//};
// teensy makey pin numbers
int pinNumbers[NUM_INPUTS] = {
  0,1,2,3,4,5,7,8,9,10,11,12,13,14,15,16,17,         // left side starting at USB connector, pin D0, skipping D6 
  26,25,24,23,22,21,20,19,18,38,39,40,41,42,43,44,45, // right side starting at USB connector, pin B6, skipping gnd and aref
  36,37,32,33,34,35,28,29,30,31                      // these are "interior" through-holes, in the center of the board
};
// cap sense thresholds for each pin
// this is proportional to the capacitance on the pin that will count as a press
// it is units of a very small unit of time, in iterations through an unrolled loop
// higher values make it less sensitive (i.e. require larger capacitance)
//int capThresholds[NUM_INPUTS] = {
//  1, 1, 1, 1, 1, 2,
//  2, 2, 1, 1, 1, 1,
//  2, 1, 1, 1, 1, 1,
//};
int bounceCounter[NUM_INPUTS];    
boolean pressed[NUM_INPUTS];
void setup(){
  for (int i=0; i
void loop() { 
  for (int i=0; i1){                                                       // if we detect a touch on the pin
      if (!pressed[i]) {                                          // and if we're not already pressed
        bounceCounter[i]++;                                           // increment the bounce counter
        if(bounceCounter[i] > bounceThreshold){                       // if we're over the bounce threshold
          usbMIDI.sendNoteOn(60+i,127,channel);                      // send a MIDI note on
          pressed[i] = true;                                              // remember it was pressed
          bounceCounter[i]=0;                                             // reset the bounce counter
        }
      }
    } 
    else {                                                  // if we don't a detect touch on the pin
      if (pressed[i]) {                                           // if this key was pressed before
        usbMIDI.sendNoteOff(60+i,127,channel);                    // send a MIDI note off
        pressed[i] = false;                                          // remember we are not pressed
        bounceCounter[i] = 0;                                        // reset the bounce counter
      }        
    }
  }
  //Serial.println(" ");
}  
// CapacitiveSensor tutorial from  http://www.arduino.cc/playground/Code/CapacitiveS...
// readCapacitivePin
//  Input: Arduino pin number
//  Output: A number, from 0 to 17 expressing
//  how much capacitance is on the pin
//  When you touch the pin, or whatever you have
//  attached to it, the number will get higher
uint8_t readCapacitivePin(int pinToMeasure) {
  // Variables used to translate from Arduino to AVR pin naming
  volatile uint8_t* port;
  volatile uint8_t* ddr;
  volatile uint8_t* pin;
  // Here we translate the input pin number from
  //  Arduino pin number to the AVR PORT, PIN, DDR,
  //  and which bit of those registers we care about.
  byte bitmask;
  port = portOutputRegister(digitalPinToPort(pinToMeasure));
  ddr = portModeRegister(digitalPinToPort(pinToMeasure));
  bitmask = digitalPinToBitMask(pinToMeasure);
  pin = portInputRegister(digitalPinToPort(pinToMeasure));
  // Discharge the pin first by setting it low and output
  *port &= ~(bitmask);
  *ddr  |= bitmask;
  delay(1);
  // Make the pin an input with the internal pull-up on
  *ddr &= ~(bitmask);
  *port |= bitmask;
  // Now see how long the pin to get pulled up. This manual unrolling of the loop
  // decreases the number of hardware cycles between each read of the pin,
  // thus increasing sensitivity.
  uint8_t cycles = 17;
  if (*pin & bitmask) { 
    cycles =  0;
  }
  else if (*pin & bitmask) { 
    cycles =  1;
  }
  else if (*pin & bitmask) { 
    cycles =  2;
  }
  else if (*pin & bitmask) { 
    cycles =  3;
  }
  else if (*pin & bitmask) { 
    cycles =  4;
  }
  else if (*pin & bitmask) { 
    cycles =  5;
  }
  else if (*pin & bitmask) { 
    cycles =  6;
  }
  else if (*pin & bitmask) { 
    cycles =  7;
  }
  else if (*pin & bitmask) { 
    cycles =  8;
  }
  else if (*pin & bitmask) { 
    cycles =  9;
  }
  else if (*pin & bitmask) { 
    cycles = 10;
  }
  else if (*pin & bitmask) { 
    cycles = 11;
  }
  else if (*pin & bitmask) { 
    cycles = 12;
  }
  else if (*pin & bitmask) { 
    cycles = 13;
  }
  else if (*pin & bitmask) { 
    cycles = 14;
  }
  else if (*pin & bitmask) { 
    cycles = 15;
  }
  else if (*pin & bitmask) { 
    cycles = 16;
  }
  // Discharge the pin again by setting it low and output
  //  It's important to leave the pins low if you want to 
  //  be able to touch more than 1 sensor at a time - if
  //  the sensor is left pulled high, when you touch
  //  two sensors, your body will transfer the charge between
  //  sensors.
  *port &= ~(bitmask);
  *ddr  |= bitmask;
  return cycles;
}

We used Ableton, but there are lots of ways to work with MIDI Input. Here are a few options:

Ableton MIDI Input

MidiBus for Processing

Send and Recieve MIDI Signals with Arduino

With some planning, you could also hide the computer and speakers within the painting itself. We used external speakers and kept the Mac Mini separate.

This Instructable is a work in progress as we build more of these musical paintings. Feedback is welcome!