The Skittish Nightscape - Jed Diller and Ken Hoff

This was a 6 week project done as part of the Things That Think course at the University of Colorado, Boulder.

The Skittish Nightscape is an interactive art piece inspired by Aparna Rao’s piece, “Pygmies.” (A link to which can be found below.) The Skittish Nightscape is a sound sensitive, behavioral piece. In a nighttime forest scene there are stars twinkling, a firefly cruising through, and creatures lurking in the trees and shrubbery, all represented by LED’s. When it is quiet, the stars fade in and twinkle, a firefly starts to move across the scene, and the eyes of creatures start to appear sporadically, blinking once lit. Some of the creatures are curious, coming out sooner than the others, and some are a bit shyer, taking longer to come out. However, if a loud sound is made that disturbs the night, all the creatures go into hiding and the process starts over should things settle down again. The behavior continues until the display is no longer powered.

Here's what one cycle of The Skittish Nightscape's behavior looks like:


Aparna Rao: High-tech art (with a sense of humor),
Start at 1:50 for Pygmies or watch the whole thing if you're awesome.
http://www.ted.com/talks/aparna_rao_high_tech_art_with_a_sense_of_humor.html

For this project you will need (or at least what we used):

- 1 Arduino Mega

- 1 Arduino Uno

- 1 electret microphone break out board (or build you own opamp circuit if your comfortable with such things)
    - We used the SparkFun BOB-09964

- 54 LED's 
    - 12 blue (stars)
    - 42 yellow (15 eye pairs, 12 firefly positions)
    - These numbers are limited by the number of digital ports on the Arduino Mega

- 1 bread board (just nice for trial circuits but also necessary in Step 5 for ground connections)

- Stock acrylic and laser cutter for forest cut outs
    - Wood or other materials should be just fine. The forest shapes can be whatever you want.

- Plenty of single core wire in at least two colors and the means to solder

From our experience, getting your sensor to work as intended is one of the most difficult parts of this project. While light sensors are fairly stable, cheap microphones are not. If you wish to have a light sensitive display then you can skip this next section about the microphone and head down to the section about wiring. 


Getting Your Microphone Under Control:

A lot of time on this project was spent trying to get the microphone under control. We had a lot of issues getting the noise in the data to a level that was workable. The analog range on an Arduino port is 0-1023. Using the spark fun electret break out board and monitoring the analog values being read in, it was observed that the values ranged from 350-650 even in the most deathly quiet environments. To get usable data for triggering the LED behavior, a method called convolution was used. Convolution is in essence a way to smooth out data by applying a bell curve to a queue of values. In this case, convolution was applied to a queue of 7 values, the number of values in the queue can vary depending on how much smoothing is desired. Each of the seven values is multiplied by a coefficient and the sum of the seven coefficients should be equal to one. The smoothed value of interest is the sum of the seven analog values being read in multiplied by their coefficients. An example of this convolution is given below. For more accurate explanation on convolution, see the wikipedia page.

Convolution Arduino C Code Example:

---------------------------------------------------------------------------------------------------

// Convolution Example
// By Jed Diller
// April 2nd 2012

// Applies convolution to analog values being read in from a sensor on port A0
// If the value is above some threshhold, something is done. In this case
// a value is written to pin 9 on an Adruino Uno

// constants:
const int sensorPin = A0;
const int ledPin = 9;    

boolean DEBUG = true; // for print statements run serial monitor and set to true

int threshold = 550; // this values was determined through trial and error use the serial printout debug to 
                                // determine a reasonable threshold value
float a0 = 0.3;   // coeff for currrent value
float a1 = 0.2;  // coeff for plus & minus 1 from calc
float a2 = 0.1;  // coeff for plus & minus 2 from calc
float a3 = 0.05;  // coeff for plus & minus 3 from calc

//  define variables:
int sensorValue;  // sensor value for reading
int sV_adjusted;    // stored adjusted value

int sV_m3;    // sensor value 3 previous to calc
int sV_m2;    // sensor value 2 previous to calc
int sV_m1;    // sensor value 1 previous to calc
int sV_0;     // sensor value being calculated, center of curve
int sV_p1;    // sensor value 1 after to calc
int sV_p2;    // sensor value 2 after to calc
int sV_p3;    // sensor value 3 after to calc (what is actaully being read)

void setup()
{
 
  Serial.begin(9600); 
  pinMode(13, OUTPUT);
  digitalWrite(13, HIGH);
  analogWrite(ledPin, 0); //  start LED in off to avoid small flash between
                          //  each behavioral loop iteration
 
  delay(2000);
  digitalWrite(13, LOW);
 
  // get first 6 values (sV_m3 through sV_p2)
  sV_m3 = analogRead(sensorPin);
  sV_m2 = analogRead(sensorPin);
  sV_m1 = analogRead(sensorPin);
  sV_0 = analogRead(sensorPin);
  sV_p1 = analogRead(sensorPin);
  sV_p2 = analogRead(sensorPin);
  
}

void loop()
{

  // get new value
  sV_p3 = analogRead(sensorPin);
 
  // calculate adjusted value
  sV_adjusted = (int)(a3*(float)sV_m3 + a2*(float)sV_m2 + a1*(float)sV_m1 + a0*(float)sV_0 + a1*(float)sV_p1 + a2*(float)sV_p2 + a3*(float)sV_p3);
 
  // shift values around for next iteration
  sV_m3 = sV_m2;
  sV_m2 = sV_m1;
  sV_m1 = sV_0;
  sV_0 = sV_p1;
  sV_p1 = sV_p2;
  sV_p2 = sV_p3;
 
  // test adjusted value out against threshold
  if (sV_adjusted > threshold)
  {
    analogWrite(ledPin, 255);
  } else {
    analogWrite(ledPin, 0);
  }
 
  if (DEBUG)
  {
    Serial.print("Adjusted Value: ");
    Serial.println(sV_adjusted);
  }
 
}


---------------------------------------------------------------------------------------------------

Using convolution the raw analog noise of plus or minus 150 was reduced to around 10. A threshold was then chosen to be about 15 above the average convoluted values. A higher threshold translates to a larger input needed to violate the threshold. Adjusting the coefficient of the curve and making the bell curve "flatter" and using more values in the queue will further make the data stable but it will also make the event of interest harder to detect. Making the bell curve larger in the center by weighting the center values of the queue more heavily will have the opposite effect and the data will be noisier but events of interest will be detected more easily. 


Further Noise Problems and the Reason for Two Arduino's:

There was another major problem that we faced using the microphone we did. If there was any other hardware on the board, say a servo that was moved depending on the convoluted sensor value, the noise would again make the data unusable. For this reason, the Uno was used to monitor the microphone and do convolution on the data. If the values were then above the threshold a value was written to a port that the Mega was monitoring. If the Mega detected that the value it was monitoring was over some other threshold, it would accordingly change the behavior of the LED's. The microphone detection could not be done on the Mega due to noise from all the LED's being written to. 


Wiring:

The wiring of the microphone is easy enough with its labeled VCC, GND, and AUD headers. Notice that the top header in the image below is AUD as in audio, not AVD as in analog voltage. AUD is the microphone output that is read into an analog port on the Arduino.

Wiring the LED's was also easy enough, if not a little time consuming. For each LED solder one wire (of a certain color, we chose red or blue) to the longer of the two LED pins (which is the voltage in) and solder another wire (of a different color, we chose black) to the shorter pin, the ground pin of the LED. For the eye pairs, solder the short and long ends of two LED's to the same wire respectively. We used single core wire to make it easy to interface with the Arduino board ports. Be sure your wires are long enough to reach the the Arduino from wherever you may place the LED's in your display. To figure this out, it is worth drawing out what you think your display will look like, as seen in Step 4. 

As mentioned before, the Arduino Mega controls the behavior of the stars, eye pairs, and firefly. There are many things to consider when writing you own code for the behavior of your display. Below is what we chose to do.  


Objectives for The Skittish Nightscape Behavioral Code:

- 3 separate groups of LEDs: stars, eye pairs, and firefly, each with a distinct behavior.
- Individual stars and eye pairs act as separate entities
- The firefly functions as one entity
- The stars fade in and out, mimicking a twinkle. (This was done using pulse width modulation) 
- The eye pairs blink randomly
- All groups respond to sound and resume behavior continuously.


How it Works:

- During the main loop, each entity (star, eye pair, firefly) is either awake or asleep
- If awake, random numbers are used to determine when to blink/fade the eyes/stars, and the firefly continues on its path to the next LED pin
- If asleep, the 'time asleep' variable is increased


When There is a Sound (and the threshold is violated):

- Each entity is set to asleep, and time asleep begins increasing
- While an entity is asleep:
    - Each time a sleeping entity is visited, a random number is generated and compared to the time the entity has been asleep - if time asleep is greater, the entity wakes up.
    - This leads to each entity being woken up randomly at a different times. 


Issues and Advice:

- When writing the behavioral code, the built in Arduino delay() function must be avoided because it holds up all other processes going on. This is untenable with three LED groups behaving differently and simultaneously. Instead, the number of main loop iterations can be use for time dependent behavior. Note that the time it takes for each main loop iteration is dependent on the amount of stuff that you having going on within it. Trial and error and the art of refinement are your friends.   
- As discussed in Step 3, we had insurmountable microphone noise with the microphone being on the same board as the LED's being manipulated. Our solution was to totally separate the microphone by having it be the sole component on a separate board, the Arduino Uno. 


Sample Code: 

This is the code we ran on the Arduino Mega during operation. For the code to be run on the Uno see Step 2.

---------------------------------------------------------------------------------------------------------------------

// First, you need to initialize all the variables that your classes and main loop will use

int max_led = 50;
int star_twinkle_delay = 10;

const int light_sensor = A0;

int sensorValue = 0;
int sensorMin = 1023;        // minimum sensor value
int sensorMax = 0;

int threshold = 255;
float a0 = 0.3;   // coeff for currrent value
float a1 = 0.2;  // coeff for plus & minus 1 from calc
float a2 = 0.1;  // coeff for plus & minus 2 from calc
float a3 = 0.05;  // coeff for plus & minus 3 from calc

int sV_adjusted;    // stored adjusted value

int sV_m3;    // sensor value 3 previous to calc
int sV_m2;    // sensor value 2 previous to calc
int sV_m1;    // sensor value 1 previous to calc
int sV_0;     // sensor value being calculated, center of curve
int sV_p1;    // sensor value 1 after to calc
int sV_p2;    // sensor value 2 after to calc
int sV_p3;    // sensor value 3 after to calc (what is actaully being read)

// Here's where your 3 classes (types of LEDs) are defined. They each have a setup and loop function, just like the Arduino.

// Star

class star
{
public:
int pin;
int fading;
int brightness;
int awake;
int time_asleep;

void setup_star()
{
   awake = 1;
   fading = rand() % 2;
   brightness = rand() % max_led + 1;
}

void twinkle()
{
   if (awake == 1)
   {
     if (fading == 1)
     {
       brightness += 1;
     }
     else brightness -= 1;

     if (brightness > max_led)
     {
       brightness = max_led;
       fading = -1;
     }
     else if (brightness < 10)
     {
       brightness = 10;
       fading = 1;
     }

     analogWrite(pin, brightness);
     //delay(1);
   }
}
};

// Eyes

class eye_pair
{
public:
int pin;
int awake;
int blinking;
int blink_duration;
int time_since_blink;
int time_asleep;


void setup_eyes()
{
   awake = 1;
   time_since_blink = rand() % 1000;
}

void display_eyes()
{
   if (awake == 1)
   {
     if (blinking == 1)
     {
       digitalWrite(pin, LOW);
       blink_duration++;
       if (blink_duration > ((rand() % 20) + 10))
       {
         blinking = 0;
         blink_duration = 0;
         time_since_blink = 0;
       }
     }


     if (blinking == 0)
     {
       digitalWrite(pin, HIGH);
       if (time_since_blink > ((rand() % 1000) + 500))
       {
         blinking = 1;
         digitalWrite(pin, LOW);
       }
       else time_since_blink++;
     }
   }
}

};

// Firefly - handles it's own pin setup

class firefly
{
public:
int pins[12];
int time;
int time_to_blink;
int on_off;
int current_pin;
int awake;
int time_asleep;

void setup_firefly()
{
   awake = 1;
   time_to_blink = 100;
   for (int i = 0; i < 12; i++)
   {

     pins[i] = i + 40;
     pinMode(pins[i], OUTPUT);
     digitalWrite(pins[i], LOW);
   }
   current_pin = 0;
}

void go_firefly()
{
   if (awake == 1)
   {
     //turn off
     if ((on_off == 1) && (time > (rand() % 100) + 400))
     {
       digitalWrite(pins[current_pin], LOW);
       time = 0;
       on_off = 0;
     }

     //turn on
     if((on_off == 0) && (time > time_to_blink))
     {
       current_pin = (current_pin + 1) % 12;
       digitalWrite(pins[current_pin], HIGH);
       time = 0;
       on_off = 1;
     }
     time++;
   }
}
};

// Defines stars, eyes, and firefly objects - call firefly object frank just to distinguish it

star stars[12];// = {2,3,4,5,6,7,8,9,10,11,12,13};
eye_pair eyes[31];
firefly frank;

// Setup - initializes stars, eyes, firefly, and sensor

void setup()
{
Serial.begin(9600);
for (int i = 0; i < 12; i++)
{
   stars[i].pin = i + 2;
   pinMode(stars[i].pin, OUTPUT);
   stars[i].setup_star();
}

for (int i = 0; i < 16; i++)
{
   eyes[i].pin = i + 24;
   pinMode(eyes[i].pin, OUTPUT);
   eyes[i].setup_eyes();
}

frank.setup_firefly();

sV_m3 = analogRead(light_sensor);
sV_m2 = analogRead(light_sensor);
sV_m1 = analogRead(light_sensor);
sV_0 = analogRead(light_sensor);
sV_p1 = analogRead(light_sensor);
sV_p2 = analogRead(light_sensor);

}

// Main loop - reads in audio input, then handles LEDs

void loop()
{
sV_p3 = analogRead(light_sensor);

// calculate adjusted value
sV_adjusted = (int)(a3*(float)sV_m3 + a2*(float)sV_m2 + a1*(float)sV_m1 + a0*(float)sV_0 + a1*(float)sV_p1 + a2*(float)sV_p2 + a3*(float)sV_p3);

// shift values around for next iteration
sV_m3 = sV_m2;
sV_m2 = sV_m1;
sV_m1 = sV_0;
sV_0 = sV_p1;
sV_p1 = sV_p2;
sV_p2 = sV_p3;

if (sV_adjusted < threshold)
{

   for (int i = 0; i < 13; i++)
   {
     stars[i].twinkle();
     if (stars[i].time_asleep > ((rand() % 100000000) + 3000))
     {
       stars[i].awake = 1;
       stars[i].time_asleep = 0;
     }
     else
     {
       stars[i].time_asleep++;
     }
   }
   for (int i = 0; i < 30; i++)
   {
     eyes[i].display_eyes();
     if (eyes[i].time_asleep > ((rand() % 100000000) + 3000))
     {
       eyes[i].awake = 1;
       eyes[i].time_asleep = 0;
     }
     else
     {
       eyes[i].time_asleep++;
     }
   }
   frank.go_firefly();
   if (frank.time_asleep > ((rand() % 100000000) + 3000))
   {
     frank.awake = 1;
     frank.time_asleep = 0;
   }
   else
   {
     frank.time_asleep++;
   }
}
else
{

   for (int j = 0; j < 54; j++)
   {
     digitalWrite(j, LOW);
     for (int i = 0; i < 13; i++)
     {
       stars[i].awake = 0;
     }
     for (int i = 0; i < 30; i++)
     {
       eyes[i].awake = 0;
     }
     frank.awake = 0;
   }
}

}

---------------------------------------------------------------------------------------------------------------------


Behavioral Code Notes:

You may notice that the convolution behavior is seen in this code too. This convolution is being preformed on the analog values that are being output by the Uno. If a more stable and less noisy microphone (or a light sensor) were used, the use of the Uno could be eliminated and this code would not have to be changed.

The building of your display is really up to you but the basic steps of how we did it are given below.

1. Assemble the basic structure of your display. We went with an open box rather than a flat panel in an attempt to block out some of the ambient light. We made our structure out of wood.

2. Once you have your display, draw out where you want parts of your display subject to go. In our case, we drew out where the trees, bushes, eye pairs, stars, and firefly would be in relation to each other. While the final positions ended up being a little different, this is a good first approximation.

3. With a decent idea of where we wanted our stars, eye pairs, firefly, trees, and bushes, we cut out the holes for each. This step was done in parallel with the next step.

4. Build your display subject. We used a laser cutter and acrylic to cut out 2D shapes of trees and bushes for the LED creatures to linger in. We layered the 2D cutouts to give the impression of distance. Eye pairs, stars, and firefly LEDs were place on all levels. This required holes to be made in the trees and bushes as well as the box.

5. Paint away! With the holes cut and the trees and bushes positioned, we painted the display. We went with a solid matte black since our display is a forest at night.

6. With the display painted, the LED were put into position.
 

With the display all together physically, the next step is to place all the LED's in their proper positions and then connect their wiring to the correct ports on the Arduino Mega. This process can be a little messy and planning ahead goes a long way (something we didn't quite do as may be obvious from the rat's nest of wiring we ended up with). The best way to do this is to plug in each LED to its intended port, then run the behavior code to be sure the LED you just placed is working correctly. If it is, continue on to the next one until all the LED's are wired up. While there is a port on the Mega for each of the voltage pins of the LED's, there are only three ground ports. The LED's were connected to ground using a bread board, a couple rails of which were connected to ground ports on the Adruino board.  
 

Future Work:

With more time there are several things we would like to add and refined in The Skittish Nightscape.

- We would like to add a mechanical component, namely parts of the forest growing and retracting as another sound sensitive behavior.
- You may notice that the stars go out along with the eyes and firefly.That was an easily changeable mistake that made it into the video above. 
- We would like to use a finer resolution, less noisy microphone and do further sound analysis, weeding out ambient noise as a source for triggering the behavior.
- With more time we would like to better characterize the board level noise issues seen with the microphone.