Urban Design - Music From Facial Expressions - the Sound of Emotion

Project Overview

Can you imagine complete strangers walking up to an art installation on the street and collaborating with each other to create music driven solely by their facial expressions?

By strengthen the connection between people and the community they share, neighborhoods may begin to positively influence their physical, social and emotional well being. Below is a short video on a prototype designed to encourage such group dynamics, plus a 13 step Instructables on how "The Sound of Emotion" project came together!

Market Street runs through the center of San Francisco. This expansive boulevard was once the jewel of the City, but over the years has become gritty, uninviting and at times, dangerous. Mass transit from BART and MUNI service the street so it's well traveled, yet pedestrians often calculate their trajectory carefully to avoid confrontation when using the sidewalks.

In an effort to bring about change for Market Street, the Yerba Buena Center for the Arts (YBCA), the San Francisco Planning Department, and the James L. Knight Foundation joined forces and announced an open call for proposals to reimagine and reinvent this area. Their intention was to select and empower 50 planners/designers/artists to create prototypes for inclusion in a three day event, "Market Street Prototyping Festival" (MSPF).

I was excited to initially discover the MSPF site and very impressed with their vision and determination to take on such a challenging project! The proposed festival would provide an amazing opportunity to innovate and get significant feedback.

After considering the mission as described by the festival planners, I began to explore possibilities for an installation that would honor San Francisco's unique qualities, plus bring people together in manner that would strengthen their connection to one another. Music, art, science and technology are all attributes that tie into San Francisco's cultural and social identity - I felt it would be ideal to incorporate as many of those facets as possible.

Although a small stipend would be provided to the selected teams, developing a project like this would require significant time and expense. After extensive brainstorming with potential co-developers (that’s code for coercing my close friends into agreeing to work for free), I formed a five person team, then created and submitted a proposal for a project: The Sound of Emotion.

The SOE team:

• Paul Pillitteri – Lead, Design & Fabrication

• Seth Neiman – Advisor & Technology

• Josh Coleman – Community & Composer

• Howard Cohen – Software

• Evan Kuester – Design

Read more about each team member at the following URL: http://thesoundofemotion.com/team/

For its forward thinking and unusual "crowd sourcing" effort, it is my understanding that the Yerba Buena Center for the Arts and the SF City Planning Department received nearly 250 proposals from four countries. It would have been very interesting to see the range of ideas that were delivered! After extensive due diligence on their part, they selected the top 50. Fortunately, The Sound of Emotion was given the official nod of acceptance.

High fives all around - it's time to celebrate!! No wait... we actually have to build this thing now!

The proposed project centered around two major attractions.

Attraction 1: Music as a universal language that brings people together.

Researchers Chris Loersch and Nathan Arbuckle recently published a paper in the Journal of Personality and Social Psychology that explores music as an evolved group process. Their combined findings provide evidence that music may have evolved to serve the intense social needs of our species. This suggests that the powerful psychological pull of music in modern life may derive from its innate ability to connect us to others.

It felt intuitive that music would strengthen the connection between the denizens of Market Street as it has over history of our evolution. This would become the backbone of our project and proposal.

Attraction 2: Science and Technology.

The team was extremely fortunate to have special access to the emerging technology of "expression recognition" courtesy of Emotient, Inc.

The founders of Emotient are the creators of automated facial expression recognition, and are leaders in the machine learning and cognitive behavior fields. The technology works by detecting the small muscle movements in people’s faces – the same way that people do. The technology is capable of recognizing the seven basic emotions as well as many other higher level states such as happy, sad, confusion, engagement, excitement, pose, and gaze.

After several software development meetings, an architecture was proposed and agreed upon. "ThinSOM," our custom iPad app would be created to capture facial expressions and send them to our proprietary "SOM" server. The server calls Emotient's recognition engine to analyze faces, then acting as a controller, triggers a software program, Ableton Live, for executing the music tracks.

Although we would analyze all seven emotions, after conducting initial analysis on the complexity for an installation on the street, we made a simplifying decision to ask participants to mime only two states (happy and sad).

There would be four iPads, each with the custom app and one MAC to run the server and Ableton, hidden in an electronics drawer at the bottom. Communication between the devices would be conducted via a wireless ad hoc network.

The architecture changed several times over the course of development as tradeoffs were made about how the app and server communicated with each other, which entity would do the majority of the processing and how status was updated to the participant. Over the period of development, roughly 4000 lines of code were written in total. The primary languages were C, C++ and mod_perl.

With the software underway, a spec was developed for the musical compositions. We wanted to celebrate the diversity of San Francisco with interesting cross-cultural music. 15 unique compositions were ultimately created. We knew up front there would be at least two distinct challenges:

Challenge1: The musical compositions needed to sound good and stay in sync with any of the 64 possible musical combinations. The four unique instruments (one instrument per iPad) and two states (happy and sad) would require a total of eight custom musical tracks. The tracks would turn on and off, depending on the number of participants and their facial expressions (e.g. all four instruments could be playing in "happy" mode or all four in "sad" mode or two happy tracks and two sad tracks, etc.).

Solution 1: Build the composition in layers and test the tracks against each other in multiple combinations. Based on the design only four tracks could be played at the same time, however, is was useful to compose and listen to all eight tracks simultaneously, checking for dissonance and sync issues.

==========

Challenge 2: The musical representation of happy versus sad is very subjective. The project did not have the budget or time to conduct market acceptance testing.

Solution 2: The debuted composition encompassed four instruments; gamelan bells, percussion, strings and vocals. Even though the happy and sad states used the same basic instrument, focus was placed on creating a discernible and audible difference between each state, so participants knew they were controlling the composition. In the end, the difference was noticeable, but I believe the change between happy and sad could be more pronounced, given more development time.

As the software and music took shape, we were keenly aware that latency issues would be critical. The musical change would need to be noticeable and ideally instantaneous once a participant changed expressions. The code associated with triggering the musical track change was extremely complicated. We explored numerous ways to deliver the music and experimented with different file formats. After much research and deliberation, we confirmed the use of Ableton Live as the program to assemble and sync the tracks.

To facilitate this, we wrote code to mimic a MIDI controller, capable of triggering Ableton similar to a keyboard. Countless hours were spent on discussing and experimenting how and when to shift each instrument's change between the emotion states; would we do it on the beat, the bar or the phrase? In the end, we came up with an elegant and simple solution:

1) All eight tracks would be seamless loops of one to two minutes in length.

2) The installation would launch all eight tracks at once, but only turn on the speaker for the appropriate instrument(s) that was being driven by a participant, if any. This technique would eliminate the sync problem as all tracks would begin at the same time and play continuously, even if they weren't being delivered to the master output and thus the external speakers.

With the music and software in full development, I turned attention to the physical structure. Since this prototype was to live on the street for only three days, I was not overly concerned about hardening it for street abuse, but did secure the iPads and MAC in a manner that made it difficult to remove without special tools.

Multiple designs were created and once we landed on the current shape, the outlines were defined and post processing was finalized in Vcarve Pro. The five uprights, the iPad cases, iPad mounts and the electronic housing were all CNC machined from wood and/or acrylic. Each upright was pocketed to accommodate the 26 clear 1/2" acrylic slats.

The initial plans were created without specifications for acceptable tolerances, iPad housings, mounts, fasteners and assembly details, so the fabrication timeline became somewhat organic in nature. There were a thousand details that either created obstacles or opportunities - usually depending on how much time was left to the festival date, the initial (or deviation) cost, and the risk of destroying nearly finished materials (a significant monetary and time risk - particularly when working on community CNC equipment where machine time was limited). Fabrication was conducted at the San Francisco TechShop.. a truly amazing place and one of the partners for the festival.

Bringing the entire assembly together was frenetic - we were simply running out of time. Although we did testing at various milestones, the public build would be the first time everything was tested in the final configuration.

The physical assembly on Market Street went smoothly. Since much of the installation was pre-assembled, it took three of us about two hours. During the final steps of bringing everything online, it was evident the iPads were not communicating with the server on the MAC. None of the code had changed, so there wasn't an immediate and obvious reason for the failure. We conducted a significant series of tests before concluding the default channel we were using for the wireless network was overloaded with hundreds of established signals emanating from the surrounding Market Street companies! Once a network was created using a different channel, our communications functioned correctly.

The first day of the festival was Thursday - similar to industry day for trade shows. We were situation in the Financial District where professionals poured out of their offices during lunch to be greeted by some pretty cool projects on the street! Many who saw The Sound of Emotion at lunch, came to spend more time at the end of their workday. That was a great show of interest. It was extremely satisfying to see all our efforts finally come to fruition!

On many occasions over the three day installation period, groups of complete strangers were engaged and verbally collaborating to drive musical results. Chants rang out - “everyone frown -- now smile” “You go sad and I’ll go happy...” Onlookers would even chime in with suggestions!

It was nearly as enjoyable to watch and photograph the interactions as it was gratifying to hear the compliments of how much people enjoyed the installation.

More than a few of the participants walked away carrying on conversations with their newly found "fellow musicians!"

Do we consider the project to be a success? Overwhelming yes! Of course there are many aspects about the installation that could be fine-tuned and improved, but the project achieved ever one of its intended goals. In general, the project was:

• Inclusive
• Transformative
• Collaborative
• Sociable
• Dynamic
• Context-Specific
• Capable of creating a "destination"
• Community focused

----------

Many thanks to my team for donating their expertise and time, the YBCA, SF Planning Department and sponsors for their bold initiative and tireless efforts, Jason Kelly Johnson from Future Cities Lab for his expert guidance and all that assisted in the project!!

----------

Comments and questions are welcome!