Adafruit Audio-to-Vibe, Feel the Music!

The first section and accompanying video explain the problem this setup addresses. If you want to jump straight to the instructions, see the step-by-step guide below. Happy creating!

Problem statement 

Music plays a fundamental role in human society and culture. It is an important part of our culture to be able to express ourselves and transmit our feelings to other people. Hearing loss affected people cannot enjoy this to the fullest anymore, which can lead to feelings of loneliness, frustration, and isolation [1]. By 2050, around 2.5 billion people will suffer from some degree of hearing loss, which by then will amount to almost 25% of the world’s population [2]. Adding a new dimension to music ensures this rising number of people won’t feel left out at public gatherings. Haptic technology can provide this dimension and transmit the energy of music to the listener/feeler. 

State of the art 

There are several systems that have been developed to improve music perception for hearing impaired listeners using haptic technology [3]. One of these technologies works on a large scale trough vibration of the floors of a building to enhance the music listening experience. In a study from 2016 it’s proved that this technology can improve the quality of a concert experience for hearing impaired people. A limitation of this technology is the challenge in designing such a building [4,5]. 

Another type of a haptic technology on a medium scale is a wearable suit or vest that uses haptic actuators to give whole body music experiences to the users. An example of such a wearable system is the LIVEJACKET, this is a vest with 22 haptic actuators attached to the arms and torso of the user. This suit can enhance the music listening experience for normal hearing participants but was not proven to aid significantly for people with hearing loss [6]. This type of suit would map each different music instrument to a different haptic simulator around the body, a significant drawback of this method is that it necessitates access to each individual instrument within a musical composition, which is usually not feasible [7]. This haptic suit could also be impractical to wear for a long period of time.  

It is also possible to do this on a smaller scale, with compact wearable devices. An example of such a device is the Pump-and-Vibe, this is a device that is worn on the arm. The Pump-and-Vibe device features eight vibration motors attached to the forearm and an air pump on the upper arm to adjust pressure. A qualitative assessment of the Pump-and-Vibe examined the mood elicited by a musical piece across three conditions: audio alone, haptic alone, and combined haptic and audio, involving young participants without any specified hearing impairment. The results indicated that the system could evoke moods and affect the mood induced by the audio [8]. The effect is not yet tested for hearing impaired listeners.  

Another existing technology is a system designed to enrich the musical experience for the deaf by utilizing sensory inputs beyond auditory reception. This system comprises a vibrating "Haptic Chair" and a computer display that presents visual effects corresponding to musical features. Developed with input from a diverse group of deaf individuals and profoundly deaf musicians, a study involving 43 deaf participants indicated that the system significantly enhances their musical experience. Most users favoured either the Haptic Chair alone or in combination with the visual display, suggesting this technology could transform how the deaf community engages with music [9]. However, the study had several limitations, such as the lack of control for novelty or placebo effects and the potential influence of audio from the contact loudspeakers.  

Other examples of more compact systems include the Mood Glove and the mosaicOne series of devices. The Mood Glove features eight motors, with five positioned on the back of the hand and three on the palm. The device adjusts stimulation frequency and intensity to convey different moods in musical pieces. However, the Mood Glove requires the mood intended for each section of the musical piece to be manually extracted and inputted, significantly limiting its practicality for real-world applications [10]. 

The mosaicOne_B is equipped with two sets of six haptic stimulators positioned along the forearm. It translates the fundamental frequency of sound into tactile sensations on the skin, enabling users to discern frequency differences as small as 1.4%. This exceeds the discrimination abilities of most cochlear implant users [11]. The device incorporates an effective noise-reduction strategy but requires further testing with harmonic background noise. While informal testing indicates enhanced musical enjoyment, formal evaluation is pending. Another version, the mosaicOne_C, is under development but has not yet been tested [12] . 

References

[1] American Academy of Audiology, “Depression and Hearing Loss - American Academy of Audiology,” American Academy of Audiology, May 27, 2022. https://www.audiology.org/consumers-and-patients/hearing-and-balance/depression-and-hearing-loss/ 

[2] World Health Organization: WHO, “Deafness and hearing loss,” Feb. 02, 2024. https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss 

[3] M. D. Fletcher, “Can haptic stimulation Enhance music perception in Hearing-Impaired Listeners?”, Frontiers in Neuroscience, vol. 15, aug. 2021, doi: 10.3389/fnins.2021.723877. 

[4] Shibasaki, M., Kamiyama, Y., and Minamizawa, K. (2016). “Designing a haptic feedback system for hearing-impaired to experience tap dance,” in Proceedings of the 29th Annual Symposium on User Interface Software and Technology, (Tokyo), 97–99. 

[5] Tranchant, P., Shiell, M. M., Giordano, M., Nadeau, A., Peretz, I., and Zatorre, R. J. (2017). Feeling the beat: bouncing synchronization to vibrotactile music in hearing and early deaf people. Front. Neurosci. 11:507. doi: 10.3389/fnins.2017.00507 

[6] Hashizume, S., Sakamoto, S., Suzuki, K., and Ochiai, Y. (2018). “LIVEJACKET: wearable music experience device with multiple speakers,” in Distributed, Ambient and Pervasive Interactions: Understanding Humans, Dapi 2018, eds N. Streitz and S. Konomi (Berlin: Springer), 359–371. doi: 10.1007/978-3-319-91125-0_30 

[7] Gunther, E., Davenport, G., and O’modhrain, M. S. (2003). Cutaneous grooves: composing for the sense of touch. J. New Music Res. 32, 369–381. doi: 10.1076/jnmr.32.4.369.18856 

[8] Haynes, A., Lawry, J., Kent, C., and Rossiter, J. (2021). FeelMusic: enriching our emotive experience of music through audio-tactile mappings. Multimodal Technol. Interact. 5, 1–21. doi: 10.3390/mti5060029 

[9] Nanayakkara, S., Taylor, E., Wyse, L., and Ong, S. H. (2009). “An enhanced musical experience for the deaf: design and evaluation of a music display and a haptic chair,” in Proceedings of the 27th Annual Chi Conference on Human Factors in Computing Systems, (New York, NY: Association for Computing Machinery), 337–346. doi: 10.1145/1518701.1518756 

[10] Mazzoni, A., and Bryan-Kinns, N. (2016). Mood glove: a haptic wearable prototype system to enhance mood music in film. Entertainment Comp. 17, 9–17. doi: 10.1016/j.entcom.2016.06.002 

[11] Kang, R., Nimmons, G. L., Drennan, W., Longnion, J., Ruffin, C., Nie, K., et al. (2009). Development and validation of the University of Washington clinical assessment of music perception test. Ear Hear. 30, 411–418. doi: 10.1097/AUD.0b013e3181a61bc0 

[12] Fletcher, M. D., Thini, N., and Perry, S. W. (2020c). Enhanced pitch discrimination for cochlear implant users with a new haptic neuroprosthetic. Sci. Rep. 10:10354. doi: 10.1038/s41598-020-67140-67140 

First, let’s start with an overview of all the components:

1. Arduino (Micro) 
2. Breadboard 
3. Arduino connection cable 
4. Haptic motor: Drake LFi Impact 
5. Adafruit DRV2605L Drive 
6. Connector screw terminals X2 
7. Audio jack (Aux cable) 
8. Capacitor, 1 µF 
9. Cables 

Before you can start building your circuit, you need your components to be ready. There are two options available to connect the audio jack to the circuit. You can buy an audio jack connector or alternatively strip an audio jack cable. In our setup, we have decided to strip an old audio jack that is no longer in use. Since it is quite difficult to insert the stripped audio jack cable directly into a breadboard, we will use a connector screw terminal to facilitate this connection. 

The circuit is quite straightforward.

The DRV2605L driver is connected to the 5V and GND pins of the Arduino. It's crucial to check the pin functions specific to your Arduino board. You can find the specific pin configurations for other Arduino models in their respective documentation online. We are using the Arduino Micro, so: 

• Connect the SDA pin of the driver to the digital pin 2 of the Arduino. 
• Connect the SCL pin of the driver to the digital pin 3 of the Arduino. 

These pin assignments differ depending on the Arduino model you are using. For example, with the Arduino Uno: the SDA pin of the driver needs to be connected to the digital pin 18 of the Arduino Uno and the SCL pin of the driver to the digital pin 19 of the Arduino Uno.

Depending on the type of audio cable you are using, it may have either two or three wires. However, only two wires are necessary to connect for one haptic motor: 

• Connect the black wire of the audio jack to the GND pin of the Arduino. 
• Connect the red or white wire to the IN pin of the driver via a 1 μF capacitor. 

Lastly, connect the haptic motor to the output pins of the driver. 

We decided to connect one additional wire for debugging purposes and to help display the signal in the Arduino Plotter. This wire connects the IN pin of the driver to the analog pin 0 of the Arduino. This additional connection is not required for the audio-to-vibration function to work but may be useful during debugging. 

We will use the Arduino IDE to program the Arduino. Therefore, please download the Arduino IDE from the developer's homepage (https://www.arduino.cc/) and install the software by following the steps of the installer. 

After installing the Arduino IDE we need to add the Adafruit library to the IDE: 

1. Go to Sketch > Include Library > Manage Libraries 
2. Search for Adafruit_DRV2605 in the search bar. 
3. Click on Install 

You now have all the necessary software to complete this project. 

The Adafruit DRV2605 drive has a built-in audio-to-vibe function. The code is therefore rather simple, since the Arduino is really only used to put the drive into the audio-to-vibe function. The void loop can be left completely empty. However, we have decided to insert a Serial.println() as this will show us the input values for in the Serial Monitor and Serial Plotter, which can help in debugging the setup. 

Remember, to select the right Arduino Board at the top of the screen, otherwise you will not be able to upload your code to the Arduino. 

#include <Wire.h> 
#include "Adafruit_DRV2605.h" 
 
const int audioInputPin = A0;     // Analog input pin for audio signal  

Adafruit_DRV2605 drv; 

void setup() { 
  Serial.begin(9600); 
  Serial.println("DRV2605 Audio responsive test"); 
  drv.begin(); 

  drv.setMode(DRV2605_MODE_AUDIOVIBE); 

  // AC coupled input, puts in 0.9V bias 
  drv.writeRegister8(DRV2605_REG_CONTROL1, 0x20);   

  // Analog input 
  drv.writeRegister8(DRV2605_REG_CONTROL3, 0xA3);   
} 

void loop() { 
  // These two lines are strictly speaking not necessary, but they help in understanding the signal. 
  int audioValue = analogRead(audioInputPin);  
  Serial.println(audioValue); 
} 

If everything is done correctly, you should now be able to play a song on your computer and feel the haptic motor vibrate along with the music. To do this, you just play a song as if you were listening to it with headphones. The motor will not only vibrate in accordance with the sound but also with the volume at which it is played. Therefore, make sure that your volume is turned up high.  

The haptic motor will always follow the part of the loudest part of the music, which often happens to be the drums or bass. For the best effect, we recommend playing a song with a recognizable drum or bass set. A good to test your setup on is We will rock you by Queen: https://www.youtube.com/watch?v=-tJYN-eG1zk. As you can see in our video, the haptic motor moves up during the double boom and downwards during the clap.

If your setup is not working, make sure to go to step 6 for help with debugging. 

If you do not get the desired output, you can find some possible solutions to problems that we ourselves encountered: 

• As always, the first step in debugging is verifying whether your circuit is correctly made. Are the SDA and SCL pins connected to the correct Arduino pins? Is your drive connected to the power supply? Is the ground of the audio jack connected to the ground of the Arduino? Etc. 
• Verify the output of your audio jack, not every audio jack has the same colour code. Therefore, check which cable corresponds to the ground and which to the positive output. If there are three cables, two will be the output. In most cases these two will give the exact same output. 
• Verify the output signal. Check using the functions we defined in the void loop what your signal looks like. We recommend playing a noise or sound which is very repetitive in nature, this will help you see the peaks in the signal. It is normal to have noise on the signal, but if all you are seeing is noise then go to the next tip. 
• Check if your audio is set sufficiently loud. It may just be the case that your volume on your device is set to low, simply crank it up to the maximum level and see if it works. Remember to put your volume back down when you are plugging in your headphones again, hearing loss is no fun.