Dancing TWIP Robot

For a minimum viable product, our robot-computer system should be able to listen to a piece of music, identify the beats, and move in time with the beats with minimal delay.

1. make a recording of music that is playing
2. detect the tempo of the music being played
3. analyze the recording
4. generate movement in relation to the tempo
5. dance in time with the continuing music.

Our next goal is to have the robot balance during moves (the TWIP aspect of this project).

Further goals include adding more dance moves.

• Elegoo tumbllr robot + cable
• Acrylic stand
• Set of screws etc. for attaching acrylic stand to robot
• Laptop with microphone

The Python program records 10 seconds of audio using the laptop's microphone.

The record() function, which records and analyzes audio, uses the "Audio Recorder" program from here.

Beat detection is also completed within the Python file.

record() creates a list of times at which a beat was detected.

This part of the code modifies sample programs from the librosa documentation as well as code from this website.

The Python program sends information to the Arduino via the Serial Monitor.

The Arduino program started as balance1 from the balance TWIP assignment. It also incorporates the "Arduino Code" program from here.

The Arduino setup and write_read() function uses the "Python Code" program from here.

1) The first issue we ran into was getting the Python program for music analysis and the Arduino program for robot motion to work together. Using PySerial, we were able to get output from the Python program sent to the Arduino Serial Monitor and read by the Arduino program.

1.1) What information should the Python program send to the Arduino program?

• A number relating to time, in order to calculate delay and account for it
• The time at which the first beat occurs, so that the robot can delay for the appropriate time
• The timestamp for the second beat, to calculate the beat interval

We eventually chose to send the delay between the start of the program and the time it began recording, the timestamp for the first beat, and the time interval between beats.

2) How could we eliminate delay?

• We tried using different libraries to get epoch time for the Arduino, so that the Arduino program could compare the time sent by the Python program to its own time in order to determine delay. Choosing to perform most of the calculations in Python made this unnecessary.
• We decided to use the Arduino function millis() which counts the number of milliseconds that have elapsed from the start of the program. Using the value, we subtracted the sum of the time it began recording and the timestamp for the first beat. With the new value, we mod it by the interval to detect when to start the movement.
• After determining when to initially move, we create a loop such that the robot does a dance move every time the interval has elapsed. 
• This led to the problem of the loop sometimes missing an interval. For example, if the number of milliseconds elapsed has an interval of 100 milliseconds, then the elapsed time loop would go from 99 straight to 101. To account for this error, we created an error room of 3 milliseconds.

The robot currently has 2 moves: twisting side to side, and rocking back and forth. The acrylic stands are necessary in order for the robot to remain somewhat upright after rocking. Every time the robot needs to make a move, the Arduino program picks a random number corresponding to the move.

With the addition of code from balance7, an additional move for balancing can be incorporated into the code. The robot will alternate balancing with one of the predefined moves, which is either twisting or tipping to one side (rocking back and forth doesn't work well with balance).

One issue we ran into was that the code for balancing had a processing time longer than the beat interval. As a result, every time the robot chose to balance, it would not move for several beats before it finally balanced. This was resolved by eliminating test statements in the Arduino code that were printing to the Serial Monitor.

The first two videos are from before balancing was added as a move. The third video (in which the robot dances to "Circle of Life") shows balancing and is our final result.

With our current progress, the robot can attempt to analyze any kind of music, but the robot has a better response to music with certain qualities.

• The tempo has to stay consistent throughout the piece
• A percussion element is necessary for the program to "hear" the beat
• Percussion elements have to sound on the main beats

When we added balancing, this limited us to using songs slow enough that the robot would have time to balance in between beats.

Our project uses the following libraries, and code from the following sites:

• pyaudio (for recording)
• https://www.thepythoncode.com/article/play-and-record-audio-sound-in-python 
• librosa
• https://librosa.org/doc/latest/index.html
• https://www.analyticsvidhya.com/blog/2018/02/audio-beat-tracking-for-music-information-retrieval/
• PySerial
• https://create.arduino.cc/projecthub/ansh2919/serial-communication-between-python-and-arduino-e7cce0 

Test music was downloaded from https://www.classicalarchives.com/prs/free.html

We consulted the following resources when coming up with the Python-Arduino communication:

• https://forum.arduino.cc/t/how-to-make-arduino-read-txt-file-on-pc/574410/8 (showing that it's possible)
• https://arduinogetstarted.com/tutorials/arduino-serial-monitor (examples)
• https://www.oreilly.com/library/view/arduino-cookbook/9781449399368/ch04.html (examples)
• https://realpython.com/arduino-python/#hello-world-with-arduino-and-python (alternate method)

And when learning how to use librosa:

• https://medium.com/@patrickbfuller/librosa-a-python-audio-libary-60014eeaccfb

The Arduino code is based off of balance1, which was provided for the lab assignment, and balance7:

• http://www.cs.cmu.edu/~cga/controls-intro/lab1/balance1.zip
• http://www.cs.cmu.edu/~cga/controls-intro/lab1/balance7.zip
• as well as the Arduino code from the PySerial website