Morse Code Wrist-Device for Texting for Visually Impaired People - Haptic Interfaces Project by Minh Quan Pham and Izabela Trepacova

I/ Problem:

We tend to take for granted how easy it is to perform short, long-distance communication for normal people: we could just text others. However, this is not so easy for blind people, and even worse for deaf-blind people. Currently, blind people have to rely on voice-to-text dictation to be able to send text messages and the reverse for receiving texts. This method provides little privacy, especially when sending messages, and does not help deaf-blind people at all. With this makeshift watch, we aim to use haptic technology to make a private, standardized method of texting between people in cases where at least one person is visually impaired.

II/ State of the art:

As mentioned above, currently, the most popular method of texting among blind people is voice-to-text dictation and vice-versa [1]. This method involves the blind person talking into a microphone, and software on their device (e.g. VoiceOver for iPhones) will transcribe their spoken words into texts. Technologies are being developed to further fine-tune this technology to accurately transcribe different languages and accents as well as the reverse: dictating bland texts into human-like speeches [2,3]. However, this method is only helpful for people who only lose their sight and provides little privacy as the user still has to speak clearly to get the correct message, even with the best voice-to-text algorithm. As for those who prefer more private texting and those who are blind-deaf: losing both eyesight and hearing, this technology is redundant. Some other methods of texting for blind people include a braille display device, which provides users with a physical display of braille characters on a screen [4]. With this method, not only texts can be displayed, but also its layout, allowing the display of the entire screen to the braille reader [4,5]. The blind user can also use the braille devices to write on the screen with braille characters, which will be transcribed into normal texts [4]. Most braille devices operate this way without the need for voice inputs, and voice outputs is usually optional [4], meaning there is more privacy when writing and sending messages in public. However, these braille devices are usually very cumbersome, and expensive, often costing more than a normal smartphone or a laptop [6], and a lot of people will have to learn to read and write in braille [7]. Another solution is the digital braille keyboard, like the Android TalkBack keyboard. This is a built-in keyboard in the Android operating system that lets users use the braille keyboard to type and interact with the screen [8]. This is very convenient for the user since they do not need any further software installations or any extra attached devices. However, the keyboard takes up the entire screen and sometimes requires the user to hold the phone differently than normal to type out messages. Not only that, the only feedback that the user receives is the screen reader saying out loud what is being typed, and any mistakes in the braille will not be able to be deleted like normal texting [9].

Taking all of these into account, we have determined a cheap and portable solution for helping blind and deaf-blind people text. We have made a Morse code texting device that is worn around the wrist like a watch. Ours is a cheaper, wearable, and more simplified version of an already existing system developed in India [10], with only one output actuator needed to avoid confusion, unlike in the existing version where the motors are also actuated when a message is being typed with the push buttons in addition to also being the output Morse code, which can cause some confusion for the user. Due to severe time constraints, advanced functions such as integration with actual mobile devices will not be implemented, unlike the Indian system which does have integration with GSM [10]. There is a possibility to make the device portable by using a different Arduino [11], but we currently do not have access to them and the programming is too complicated for us.

1 PC with CAD modeling software and Arduino IDE installed

( A 3D printer) or someone who offers a 3D printing service

2 NO (normally opened) push-buttons

1 self-designed 3D-printed watch face

1 TacHammer Drake LFi (Low-Frequency impact ) haptic actuator with leg block

1 Arduino Micro

1 breadboard

1 Haptic drive: DRV2605L

Breadboard compatible wires

Elastic wristband

Sewing kit

To mount our push buttons on a surface, we need 2 holes whose diameters are both 13 mm on the watch face, one for each button. The size of the watch face should be small enough to wear comfortably on the user's wrist, and it should be made of two different parts to make it easy to assemble buttons and wiring. The watch band can be of any material, as long as attaching the watch face to it is possible.

Mount the buttons in the holes made in the previous step. The motor is wrapped between two wristband layers as shown in the pictures. The wristband's total length depends on the size of the intended user's wrist and has to be made tight enough to feel the Tachammer's vibrations.

The wiring diagram above shows part of the circuit. One of the two terminals of both buttons is connected to the ground pin. The other terminal of one button is connected to pin A0 (digital pin D18), and the other to pin A1 (digital pin D19). The haptic driver DRV2605L operates on I2C protocol, so the SCL, SDA pins of the drive are connected to the corresponding pins of the Arduino Micro. The ground pin of the drive is connected to the ground of the Arduino, and the power pin of the drive (Vin) is connected to the +3V3 pin of the Arduino. The INT pin of the drive is connected to pin D13 of the Arduino. The TacHammer is not shown on the fritzing diagram because there is no fritzing file for it. In reality, it is connected via a leg block whose terminals are connected to the two motor outputs of the drive DRV2605L as shown in the second picture.

Once all connections, mounting, and sewing in the previous steps are finished, we should have the above setup.

In the Arduino IDE, first import the <Wire.h> library and define the pin for PWM modulation, in this case, we use pin D13. Then we setup the I2C communication in the next two lines.

#include <Wire.h>
#define PWM13       OCR4A // pin 13 with PWM modulation
//  SETUP I2C
byte DRV = 0x5A;    //DRV2605 slave address - uniq adress
byte ModeReg = 0x01;

After this, we configure the PWM signal that will be driving the TacHammer actuator:

// Configure the PWM clock
#define PWM12k  5   //  11719 Hz

void pwm13configure() {
  // TCCR4A configuration
  TCCR4A = 0;

  // TCCR4B configuration
  TCCR4B = PWM12k;

  // TCCR4C configuration
  TCCR4C = 0;

  // TCCR4D configuration
  TCCR4D = 0;

  // TCCR4D configuration
  TCCR4D = 0;

  // PLL Configuration
  // Use 96MHz / 2 = 48MHz
  PLLFRQ = (PLLFRQ & 0xCF) | 0x30;
  // PLLFRQ=(PLLFRQ&0xCF)|0x10; // Will double all frequencies

  // Terminal count for Timer 4 PWM
  OCR4C = 255;

  //pwmSet6();
  pwmSet13();
}


// Set PWM to D13 (Timer4 A)
void pwmSet13() {
  OCR4A = 0;  
  DDRC |= _BV(7); 
  TCCR4A = 0x82; 
}

Then we need to initialize the actuator drive:

void initializeDRV2605() 
{
  Wire.beginTransmission(DRV);
  Wire.write(ModeReg);            // sets register pointer to the mode register (0x01)
  Wire.write(0x00);               // clear standby
  Wire.endTransmission();

  Wire.beginTransmission(DRV);
  Wire.write(0x1D);              // sets register pointer to the Libarary Selection register (0x1D)
  Wire.write(0xA8);              // set RTP unsigned
  Wire.endTransmission();

  Wire.beginTransmission(DRV);
  Wire.write(0x03);
  Wire.write(0x02);              // set to Library B, most aggresive
  Wire.endTransmission();

  Wire.beginTransmission(DRV);
  Wire.write(0x17);               // sets full scale reference
  Wire.write(0xff);               
  Wire.endTransmission();

  Wire.beginTransmission(DRV);
  Wire.write(ModeReg);               // sets register pointer to the mode register (0x01)
  Wire.write(0x03);               // Sets Mode to pwm
  Wire.endTransmission();

  delay(100);

}

Now, we can setup the program with the void setup() function:

void setup() {
  Wire.begin(); // 
  Serial.begin(9600);
  Serial.setTimeout(1000); // read maximum timeout
  pwm13configure();
  delay(2);
  initializeDRV2605();  //initializes DRV2605 - driver
}

We will add more code to this setup() function later on when we program the buttons in the next steps. For now, the actuator driver setup is complete, and we can proceed with defining some functions that the TacHammer can carry out.

Controlling the PWM signal with i2c protocol:

We define two functions: standbyOnB() and standbyOffB() to control the standby and active mode of the driver. Doing this helps manage the power consumption of the driver:

void standbyOnB() 
{
  Wire.beginTransmission(DRV);
  Wire.write(ModeReg);            // sets register pointer to the mode register (0x01)
  Wire.write(0x43);               // Puts the device pwm mode
  Wire.endTransmission();
}

void standbyOffB() 
{
  Wire.beginTransmission(DRV);
  Wire.write(ModeReg);               // sets register pointer to the mode register (0x01)
  Wire.write(0x03);               // Sets Waveform Mode to pwm
  Wire.endTransmission();
}

Functions:

The most basic functions we are going to define first are pulse(), pause() and usdelay():

void pulse(double intensity, double milliseconds) 
{
  int minimumint = 140;
  int maximumint = 255;
  int pwmintensity = (intensity * (maximumint - minimumint)) + minimumint;
  standbyOffB();
  PWM13 = pwmintensity;
  usdelay(milliseconds);
  standbyOnB();
}

pulse() drives the hammer into the other end of the TacHammer. The parameter 'intensity' defines the strength of the pulse, ranging from 0 to 1. The parameter 'miliseconds' defines the length of time that the signal is active. If this parameter is too large, the hammer might hit the other end, creating a sensation of two pulses, which is to be avoided.

void pause(double milliseconds) 
{
  double us = milliseconds - ((int)milliseconds);
  standbyOnB();
  for (int i = 0; i <= milliseconds; i++) 
  {
    delay(1);
  }
  delayMicroseconds(us * 1000);
}

void usdelay(double time) 
{
  double us = time - ((int)time);
  for (int i = 0; i <= time; i++) 
  {
    delay(1);
  }
  delayMicroseconds(us * 1000);
}

With these functions, we can define the vibration() function which can make the TacHammer actuator vibrate how we want it to:

void vibrate(double frequency, double intensity, double duration, int dutycycle) 
{
  int max_hit = 12;
  int min_hit = 1;
  int crossover = 60;
  int hitduration = 10 * dutycycle / frequency;

  boolean hold = false;
  double delayy;
  delayy = (1 / frequency * 1000) - hitduration;
  double timedown;
  timedown = duration * 1000;

  if (duration == 0)
  {
    hold =  true;
  }

  while (hold)
  {
    pulse(intensity, hitduration);
    pause(delayy);
  }

  while (timedown >= 0 && frequency < crossover)
  {
    pulse(intensity, hitduration);
    pause(3);
    pulse(0.002, delayy-3);
    timedown -= (delayy + hitduration);
   }
  
  while (timedown >= 0 && frequency >= crossover) 
   {
     pulse(intensity, hitduration);
     pause(delayy);
     timedown -= (delayy + hitduration);
   }
}

Several parameters are needed for this function. First of all, the parameter 'frequency' determines the frequency of the vibration in hertz (Hz). The 'intensity' parameter defines the strength of the vibration, ranging from 0 to 1. The 'duration' parameter defines the length of time that the vibration can be felt in seconds. Lastly, the 'dutycycle' parameter defines the percentage of a vibration period that the TacHammer is active.

When receiving a message, the user of this device can feel the vibration in Morse code form. In this section, we will simulate the normal message input with string from the Serial Monitor because we do not have a GSM or IoT module to receive messages from other sources, and the output with the haptic vibration of the TacHammer actuator.

First things first, there are some more variables that we need to define, including the pins where the buttons are going to be connected to and the pins' state. We also have to include the Morse code translation library, a string where the dot and dash sequence is saved, a buffer string where we insert the translated message from Morse code, the mode state, saved as a boolean variable "morse_mode", the different time intervals in milliseconds of each factor in Morse code: dots, dashes, and the different spaces, a variable for good code functioning, and initialize some other variables without assigning any values to these.

#define BUTTON_PIN_MORSE 18 
#define BUTTON_PIN_MODE 19

char letters_and_numbers[36]={'A','B','C','D','E','F','G','H','I','J','K','L','M','N','O','P','Q','R','S','T','U','V','W','X','Y','Z','0','1','2','3','4','5','6','7','8','9'};
String morse_code_strings[36]={".-","-...","-.-.","-..",".","..-.","--.","....","..",".---","-.-",".-..","--","-.","---",".--.","--.-",".-.","...","-","..-","...-",".--","-..-","-.--","--..",
"-----",".----","..---","...--","....-",".....","-....","--...","---..","----."};

String morse_string;
String buffer;

bool morse_mode;

const unsigned long dot_interval = 100;
const unsigned long dash_interval = 300;
const unsigned long letter_interval = 1000;
const unsigned long word_interval = 2000;

unsigned long current_millis;
unsigned long prev_millis;

int dot_dash_flag;
int letter_flag;
int word_flag;
int dot_dash_count;
int prev_state_mode = HIGH;

Secondly, we have to set the pins of the buttons to being pull-up inputs and then set the default values of some previously initialized variables. We add these in the void setup() function, in addition to the drive initialization above.

  pinMode(BUTTON_PIN_MORSE, INPUT_PULLUP);
  pinMode(BUTTON_PIN_MODE, INPUT_PULLUP);
  prev_state_mode = HIGH;
  morse_mode = false;
  dot_dash_flag = 0;
  letter_flag = 0;
  word_flag = 0;
  prev_millis = 0;
  morse_string = "";
  buffer = "";
  dot_dash_count = 0; 

We have two push buttons in this system: one for compiling the message to be sent in Morse code form, and the other button is used to change between the modes: the receiving mode (text mode) and the sending mode (Morse mode). The main program is written in the "void loop()" so that the device can perform either task as long as the Arduino is powered on. In this step, the code for text mode is shown below:

void loop() {
  int button_state_morse = digitalRead(BUTTON_PIN_MORSE);
  int button_state_mode = digitalRead(BUTTON_PIN_MODE);

  if(!morse_mode) { // TEXT MODE
    if (button_state_mode == LOW && prev_state_mode != button_state_mode) {
      morse_mode = true;
      Serial.println("MORSE MODE -> WAIT FOR MESSAGE");
      delay(100); 

    } else {
      if (Serial.available() > 0) { // if there is something to read - number of bites
        String str = Serial.readString();
        for (int i = 0; i < str.length(); i++) {
          char c = str.charAt(i);
          if (isWhitespace(c)){ // check space
            pause(word_interval);
          } else {
            fromCharToMorse(c);
          }
          //Serial.print(c); info for us
          pause(letter_interval); 
        }
        Serial.println();
      }
    }

Since "morse_mode" is defined as "false" by default in the previous part, as soon as the program is uploaded to the Arduino, we start in Text mode. The "if-else" structure of this part of the code ensures that the button press is registered only once, and the mode is not changed erratically if one holds down the change-mode button. This mode waits to take in a string input from the Serial Monitor, then translates that string into a string of Morse code. As the program reads the message, every letter is assigned to its Morse representation. To change letters, we pause for 3 times a dot duration, but in this case, we make it 10 to feel the difference in the motor vibration better. Similarly, a space between words is represented by a pause whose duration is usually 7 times the duration of a dot, but in this case, we make it 20 times.

The translation of the written message and the dots and dashes outputs are implemented in the function fromChartoMorse(), shown below:

void fromCharToMorse(char c){
  for(int i = 0;i < sizeof(letters_and_numbers); i++) {
    if(c == letters_and_numbers[i]){
      for (int j = 0; j < morse_code_strings[i].length(); j++) {
        char morse = morse_code_strings[i].charAt(j);
        //Serial.print(morse); info for us
        if(morse == '.'){
          pulse(0.8,100);
        } else if (morse == '-'){
          vibrate(130,0.4,0.3,41);
        }
        pause(100);
      }
    }
  }
}

With this function, the TacHammer will output a pulse for every dot, and vibrate for every dash.

For the visually impaired user to send messages, the mode has to be switched from the default "Text mode" to "Morse mode" by pressing the change-mode button. After that, with the other button, he/she can press and hold the button depending on whether a dot or a dash has to be the next character. The timing of holding the Morse button as well as the duration of the pauses are crucial to compile a coherent message for the system to translate into normal text without errors. This unfortunately requires extensive training on the user's part on how to make a Morse code message.

While inserting the Morse code, every dot and dash string is translated from Morse to normal letters, it happens after checking that the size of the string is larger than 5 or the time between the pushes is longer than the defined pause interval between letters. To translate the string containing dots and dashes we make a function fromMorsetoChar(), shown below:

char fromMorseToChar(String morse_letter){
  for(int i = 0;i < sizeof(letters_and_numbers); i++){
    if(morse_letter == morse_code_strings[i]){
      return letters_and_numbers[i];
    }
  }
  return '*';
}

If the string of dots and dashes is not recognized, a star character is returned instead. Each letter is inserted into a buffer string, empty by default. If the duration between the pushes is longer than the defined pause duration between words (currently 2 seconds), a space will be inserted into the buffer string.

Once the Morse message is completed, the user has to press the change-mode button again to send the translated message saved in the buffer string, and to change back to the message-receiving mode or "Text mode".

Code snippet for this part:

else { // MORSE MODE
    if (button_state_mode == LOW && prev_state_mode != button_state_mode) {
      morse_mode = false; 
      Serial.print('\n'); 
      Serial.print("MESSAGE: "); 
      Serial.print(buffer);
      buffer = "";
      Serial.print('\n'); 
      Serial.println("TEXT MODE -> WRITE MESSAGE");
      delay(100); 
    } else {
      if (button_state_morse == LOW) {
        if (dot_dash_flag == 0) {
          prev_millis = millis();
          dot_dash_flag = 1;
        }
      } else {
        current_millis = millis();
        long diff = current_millis - prev_millis;
        if (word_flag == 1 && diff >= word_interval) { // WORD
          //Serial.print(' '); info for us
          buffer += " ";
          word_flag = 0;
          morse_string = "";
          letter_flag = 0;
          dot_dash_count = 0;
          prev_millis = millis();
          dot_dash_flag = 0;
        }
        if (letter_flag == 1 && (dot_dash_count == 5 || diff >= letter_interval)) { // LETTER
          char letter = fromMorseToChar(morse_string);
          //Serial.print(letter); info for us
          buffer += letter;
          morse_string = "";
          word_flag = 1;
          letter_flag = 0;
          dot_dash_count = 0;
          prev_millis = millis();
          dot_dash_flag = 0;
        }
        if (dot_dash_flag == 1) {
          if (diff >= dash_interval) {
            //Serial.print("-"); info for us
            morse_string += "-";
            dot_dash_count++;
          } else if (diff >= dot_interval) {
           // Serial.print("."); info for us
            morse_string += ".";
            dot_dash_count++;
          }


          letter_flag = 1;
          prev_millis = millis();
          dot_dash_flag = 0;
        }
      }
    }
  }

At the end of the void loop(), after programming everything else in this step and the previous step, we set the previous button state variable with a 50-millisecond delay:

  delay(50);
  prev_state_mode = button_state_mode;
}

Embedded below is the entire program if you do not feel like programming everything yourself.

In the embedded video, you can see a short demonstration of this device's functioning.

REFERENCES:

[1]: https://www.perkins.org/resource/texting-etiquette-low-vision/#:~:text=Blind%20users%20or%20people%20who,use%20dictation%20to%20compose%20messages.

[2]: https://www.techtarget.com/searchunifiedcommunications/tip/AI-drives-new-speech-technology-trends-and-use-cases

[3]: Jeffrey Jin, 18 Jan 2024: https://www.linkedin.com/pulse/latest-advancements-open-source-text-to-speech-tts-technology-jin-4mxac/

[4]: Braille display and readers: https://www.rnib.org.uk/living-with-sight-loss/assistive-aids-and-technology/reading-and-writing/an-rnib-guide-to-braille-displays-for-blind-and-partially-sighted-people/#:~:text=A%20braille%20display%20or%20braille,lines%20to%2080%20cell%20lines.

[5]: https://www.perkins.org/resource/benefits-using-braille-display-emerging-readers/

[6]: Humanware store: https://store.humanware.com/hus/braille-devices/braille-displays

[7]: https://nelowvision.com/what-to-consider-when-choosing-braille-readers/#:~:text=Limited%20accessibility%3A,are%20blind%20or%20visually%20impaired.

[8]: How to use the TalkBack braille keyboard: https://www.youtube.com/watch?v=VExr6R5i1FQ

[9]: https://www.nngroup.com/articles/screen-reader-type-control/

[10]Drisya. M. K, Pooja. P, Priya. K, Sarath Raj, Jyothi engineering College: https://www.irjet.net/archives/V8/i7/IRJET-V8I7505.pdf 

[11] Over-the-air: https://docs.arduino.cc/arduino-cloud/features/ota-getting-started/