The EY3 : Assistive Technology for the Visually Impaired

The purpose of this project was my thesis to develop an assistive device for the visually impaired.

Living Human

3D Printer and Filament

Soldering Kit

22-26 gauge wire

Project Parts:

1 Adafruit ESP32 Feather V2 - 8MB Flash + 2 PSRAM 19.95

1 3.7v 500mAh Lithium Ion Polymer Battery 7.95

2 VL53L4CX Time of Flight Distance Sensors 14.95 each

1 PCA9546 Multiplexer 3.95

1 Mini Oval Speaker - 8 Ohm 1 Watt 1.95

1 Headphone Jack 0.95

1 PAM8302 Amplifier 3.95

3 Stemma QT - JST SH 4-pin cables (2 x 50mm and 1 x 300mm) 0.95-1.50 each

2 Vibrating Mini Motor Discs 1.95 each

1 Red and 1 Green 5mm LED 4.95

2 Tactile Buttons 2.50

Software:

Arduino IDE

Drivers:

Chipset Drivers for Adafruit Feather

STM32duino VL53L4CX for ToF sensors

pitches.h (provided below)

Adafruit MMC56x3 for Magnetometer

Getting Started:

Install the Chip Driver for the Adafruit feather

Install the Boards Manager by placing this link in the "Additional Boards Manager URLs" of the Preferences menu in Arduino IDE: https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_dev_index.json

(File > Preferences> link goes in there)

Now, go to Tools > Boards Manager > Install the "esp32" board manager

Go to Manage Libraries and Install "Adafruit Testbed"

Plug the Feather charging cable into a computer

Ensure that the device appears connected under Tools > Port > (COM #)

Upload the "Blink" example sketch to ensure that the Feather microcontroller is able to be programmed

Plug in the I2C multiplexer to the I2C port on the Feather

Plug the 2 ToF sensors into the ports 0 and 1 on the multiplexer

Run I2C Scan from the Adafruit Testbed Examples to show that the I2C modules are all connected

Tinkercad/Fusion 360 downloadable, printable models

Download and print the body and lid

This sketch also included an MMC56X3 but I couldn't get the codes to work alongside the Time of Flight sensors. Provide feedback below if you know how to fix that part, or else, you could ignore the magnetometer module in this sketch.

Wire and solder the following parts together. It may be useful to test the wiring connections of each piece individually before combining everything together. I ordered a large back of vibration motors on Amazon and several of them were faulty. It would have wasted a ton of time if I had soldered them on before finding out they didn't work.

Copy my code below or download it using the file at the bottom. I added pitches.h, which will need to be in the folder with the Arduino IDE file (.ino) to read the pitches library.

Upload this sketch to the Feather that is now wired and soldered.

Button 2 toggles the magnetometer off and on because it interferes with the other modules, so the other button can disable the tof sensors and switch to magnetometer feedback. If an error occurs, the device will need to be restarted, reset, or have the code uploaded to it again.

/*Danielle Biohax (Danielle Muir)
 * Bachelor of Fine Arts in Digital Art (pending Spring 2023)
 * minors: Anthropology, Kugelman Honors Program
 * University of West Florida

 * 
 * BASED ON :
 * 
 * "SPDX-FileCopyrightText: 2023 Liz Clark for Adafruit Industries
 * SPDX-License-Identifier: MIT
 * PCA9546 I2C Multi Sensor Example 
 * Using two VL53L4CD sensors on ports 0 and 1"
 * https://learn.adafruit.com/adafruit-pca9546-4-channel-stemma-qt-multiplexer/arduino
 * 
 * AND
 * 
 * SPDX-FileCopyrightText: 2011 Peter Knight
 * SPDX-License-Identifier: GPLv2
 * "Talkie library"
 * (Vocab_US_TI99) and (Vocab_US_Large)
 * 
 * AND
 * 
 * Danielle Muir - August 2021
 * VooDoo Neopixel Ring Game Button States
 * 
 * AND
 * 
 * STMicroelectronics Example code
 * "VL53L4CX_Sat_HelloWorld.ino"
 * https://learn.adafruit.com/adafruit-vl53l4cx-time-of-flight-distance-sensor/arduino
 * (Copyright Statement Below)
*/
/********************************************************************************
 * @file  VL53L4CX_Sat_HelloWorld.ino
 * @author SRA
 * @version V1.0.0
 * @date  16 March 2022
 * @brief  Arduino test application for the STMicrolectronics VL53L4CX
 *     proximity sensor satellite based on FlightSense.
 *     This application makes use of C++ classes obtained from the C
 *     components' drivers.
 ******************************************************************************
 * @attention
 *
 * COPYRIGHT(c) 2022 STMicroelectronics
 *
 * Redistribution and use in source and binary forms, with or without modification,
 * are permitted provided that the following conditions are met:
 *  1. Redistributions of source code must retain the above copyright notice,
 *   this list of conditions and the following disclaimer.
 *  2. Redistributions in binary form must reproduce the above copyright notice,
 *   this list of conditions and the following disclaimer in the documentation
 *   and/or other materials provided with the distribution.
 *  3. Neither the name of STMicroelectronics nor the names of its contributors
 *   may be used to endorse or promote products derived from this software
 *   without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
 * AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
 * DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
 * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
 * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
 * SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
 * CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
 * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
 * OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 *
*/
#include <Arduino.h>
#include <Wire.h>
#include <vl53l4cx_class.h>
#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include <stdint.h>
#include <assert.h>
#include <stdlib.h>

#define PCAADDR 0x70
#define DEV_I2C Wire
#define SerialPort Serial


// Components.
VL53L4CX sensor_vl53l4cx_sat1(&DEV_I2C, A1);
VL53L4CX sensor_vl53l4cx_sat2(&DEV_I2C, A1);


// Talkie library (Vocab_US_TI99)
// Copyright 2011 Peter Knight
// This code is released under GPLv2 license.
//
// The following phrases are derived from those built into the
// Texas Instruments TI99/4A Speech System add-on from 1979.
//
// A deep male voice with a southern USA accent.
//
// Due to the large vocabulary, this file takes up 32Kbytes of flash.
// It will not fit in most Arduinos as is, so just copy and paste
// out the words you need.
//
// Note that some words/letters are repeated with different spellings.
// eg. 'TWO', 'TO', 'TOO' or 'YOU' and 'U'

#include "Talkie.h"
#include "Vocab_US_TI99.h"

// Talkie library (Vocab_US_Large)
// Copyright 2011 Peter Knight
// This code is released under GPLv2 license.
// Sound is output on digital pin 3 and/or 11. It can drive headphones directly, or add a simple audio amplifier to drive a loudspeaker.
//
// Armin Joachimsmeyer 11/2018 converted to .c and .h files
//  and made them unique and named duplicate words with _1.
//
// The following phrases are derived from VM61002/3/4/5 ROMs
//
// A male voice with a US accent.
//
// These phrases have a very military bias
// with lots of very useful engineering words.
// They also have good expression.

#include "Vocab_US_Large.h"

Talkie voice;


/*
 Melody (toneMelody Example library)

 Plays a melody

 circuit:
 - 8 ohm speaker on digital pin 8

 created 21 Jan 2010
 modified 30 Aug 2011
 by Tom Igoe

 This example code is in the public domain.

 http://www.arduino.cc/en/Tutorial/Tone
*/

#include "pitches.h"

/*Modified From:
 *August 2021 Danielle Muir
 * "NeopixelRingButtonVooDoo8NowWithSound.ino"
 * from final project in Summer Internship with AdventureVue
*/


//button Definitions
#define BUTTON_PIN1 14 //back
#define BUTTON_PIN2 15 //front

//button toggle and state Definitions
boolean oldState1 = HIGH;
boolean newState1 = HIGH;
boolean toggle1 = LOW;

boolean oldState2 = HIGH;
boolean newState2 = HIGH;
boolean toggle2 = LOW;
//end VooDoo buttons


#include <Adafruit_MMC56x3.h>

/* Assign a unique ID to this sensor at the same time */
Adafruit_MMC5603 mmc = Adafruit_MMC5603(12345);

void pcaselect(uint8_t i) {
 if (i > 3) return;
 
 Wire.beginTransmission(PCAADDR);
 Wire.write(1 << i);
 Wire.endTransmission();  
}
/* Setup ---------------------------------------------------------------------*/

void setup()
{

//Danielle Biohax LEDs and haptic output setup  
 pinMode(13, OUTPUT);
  pinMode(33, OUTPUT);
 pinMode(12, OUTPUT);
  pinMode(27, OUTPUT);
//end Danielle Biohax


 if (!mmc.begin(MMC56X3_DEFAULT_ADDRESS, &Wire)) { // I2C mode
  /* There was a problem detecting the MMC5603 ... check your connections */
  Serial.println("Ooops, no MMC5603 detected ... Check your wiring!");
  while (1) delay(10);
 }


 /* Display some basic information on this sensor */
 mmc.printSensorDetails();

 mmc.setDataRate(100); // in Hz, from 1-255 or 1000
 mmc.setContinuousMode(true);


/*Modified From:
 *August 2021 Danielle Muir
 * "NeopixelRingButtonVooDoo8NowWithSound.ino"
 * from final project in Summer Internship with AdventureVue
*/

pinMode(BUTTON_PIN1, INPUT_PULLUP);
pinMode(BUTTON_PIN2, INPUT_PULLUP);
//end VooDoo buttons


//Danielle Biohax modified from: 
//2023 Liz Clark and Adafruit Industries "PCA9546 I2C Multi Sensor Example"
Wire.begin();
   
 // Initialize serial for output.
 SerialPort.begin(115200);
 SerialPort.println("Starting Multisensor...");

 // Initialize I2C bus.
 DEV_I2C.begin();

   
  // define the port on the PCA9546 for the first sensor
  pcaselect(0);
  // setup the first sensor

   // Configure VL53L4CX satellite component.
 sensor_vl53l4cx_sat1.begin(); 
  // Switch off VL53L4CX satellite component.
 sensor_vl53l4cx_sat1.VL53L4CX_Off();
 //Initialize VL53L4CX satellite component.
 sensor_vl53l4cx_sat1.InitSensor(0x12);
 // Start Measurements
 sensor_vl53l4cx_sat1.VL53L4CX_StartMeasurement();

   // define the port on the PCA9546 for the 2nd sensor
  pcaselect(1);
  // setup the 2nd sensor
   
 // Configure VL53L4CX satellite component.
 sensor_vl53l4cx_sat2.begin();
 // Switch off VL53L4CX satellite component.
 sensor_vl53l4cx_sat2.VL53L4CX_Off();
 //Initialize VL53L4CX satellite component.
 sensor_vl53l4cx_sat2.InitSensor(0x12);
 // Start Measurements

 sensor_vl53l4cx_sat2.VL53L4CX_StartMeasurement();
}
//end Danielle Biohax modding Liz Clark and Adafruit's "PCA9546 I2C Multi Sensor Example"
void loop()
{

 //Danielle Biohax modified from: 
//2023 Liz Clark and Adafruit Industries "PCA9546 I2C Multi Sensor Example"
 VL53L4CX_MultiRangingData_t MultiRangingData;
 VL53L4CX_MultiRangingData_t *pMultiRangingData = &MultiRangingData;
 uint8_t NewDataReady = 0;

 int no_of_object_found = 0, j;
 char report[64];
 int status;

//Danielle Muir 2021 Voodoo button states
updateButtonState1();
updateButtonState2();


 if ((newState1 == LOW) && (oldState1 == HIGH)) {
 newState1 = digitalRead(BUTTON_PIN1);
 if (newState1 == LOW) {
  toggle1 = !toggle1;
 }
 }
 oldState1 = newState1;

if (toggle1 == HIGH) {
delay(1);

} else {
 // define port on the PCA9546
 pcaselect(0);

 //Danielle Biohax modified from:
 //STMicroelectronics Example code
 // "VL53L4CX_Sat_HelloWorld.ino"
 do {
  status = sensor_vl53l4cx_sat1.VL53L4CX_GetMeasurementDataReady(&NewDataReady);
   
 } while (!NewDataReady);


 if ((!status) && (NewDataReady != 0)) {
   
  status = sensor_vl53l4cx_sat1.VL53L4CX_GetMultiRangingData(pMultiRangingData);
  no_of_object_found = pMultiRangingData->NumberOfObjectsFound;
  snprintf(report, sizeof(report), "ToF1: Count=%d, #Objs=%1d ", pMultiRangingData->StreamCount, no_of_object_found);
  SerialPort.print(report);
  for (j = 0; j < no_of_object_found; j++) {
   if (j != 0) {
    SerialPort.print("\r\n                ");
   }
   SerialPort.print("status=");
   SerialPort.print(pMultiRangingData->RangeData[j].RangeStatus);
   SerialPort.print(", D=");
   SerialPort.print(pMultiRangingData->RangeData[j].RangeMilliMeter);
   SerialPort.print("mm");
 //end Danielle Biohax modding STMicroelectronics Example

  
 //Danielle Biohax Logic for LEDs and haptic motors
if (500>pMultiRangingData->RangeData[j].RangeMilliMeter> 1000){
 digitalWrite(13, HIGH);  
 digitalWrite(12, HIGH);  
   tone(A1, NOTE_A2, 10);
 delay(12);            

} 

else if (pMultiRangingData->RangeData[j].RangeMilliMeter< 500){
 digitalWrite(13, HIGH);  
 digitalWrite(12, HIGH);  
   tone(A1, NOTE_B4, 10);
  
 delay(3);   
   
}
else{
  digitalWrite(13, LOW); 
 digitalWrite(12, LOW); 
 delay(12);
}
//end Danielle Biohax Logic for LEDs and haptic motors
  }
  SerialPort.println("");
  if (status == 0) {
   status = sensor_vl53l4cx_sat1.VL53L4CX_ClearInterruptAndStartMeasurement();
    
  }
   
 }

  
pcaselect(1);

 //Danielle Biohax modified from:
 //STMicroelectronics Example code
 // "VL53L4CX_Sat_HelloWorld.ino"
 do {
  status = sensor_vl53l4cx_sat2.VL53L4CX_GetMeasurementDataReady(&NewDataReady);
   
 } while (!NewDataReady);


 if ((!status) && (NewDataReady != 0)) {
  status = sensor_vl53l4cx_sat2.VL53L4CX_GetMultiRangingData(pMultiRangingData);
  no_of_object_found = pMultiRangingData->NumberOfObjectsFound;
  snprintf(report, sizeof(report), "ToF2: Count=%d, #Objs=%1d ", pMultiRangingData->StreamCount, no_of_object_found);
  SerialPort.print(report);
  for (j = 0; j < no_of_object_found; j++) {
   if (j != 0) {
    SerialPort.print("\r\n                ");
   }
   SerialPort.print("status=");
   SerialPort.print(pMultiRangingData->RangeData[j].RangeStatus);
   SerialPort.print(", D=");
   SerialPort.print(pMultiRangingData->RangeData[j].RangeMilliMeter);
   SerialPort.print("mm");

 //end Danielle Biohax modding STMicroelectronics Example

  
 //Danielle Biohax Logic for LEDs and haptic motors
if (500>pMultiRangingData->RangeData[j].RangeMilliMeter> 1000){
 digitalWrite(33, HIGH);  
 digitalWrite(27, HIGH);  
   tone(A1, NOTE_D2, 10);
 delay(12);            

} 

else if (pMultiRangingData->RangeData[j].RangeMilliMeter< 500){
 digitalWrite(33, HIGH);  
 digitalWrite(27, HIGH);  
        tone(A1, NOTE_E4, 10);
         
 delay(3);    
}
else{
  digitalWrite(33, LOW); 
 digitalWrite(27, LOW); 
 delay(12);
}
//end Danielle Biohax Logic for LEDs and haptic motors
  }
  SerialPort.println("");
  if (status == 0) {
   status = sensor_vl53l4cx_sat2.VL53L4CX_ClearInterruptAndStartMeasurement();
  
   
  }
 }
}

    
//button 2 RED
if ((newState2 == LOW) && (oldState2 == HIGH)) {
 newState2 = digitalRead(BUTTON_PIN2);
 if (newState2 == LOW) {
  toggle2 = !toggle2;
  }
 }
 oldState2 = newState2;

 if (toggle2 == HIGH) {
sensors_event_t event;
 mmc.getEvent(&event);
 Serial.print("X: ");
 Serial.print(event.magnetic.x);
 Serial.print(" ");
 Serial.print("Y: ");
 Serial.print(event.magnetic.y);
 Serial.print(" ");
 Serial.print("Z: ");
 Serial.print(event.magnetic.z);
 Serial.print(" ");
 Serial.println("uT");

 if( event.magnetic.y > 80){
      voice.sayQ(sp5_INBOUND); 
      delay(10);
  }
 
 } else {

 }
   
//end Danielle Muir 2021 Voodoo
  

}

 //end Danielle Biohax modding Liz Clark and Adafruit's "PCA9546 I2C Multi Sensor Example"


 
//button sequence
void updateButtonState1() {
 newState1 = digitalRead(BUTTON_PIN1);
}

void updateButtonState2() {
 newState2 = digitalRead(BUTTON_PIN2);
}