DrishtiY

This is the wearable glove that uses AI to help visually impaired people understand their surroundings

I am Parthib Banerjee, a 14-year-old robotics enthusiast (and also a 10th grader). I believe that my latest idea, which I will be showing you guys, will truly matter, especially to the targeted users. This project is built for the visually impaired people. The name of the project is DrishtiY. Drishti is a Bengali (my mother tongue) word which means vision. What it does is it describes the scene that the user points at. Demo video and Fusion 360 3D model are coming up too. Let's get started helping visually impaired people understand their surroundings!

Items that you will need to make the project yourself (You can buy them from amazon, any other site or locally by searching up the names of the components. The reason why I am not giving you the links from where you can buy the parts is because the listings may depend on the region. Approximate USD and INR prices are given):

1. Raspberry Pi Zero W with a SD card (8-16 GB) ~$15 + $10, ~₹1200-1700 + ₹350
2. SD card reader ~$4, ~₹200
3. ESP32-CAM, if it doesn't come with built in USB or USB shield, you will also need FTDI programmer. ~$8-10 + (FTDI (conditional)->) $5, ~₹700 + (FTDI (conditional)->) ₹250
4. Push Button ~$<1, ~₹3
5. 3 Female to Female Jumper Wires ~$3, ~₹90 (pack of 30)
6. One glove ~$4, ~₹110
7. A small piece of foam board ~$1, ~₹40
8. Li-Ion Battery (or any other power source for ESP32-CAM) ~$6, ~₹80
9. Power Bank (or any other power source for the Raspberry Pi Zero W) ~$8, ~₹180
10. USB cable (A to micro B) ~$2, ~₹40
11. Double-sided tape ~$1-2, ~₹30 (roll)
12. USB headphones (I don't have them, so I will show a work around but it would be required for the main method) ~$10, ~₹400
13. OTG cable (If you are using the main method) ~$2, ~₹70

Wire the push button to GPIO 17 and GND of the Raspberry Pi Zero W (using the female to female jumper wires). The end product is shown in the picture. You can take help from the second image.

Download and open Arduino IDE. Download ESP 32 boards package (from espressif) from board manager. Select your ESP32-CAM board in the Arduino IDE interface (If you don't know the model, you will have to do trial and error. Start with Wrover module). Open the example camera web server code by following the image given above. Select your ESP32-CAM board in the code (by removing the hashtag before your specific model from line 13 to line 32). Apply the settings shown in the 2nd image. In the code, also edit ssid and password variables to your Wi-Fi name and password. Connect GPIO 0 with GND, select the port and click upload. After uploading is done, remove the wire and click reset. Note the IP address that will appear in the console. Note that if you don't have a board with USB shield or built in USB port, you will have to wire the board with an FTDI programmer. You can follow the attached YouTube video from my channel to do the same process, if you like videos more than text and 2 still images 😉)

To set up the Raspberry Pi Zero W with Raspberry Pi OS Lite:

1. Install Raspberry Pi Imager from Raspberry Pi's official website.
2. Put SD card in your SD card reader.
3. Use Raspberry Pi Imager to install the OS on the SD card. Select Raspberry Pi Zero in the first field, choose Raspberry Pi OS Lite in the second field and choose your SD card reader with SD card in the last field.
4. Click on NEXT. In the popup, click on edit settings. Set the hostname to something you will remember. Do the same for username and password. Now enter your Wi-Fi SSID, password and WLAN country (basically your country). Go to services and click to enable SSH. Finally, click on save and then yes.
5. Wait for imaging to complete, eject the drive, take out the SD card from reader, put it in the Raspberry Pi Zero W and power the Pi.
6. Note the IP address from a scanner (like Advanced IP Scanner). Access the shell through software like Putty.
7. If you faced any problems in the last 2 steps, you should just attach a monitor and the problem will show itself, in a pretty self-explanatory way. If you still can't figure out the problem, feel free to comment below.

Make sure they (Raspberry Pi Zero W and ESP32-CAM) are on the same LAN.

Steps to make the basic structure:

1. Hot glue gun the foam board to the upper side of the glove.
2. Double side tape the ESP32-CAM to the foam board so that it points forward
3. Double side tape the Raspberry Pi Zero W.
4. Double side tape the push button to the bottom side of the glove in a way so that you can press it with your thumb. Secure the button with tape attached to the wires.

Firstly, after accessing the shell, enter your username and password. Before we start with the coding, we will need to make an environment where our code will be present. Don't worry, I will show you all the commands necessary. No previous configuration steps have to be taken in the OS. You can start from scratch, just after inserting the newly imaged SD card in the Pi.

1. Let's update our package list and install some packages!

1. Optional packages for building python modules

1. Create a project folder (this is optional, but makes the thing clean). We also have to switch to the folder.

1. Let's create a virtual environment and activate it...

1. Let's install some important python libraries too.

1. Finally, let's open the coding environment!

Now, if you guys have a USB headphone lying around, your final product would work even better. If you don't, you just need to run a program on your computer. What it will do is receive the audio files from the pi and play them on the PC. I will give you the following three codes:

1. The USB headphone version of the Pi code (main method)
2. The Wi-Fi transmitter Pi code
3. The Wi-Fi receiver code that will run on your device with music output

If you are using USB headphones, you will also have to install:

Let's start with the method of using the product without USB headphones. Download mainWIFI.py and main-server.py.

Replace the following variables with ESP32-CAM IP that I told you to note before, PC IP where you will run main-server.py. Make an API key for Gemini from Google AI Studio and replace the variable with the KEY.

To find PC IP: (on your PC) (If windows:)

(If Linux/MacOS:)

To get an API key for Gemini:

Google Cloud Console (https://console.cloud.google.com/) -> Select Project -> New Project -> Give Name

Google AI Studio -> Get API Key (https://aistudio.google.com/apikey) -> Create API Key -> Select your created project -> Create API key in existing project

ESP32CAM=""

PC_IP = ""

API_KEY = ''

Copy the final code and paste it into the Nano environment on the Pi. Finally, press control + S and control + X (Ctrl + S to save and Ctrl + X to exit Nano) on the Pi, and its ready for execution! Shutdown the Raspberry Pi Zero W so that you can place your power source in step 6. You can do that by

Now for the lucky ones who have a USB headphone or adapters.

Connect the USB headphone to the Pi through an OTG cable. You can also connect a normal headphone through a USB adapter. The main.py code is for makers with USB headphones. Here, you will only need to replace the ESP32CAM IP variable and the API key variable. Shut down the Raspberry Pi Zero W by following the steps shown in the previous section.

Attach the power source of the ESP32-CAM and the Raspberry Pi Zero W to the foam board. Attach the battery for the ESP module to the module by:

1. positive terminal to the 5V pin
2. negative terminal to the GND pin

Use a power bank or a battery with a power bank module to power the pi (if you want to do safely, otherwise there are other methods). My power bank is really big. Thus, I will rubber band or tape it to my arm. Ideal solution is to either do it with a battery and a power bank module or a really small power bank that you can fit on the foam board. As mentioned afterwards, in the version 2, the power system will be integrated. The space under the glove can't be used as that is left free for the use of the user.

Connect the power bank to Raspberry Pi Zero W's PWR port. While the Raspberry Pi Zero W turns on, make sure to run the main-server.py if you are using the project without the USB headphones. Run the following command on your PC. Make sure you are on the right directory (the one where you kept the file after downloading).

Coming back to he pi shell, enter your username and password and then proceed with these commands to switch to the project folder and activate the virtual environment.

Run the code on your shell connected to your Raspberry Pi Zero W by running the following command.

Now your project is active. You can test it by pointing at something and pressing the button. After waiting for about 15 seconds, you will be able to hear the output from the PC audio output or from the USB headphone depending on which method you are using. The back end process is explained by the given flowchart. Here is the basic manual:

1. Wear the Glove
2. Turn on the Raspberry Pi Zero W and the ESP32-CAM
3. run main-server.py on your PC (ignore this step if you are using the main method)
4. Run the given command
5. Press the button
6. Listen to the output

Demo YouTube Video is attached.

In the video, I wore the glove. I pointed at a chair and pressed the button. After some time, the AI described the whole scene to me. It talked about the chair, the pillow placed on the chair, the background and even some other details that it captured. Some of those details were not even noticed by me.

Direct Link-> https://youtu.be/GrKom1cZzqs

Press "View in 3D" below to look at the 3D model.

1. “SRG Glove” (https://skfb.ly/KJIt) by SrGeneroso is licensed under Creative Commons Attribution (http://creativecommons.org/licenses/by/4.0/).
2. LiPo Battery 3.7v 100mAh by dalidavila (https://www.autodesk.com/community/gallery/project/127300/lipo-battery-3-7v-100mah)
3. Raspberry Pi Zero by schneik-adsk Sr Director Fusion 360 Product Architecture (https://www.autodesk.com/community/gallery/project/26271/raspberry-pi-zero)
4. ESP-32 Cam by T3do (https://www.autodesk.com/community/gallery/project/176252/esp-32-cam)

So, in this section, we are going to talk about why this project matters, not only as a tool but as something that is really innovative. I will give you guys some points that explain why this is a simpler and more approachable solution than existing products.

1. This is much more cost-effective. The project requires very little money, especially when compared to alternatives that may cost hundreds of dollars. The power to see shouldn't be locked to people who have a lot of money. The total cost of DrishtiY is around 80$. Other smart glasses cost around 800$ to 2000$.
2. Most of the devices introduced in this field only read out text that appears in their view. DrishtiY, on the other hand, describes the scene with huge detail.
3. This type of system has not really been introduced to a glove before.
4. The product is very scalable. As you can see in the next section, there are endless opportunities for improvement.
5. Some smart glasses may exist for the same purpose, but they are not as advanced as this (at least most of them). Glasses may also not be a good fit if the user wants to understand views from different angles. For example, if the user wants to understand what's behind them without turning back, they of course can't turn their head 180 degrees. But with the glove, they can just point backwards. Basically, it gives them more control over what they want to "see".

DrishtiY began as a simple idea — to help the visually impaired experience the world in their own way. Now it’s a low-cost, practical step towards giving vision through AI, with bigger possibilities like offline detection and smarter interactions ahead. This isn’t just a project, it’s the beginning of a movement.

It has been a great journey making the project. Thank you for reading the whole thing! Maybe in the future, the project might actually be released to the public, maybe it will change some lives... I would love to hear your feedback and take on improving the prototype. Some future improvements I have in my mind (Planned Upgrades list might be very long because these are not only ideas, they are stuff I am working on and stuff that I will implement in future versions which, I really plan on doing):

1. Better text-to-speech.
2. More accessibility features and better build quality.
3. An offline mode using object detection. It will just read out the present objects. It may use ssd_mobilenet_v2_coco. I even have a sample code ready. (see in repo) (takes 50s for one image compared to 14s-20s of current code)
4. More comfortable to wear. (It is comfortable enough already, but still)
5. Power source integration. (ESP32-CAM and Pi will be powered by the same power source)
6. UART, SPI, or USB image transfer (ESP32-CAM->Raspberry Pi Zero W)
7. Follow-Up questions feature. After AI dictates text, users will be able to ask follow-up questions.
8. Face Recognition to identify the user's close ones.
9. Make it plug and play. Now, you have to run commands before using it. That process is very tedious and has to be done after accessing the shell. Auto run at startup has to be enabled.
10. And plenty of other ideas buzzing in my head!

DrishtiY v2 isn’t far away — until then, this is just my first step towards giving vision through AI. For now, thank you and bye-bye. (for now 😉).

GitHub repo-> https://github.com/ParthBoyCoder/DrishtiY (under development) (future and testing codes available)