Virtual Reality Using Arduino and Processing

This is a very interesting project in which we are going to learn how to implement virtual reality using Arduino and Processing. For most of us, the movie Iron man by Jon Favreau has always been an inspiration to build new things that will make our life easy and more fun. I have personally admired the Techs that are shown in the movie and have always wanted to build something similar to that. So, in this project I have tried to mimic the Virtual reality stuffs that happen in the movie, like we can simply wave our hand in front of the computer and move the pointer to the desired location and perform some tasks.

Here I will show you how you can simply wave your hand in front of webcam and draw something on your computer. I will also show you how you can toggle lights by virtually moving your hand and making clicks with your fingers in the air.

To make this happen we have to leverage the power of Arduino and Processing combined. Most of would be familiar with Arduino, but Processing might be new for you. Processing is an application just like Arduino and it is also Open source and free to download. Using Processing you can create simple system applications, Android applications and much more. It also has the ability to do Image Processing and Voice recognition. It is just like Arduino and is much easy to learn, but do not worry if you are completely new to processing because I have written this tutorial fairly simple so that anyone with interest can make this working in no time.

In this tutorial we are using Processing to create a simple System application which provides us an UI and track the position of our hand using Image processing. Now, we have to make left click and right click using our fingers. To make that happen I have used two hall sensors (one on my index finger and the other on middle finger) which will be read by the Arduino Nano. The Arduino also transmits the click status to the Computer wirelessly via Bluetooth. It might sound complicated but, Trust me; it is not as hard as it sounds. So let us take a look at the materials needed for this project to be up and running.

1. Arduino Nano
2. Hall sensor (A3144) – 2Nos
3. A small piece of magnet
4. Bluetooth Module (HC-05/HC-06)
5. 9V battery
6. Connecting Wires Dot board
7. A pair of gloves
8. Arduino IDE (Software)
9. Processing IDE(Software)
10. A Computer with Webcam and Bluetooth (you can also use external Bluetooth or Webcam for your computer)

The hardware part of this project is very simple and easy to build. The complete schematic is shown above. The Arduino, resistors and the berg stick pins are soldered onto a dot board as shown. The hall sensor and the Bluetooth module is soldered to a connector wire as shown. Once these two sections are ready it can be assembled onto gloves so that it is easy to use. I have used disposable plastic gloves which can be purchased from any medical shop near you. You should make sure that the magnet comes on your thumb finger and the hall sensor 1 and hall sensor 2 should be present before your index and middle finger respectively. I have used duck tapes to secure the components in place. Once the components are assembled it should look something like this.

The purpose of the Processing program is to create a system application which can act as an UI (User interface) and also perform image processing to track a particular object. In this case we track the blue object that we stuck to our gloves above. The program basically has four screens.

1. Calibration Screen
2. Main Screen
3. Paint Screen
4. LED toggle Screen

We can navigate from one screen to another by simply waving our hands and dragging screens on air. We can also make clicks on desired places to toggle LED or even draw something on screen.

You can copy paste the complete Processing program (given at the end) and modify it based on your creativity or simple download the EXE files from here, and follow the following steps to launch the application.

1. Install JAVA in your computer if you have not installed it before
2. Install You Cam perfect on your computer
3. Power up your Arduino and pair your Computer with the Bluetooth Module
4. Launch the application file

If everything goes fine you should be able to notice the LED on your Bluetooth module getting stable and your webcam light going ON. If you have any problems reach me through the comment section and I will help you out. Watch the video to know how to calibrate your application and use it. If you want to modify the code and build more features into this then you can use the following insights of the program The processing IDE can be downloaded from here. If you want to learn more about processing and create more interesting projects then you can visit the tutorials here. Processing has the ability to read Serial data, in this program the serial data is comes from the Bluetooth COM port. You have to select which COM port your Bluetooth is connect to by using this line

port = new Serial(this,Serial.list()[1],9600);

I have selected my 1st COM port which is COM5 in my case (see image above) and I have mentioned that by Bluetooth module runs on 9600 baudrate.

As said earlier processing also has the ability to do image processing, in this tutorial the images are sent inside the sketch using a webcam. In each image we track for a particular object. To know more about this you can visit this tutorial. I have tried my best to explain the program (given at the end) through the comment lines. You can download the files here. If you want to know more about the sketch you can reach me through the comment section and I will help you out.

Once the Hardware and software is ready, wear the gloves and get ready for some action. Now, simply power the Arduino and then launch the Application. The led on the Bluetooth module should go stable. Now it means that your System application has established a Bluetooth link with your Arduino.

You will get the following screen where you have to select the object to be tracked. This tracing can be simply done by clicking on the object. In this case the object is the Blue disc. Now you can move your object and notice that the pointer follows your object. Use a unique colour object and a bright room for best results.

Now touch your thumb finger with index finger and you should see the message “Key 1 Pressed” and the when you press your thumb with middle finger you should see “Key 2 Pressed” this indicates that everything works fine and the calibration is over. Now click on the Done button.

Once the Done button is pressed you will be directed to the main screen where you can paint on air or toggle the LED on the Arduino Board as shown in the Video.

For more information related to this project visit this page.

And don't forgot to subscribe our youtube channel Circuit Digest.

Follow us on Instagram: @circuit_digest