Arduino Hand Gesture Control of Your Computer

Hi everyone,

Sensor technology has its own vital role to play in every field. Now talking about Human Machine Interface, it comprises of hardware and software which helps in exchange of data or information between the user (human) and the machine. Human Machine Interface devices are based on sensor technology. Normally, we use switches, LED indicators, touchscreens as a part of HMI devices (Human Machine Interface)

On the other hand, another way to communicate with the machines, is with the help of some gesture recognition system. This system facilitates exchange of information between the user and the machine without any physical contact between them. There are various types of recognition system. Some of them are Face gesture recognition system, hand recognition system, sign language recognition system and so on.

Among these I have implemented a simple Arduino based hand gesture control. Instead of using mouse, keyboard, joystick, we can use hand gesture to control certain function of our computers.

PROJECT DESCRIPTION:

This project uses Python for its implementation. Using this we can control few function of the laptop or computer like

• Play and Pause a video
• Switching between the tabs
• Increase and decrease system volume.
• Mute the video

PRINCIPLE BEHIND IT:

The principle behind this project is simple. This project uses two Ultrasonic Sensors, which, when we wave or move our hands in front of the sensors it calculates the distance between the hand and the sensors. The distance thus calculated from the Arduino is collected by a Python program using a special library called PyAutoGui. This library converts the collected data into keyboard click actions. Using this information, computer performs the relevant task.

COMPONENTS USED:

• Arduino UNO - 1
• Ultrasonic sensors - 2
• USB cables (to connect Arduino with the computer) - 1
• Few connecting wires
• A laptop system with internet connection

CIRCUIT CONNECTION:

The basic circuit of Arduino alone is shown in the image-1. Comprises of Arduino UNO board , 2 Ultrasonic sensors which are connected with few connecting wires.

Arduino Part consisting of Arduino UNO and sensors are attached to the back side of our laptop as shown in the image - 2, for a proper working of the sensors.

ARDUINO PROGRAM

This Arduino program that I have programmed converts the distance measure by both the sensors into the appropriate commands for the controlling certain action.

The hand gesture in front of the Ultrasonic sensors can be calibrated so that they can perform different tasks on your computer. The tasks are

• play and pause a video
• scrolling up and down
• switching between the tabs
• increase and decrease system volume.

Arduino code:

const int trigPin1 = 11; // the number of the trigger output pin ( sensor 1 )
const int echoPin1 = 10; // the number of the echo input pin ( sensor 1 )     
const int trigPin2 = 6;  // the number of the trigger output pin ( sensor 2 ) 
const int echoPin2 = 5;  // the number of the echo input pin ( sensor 2 ) 

//distance calculating variable
long duration;                               
int distance1, distance2; 
float r;
unsigned long temp=0;
int temp1=0;
int l=0;


void find_distance (void);


void find_distance (void)                   
{ 
  digitalWrite(trigPin1, LOW);
  delayMicroseconds(2);
  digitalWrite(trigPin1, HIGH);
  delayMicroseconds(10);
  digitalWrite(trigPin1, LOW);

  duration = pulseIn(echoPin1, HIGH, 5000);
  r = 3.4 * duration / 2;    
  distance1 = r / 100.00;
  digitalWrite(trigPin2, LOW);
  delayMicroseconds(2);
  digitalWrite(trigPin2, HIGH);
  delayMicroseconds(10);
  digitalWrite(trigPin2, LOW);
  duration = pulseIn(echoPin2, HIGH, 5000);
  r = 3.4 * duration / 2;     
  distance2 = r / 100.00;
  delay(100);
}

void setup() 
{
  Serial.begin(9600);
  pinMode(trigPin1, OUTPUT); // trigger and echo pins of both the sensor are intialized as input and output
  pinMode(echoPin1, INPUT);
  pinMode(trigPin2, OUTPUT);
  pinMode(echoPin2, INPUT);
  delay (1000);
    
}

void loop()
{
  find_distance(); // this function will stores the current distance measured by the ultrasonic sensor in the global variable "distance1 and distance2"
  
  if(distance2<=35 && distance2>=15)            // once if we placed our hands in front of the left sensor in the range between 15 to 35cm this condition becomes true.
  { 
    temp=millis();                              // store the current time in the variable temp.
    while(millis()<=(temp+300))                 // this loop measures the distance for another 300 milliseconds. ( it helps to find the difference between the swipe and stay of our hand in front of the left sensor )
    find_distance();
    if(distance2<=35 && distance2>=15)          // this condition will true if we place our hand in front of the left sensor for more then 300 milli seconds. 
    {
     temp=distance2;                            
     while(distance2<=50 || distance2==0)       
     {
       find_distance();                         // call this function continuously to get the live data. 
       if((temp+6)<distance2)                   // this condition becomes true if we moves our hand away from the left sensor (**but in front of it ). here " temp+6 " is for calibration.
       {
       Serial.println("Vup");                   // send "Vup" serially.
       }
       else if((temp-6)>distance2)              // this condition becomes true if we moves our hand closer to the left sensor.
       {
        Serial.println("Mute");                 // send "Mute" serially.
        break;
       }
     }
    }
    else                                        // this condition becomes true, if we only swipe in front of the left sensor . 
    {
      Serial.println("VDown");                  // send "Vdown" serially.
    }
  }

else if(distance1<=35 && distance1>=15)         // once if we placed our hands in front of the right sensor in the range between 15 to 35cm this condition becomes true.
  { 
  
    temp=millis();                           
                              
    while(millis()<=(temp+300))             
    {
       find_distance();
       if(distance2<=35 && distance2>=15)        // usually it happens if we swipe our hand from right to left sensor
       {
         Serial.println("change");               // send "change" serially.
         l=1;
         break;
       }
    }
    
    if(l==0)                                     // this condition will become true, only if we swipe our hand in front of right sensor.
    {
    Serial.println("Play/Pause");                // send "Play/Pause" serially.
    while(distance1<=35 && distance1>=15)        // this loop will rotate untill we removes our hand infront of the right sensor
    find_distance();                      
    }
    l=0;                                         // make l=0 for the next round.
   }
   
}

PYTHON PROGRAM:

To invoke the keyboard's keypress function using python I have installed a special Python library called PyAutoGui.

Steps to follow:

• Install Python and pySerial.
• Upgrade pip version(In Python latest version pip will already be installed)

1. to check whether pip is installed or not use

pip -V

2. To upgrade

python -m pip install -U pip

• Install PyAutoGui

python -m pip install pyautogui

PYTHON CODE:

Arduino code is programmed in such a way that it writes the operation it performs on the Python console. Python program reads the operation and performs the specific task.

Python code:

<strong>import serial                                      # add Serial library for serial communication
import time
import pyautogui                                   # add pyautogui library for programmatically controlling the mouse and keyboard.

ArduinoSerial = serial.Serial('com3',9600)         # Initialize serial and Create Serial port object called Arduino_Serial
time.sleep(2)					   #sleep time is given as 2 millisecond
while 1:
    incoming_data = str (ArduinoSerial.readline()) # read the serial data and print it as line
    print (incoming_data)                          # print the incoming Serial data
     

    if 'Play/Pause' in incoming_data:
        pyautogui.typewrite(['space'],0.2)
        
    if 'VDown' in incoming_data:
        pyautogui.typewrite(['f8'],0.2)
       
    if 'Mute' in incoming_data:
        pyautogui.typewrite(['f7'],0.2) 
    
    if 'Vup' in incoming_data:
        #pyautogui.press('down')
        pyautogui.typewrite(['f9'],0.2) 
        
    if 'change' in incoming_data:
        pyautogui.keyDown('alt')  
        pyautogui.press('tab')
        pyautogui.keyUp('alt')
        
    incoming_data = "";

</strong>

GESTURES:

Gesture 1: Move your hand towards the right sensor to Play or Pause the video.

Gesture 2: Move your hand towards the left sensor to decrease the volume of the video.

Gesture 3: Move your hand away from the left sensor to increase the volume of the video.

Gesture 4: Place your hand in front of the left sensor for over 30 cm away to mute the video.

Gesture 5: Swipe your hand from right to left sensor to have the change between the tabs.

CONCLUSION:

Arduino code converts the gestures into 6 commands, which is then processed by the Python Code to perform specific keyboard and mouse click function. I have implemented a simple mini project where I can play/pause, increase /decrease the volume of the video and change between the tabs. You can also try to change the code and execute some other functions you desire. Happy coding..!!

_____________________________________