Hand Gesture Controlled Door Lock With ESP32 (OpenCV, Mediapipe, MQTT, Thingsboard)

Everyone avoids recklessly touching objects during the COVID-19 outbreak out of fear of getting infected. So I created a door lock/solenoid in this project that can be operated using hand gestures. 

When you wish to provide someone permission to enter a room, this system will also be quite convenient. This concept could actually be developed once more, perhaps as a combination of several motions to boost the door lock's security.

In addition, this device can also be developed using a webcam or other camera to be placed on a door to make it more compact. For this device to work, it requires an internet connection via wifi and an MQTT broker to send commands to the ESP32.

Hardware:

• ESP32-Cam (or any other ESP)
• FTDI/ESP32-Cam-MB
• Solenoid Door Lock
• 5v 1-channel Relay
• Step-down Regulator (XL4015)
• AC/DC Adapter 12v
• DC Barrel Jack to 2-pin Terminal Block Adapter
• Jumper wires
• Breadboard
• Soldering Iron (optional)
• Tin (optional)

Software:

• Arduino IDE
• Pycharm (or any other IDE)
• Mediapipe
• OpenCV
• MQTT

Installing the Arduino IDE is the first thing you need to do. The ESP32's code will be uploaded using this IDE.

1. The IDE can be downloaded from the official Arduino website at https://www.arduino.cc/en/software. Ensure you obtain the most recent IDE version and the version corresponding to your operating system.
2. The ESP32 board manager needs to be installed next after the IDE has been successfully installed. The following link should be added to the 'Additional Boards Managers URLs' section of the preferences menu:

https://dl.espressif.com/dl/package_esp32_index.json

ESP32 Board

1. Then, you need to go to Tools > Board: > Boards Manager...
2. Search for 'esp' in the boards management tab, simply select 'esp32' from Espressif Systems and install the most recent version.

Library

1. Afterward, we must install the library required for the MQTT connection and Thingsboard. Head to Sketch > Include Library > Manage Libraries...
2. Install 'PubSubClient' by Nick O'Leary and 'ThingsBoard' by ThingsBoard Team.

After installing the Arduino IDE, we must install PyCharm in order to execute the hand gesture recognition system.

1. It is available for download from the PyCharm website at https://www.jetbrains.com/pycharm/download.
2. Choose the community version and your operating system, as well.

Installing OpenCV, Mediapipe, & Paho-MQTT

1. Go to the settings menu after completing the installation procedure and choose "Python Interpreter".
2. To add the required packages, click the "+" button in that menu.
3. Then carry out the installation of the packages as shown in the above images.
4. Try playing around with the version of each package if you later experience a problem executing the software.

Following installing the necessary IDEs, we may proceed to configure the ThingsBoard to display the solenoid/door lock status:

1. Go to https://demo.thingsboard.io
2. Create Account and Login
3. Press the 'Devices' tab
4. Name your device
5. Make your device public
6. Get your device's "Access Token" by going to the settings on it. The ESP32 code will thereafter contain this token.

The module that will serve as the foundation of our system for tracking hands is shown in the following code. The hand will be recognized and tracked using this code. OpenCV and the Mediapipe framework were used to create this code. The following website will provide a comprehensive explanation of this code: https://www.section.io/engineering-education/creating-a-hand-tracking-module/

1. Create a new project in the PyCharm IDE
2. Add this code to the new project

import cv2
import mediapipe as mp
import time


class handDetector():
    def __init__(self, mode=False, maxHands=2, detectionCon=0.5, trackCon=0.5):
        self.mode = mode
        self.maxHands = maxHands
        self.detectionCon = detectionCon
        self.trackCon = trackCon

        self.mpHands = mp.solutions.hands
        self.hands = self.mpHands.Hands(self.mode, self.maxHands,
                                        self.detectionCon, self.trackCon)
        self.mpDraw = mp.solutions.drawing_utils

    def findHands(self, img, draw=True):
        imgRGB = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
        self.results = self.hands.process(imgRGB)
        # print(results.multi_hand_landmarks)

        if self.results.multi_hand_landmarks:
            for handLms in self.results.multi_hand_landmarks:
                if draw:
                    self.mpDraw.draw_landmarks(img, handLms,
                                               self.mpHands.HAND_CONNECTIONS)
        return img

    def findPosition(self, img, handNo=0, draw=True):

        lmList = []
        if self.results.multi_hand_landmarks:
            myHand = self.results.multi_hand_landmarks[handNo]
            for id, lm in enumerate(myHand.landmark):
                # print(id, lm)
                h, w, c = img.shape
                cx, cy = int(lm.x * w), int(lm.y * h)
                # print(id, cx, cy)
                lmList.append([id, cx, cy])
                if draw:
                    cv2.circle(img, (cx, cy), 15, (255, 0, 255), cv2.FILLED)

        return lmList


def main():
    pTime = 0
    cTime = 0
    cap = cv2.VideoCapture(1)
    detector = handDetector()
    while True:
        success, img = cap.read()
        img = detector.findHands(img)
        lmList = detector.findPosition(img)
        if len(lmList) != 0:
            print(lmList[4])

        cv2.imshow("Image", img)
        cv2.waitKey(1)


if __name__ == "__main__":
    main()

Make a different file for the finger counter code after adding the hand tracking module to the project. The hand tracking module will indeed be imported into this code. This code's foundation was taken from the video at https://youtu.be/p5ZGGRCI5s. The MQTT code is then inserted into the updated code. Then, you can try to run this program to check if everything is working.

import time
import cv2
import HandTrackingModule as htm
import paho.mqtt.client as mqtt

mqttBroker = "broker.hivemq.com"  # mqtt broker
client = mqtt.Client("client name")  # choose your own client name
client.connect(mqttBroker)

wCam, hCam = 640, 480

cap = cv2.VideoCapture(0)
cap.set(3, wCam)
cap.set(4, hCam)

detector = htm.handDetector(detectionCon=0.75)

tipIds = [4, 8, 12, 16, 20]

while True:
    success, img = cap.read()
    img = detector.findHands(img)
    lmList = detector.findPosition(img, draw=False)

    if len(lmList) != 0:
        fingers = []

        if lmList[tipIds[0]][1] > lmList[tipIds[0] - 1][1]:
            fingers.append(1)
        else:
            fingers.append(0)

        for id in range(1, 5):
            if lmList[tipIds[id]][2] < lmList[tipIds[id] - 2][2]:
                fingers.append(1)
            else:
                fingers.append(0)

        totalFingers = fingers.count(1)  
        print(totalFingers)

        if totalFingers == 3:  # you can change the value from 0-5. Total number of opened finger
            client.publish("yourtopic", 1)  # choose your topic. This line will publish the data to the MQTT broker
            time.sleep(5)  # interval before sending the data to lock the solenoid
            client.publish("yourtopic", 0)  # choose your topic. This line will publish the data to the MQTT broker

    cv2.imshow("Image", img)
    cv2.waitKey(1)

You can also test whether the program successfully publishes its data to the MQTT broker by running the code below. Please make sure the topic is declared the same way in both codes.

import time
import paho.mqtt.client as mqtt

def on_message(client, userdata, message):
    print("Received message: ", str(message.payload.decode("utf-8")))

mqttBroker = "broker.hivemq.com"  # mqtt broker
client = mqtt.Client("clientname")  # choose your client name 
client.connect(mqttBroker)

client.loop_start()
client.subscribe("yourtopic"). # make sure to have the same 'topic' name as the main code 
client.on_message = on_message

time.sleep(500)

client.loop_stop()

You can try using another free MQTT broker if the code is unable to connect to the MQTT broker. To change the broker, simply change it in the 'mqttBroker'.

After completing the program for hand tracking and finger counting. Now, we need to upload the code to the ESP32.

1. Open Arduino IDE
2. Paste the following codes
3. Go to Tools > Board: " " > ESP32 Arduino > select your ESP32 (in this case, I use the AI Thinker ESP32-CAM)
4. Change the token, WiFi SSID and Password, and MQTT Topic
5. Upload the code to the ESP32 (do not forget to choose the port where your ESP32 is connected)
6. After uploading the code, try to open the serial monitor to see if everything is working fine
7. If everything is working fine, then you will see in the serial monitor like the picture above

#include <Arduino.h>
#include <WiFi.h>
#include <PubSubClient.h>
#include "soc/soc.h"
#include "soc/rtc_cntl_reg.h"
#include "ThingsBoard.h"

#define RELAY 2 //gpio 2 in the esp
#define TOKEN "enteryourtoken" //insert thingsboard token
#define THINGSBOARD_SERVER "demo.thingsboard.io"

const char* wifiSSID = "enter your wifi ssid";
const char* wifiPass = "enter your wifi password";
const char* mqttBroker = "broker.hivemq.com";
WiFiClient client;
PubSubClient mqtt(client);

WiFiClient espClient;
ThingsBoard tb(espClient);


void connectWifi();
void connect_mqtt(); 
void mqttReceive(char *topic, byte*msg, unsigned int msgLength);


void setup()
{
    WRITE_PERI_REG(RTC_CNTL_BROWN_OUT_REG, 0);
    
    pinMode(RELAY, OUTPUT);

    Serial.begin(115200);
    connectWifi();    
    mqtt.setServer(mqttBroker, 1883);
    mqtt.setCallback(mqttReceive);
}

void loop()
{
    if(!mqtt.connected()){
        connect_mqtt();
        Serial.println("MQTT Connected");
    }
    mqtt.loop();
    delay(100);


    if (!tb.connected()) {
    Serial.print("Connecting to: ");
    Serial.print(THINGSBOARD_SERVER);
    Serial.print(" with token ");
    Serial.println(TOKEN);
    if (!tb.connect(THINGSBOARD_SERVER, TOKEN)) {
      Serial.println("Failed to connect");
      return;
    }
  }

  tb.loop();
}


void connect_mqtt(){
    while (!mqtt.connected())
    {
        Serial.println("Connecting to MQTT...");
        if(mqtt.connect("mqtt_test")){
            mqtt.subscribe("yourtopic"); //match the topic from the previous code
        }
    }
    
}


void connectWifi()
{
    Serial.println("Connecting to WiFi");
    WiFi.begin(wifiSSID, wifiPass);
    while (WiFi.status() != WL_CONNECTED)
    {
        Serial.print(".");
        delay(500);
    }
    
    Serial.println("WiFi Connected");
}


int value; 

void mqttReceive(char *topic, byte*msg, unsigned int msgLength){
        
        String data;
        for(int i = 0; i < msgLength; i++){
            Serial.print(char(msg[i]));
            data += (char)msg[i];
        }
        Serial.println("");

        int value = data.toInt();
        digitalWrite(RELAY, 0);

        switch (value)
        {
        case 0:
            digitalWrite(RELAY, 0);
            break;
        
        case 1:
            digitalWrite(RELAY, 1);
            break;
        }
    
        tb.sendTelemetryInt("lockstatus", value); //send data to thingsboard

    }

We may proceed to set up the dashboard in ThingsBoard after obtaining the data in the ESP32 serial monitor.

1. Check 'latest telemetry' to make sure that ThingsBoard receives our data
2. To configure the dashboard, head to Dashboard > + > Create new Dashboard
3. Name your dashboard and add it
4. Open dashboard
5. Enter edit mode by pressing the bottom right button
6. Add new widget
7. Choose 'cards' then 'Simple Card'
8. Create an entity alias and name it
9. Choose your device name
10. Then add the data
11. You can customize the widget to your liking in the settings menu

The next step is to build this circuit using a breadboard or a PCB after building and configuring the ThingsBoard. To avoid a short, make sure to connect every component to its proper location.