Real Time Finger Counter

Hello everyone ! In this instructable you will learn how to design a code for counting fingers in real time. This is my first published instructable. We are going to push this project to the next level by making the arduino speak using 'Talkie' library. So basically, we will write a python code in Pycharm that will count the number of fingers that are up and then send the data to the arduino which will be connected to a speaker. Though the project may seem a bit lengthy, it is very easy and it will be fun to make !

Software Required :-

• Pycharm
• Arduino IDE

Hardware Required :-

• Breadboard
• Jumper wires
• Arduino UNO board
• Transistor
• 9V DC battery
• 8 ohm speaker
• Webcam / Smart phone

Installing packages in Pycharm :-

• Open your Pycharm project where you will be writing your code.
• click on 'File' in the top-left corner of the window.
• Then click on settings.
• Click on 'Project : XXX' where XXX is the name of your project.
• Then click on Python interpreter.
• Click on '+' sign just above the 'package' column
• A pop-up window will appear and In the search bar, type 'opencv-python' and install the package.
• Then type 'mediapipe' and install it. (We will be using mediapipe from google)
• Lastly, type 'pyserial' in the search bar and install the package.

Installing Talkie library in Arduino IDE :-

• Open your Arduino IDE.
• Click on 'sketch' in the top-left corner of the window.
• Go to include library and then click on manage libraries.
• A pop-up will appear and then in the search bar, type 'Talkie'.
• Install the library.

If you already have a webcam, connect it to the computer / laptop. If you don't have a webcam, you can use your smart phone as a webcam. You just need to download XSplit VCam in your phone and computer. For more information, click on this link XSplit VCam Connect.

NOTE:- to use your smart phone as a webcam via XSplit VCam, you need stable WI-FI.

So basically, we get frames as input from our webcam and we will pass the images of the frames as img. We will pass img to the functions 'findHands()' and 'findPosition()' of the class 'handDetector()'. findHands() will detect the hands and findPosition() will output the coordinates of our finger joints. We will use these coordinates to determine whether the finger is closed or not. The coordinates of the finger joints are stored inside a list called lmList. first we need to look at the attached pic of the numbering of finger joints.

This numbering is same for left and right hands. In order to check whether a finger is up or down, we need to access the Y coordinate of the finger joints. For eg :- We want to check whether the index finger is up or not. The concept is that, if the tip of the index finger(8) is lower than point 6, the finger is closed i.e. if Y coordinate of point 8 is more than point 6, index finger is closed. However, this doesn't hold true for the thumb. To determine whether the thumb is closed or not, we need to check if the X coordinate of point 4 is less than point 3 or not (in case of right hand). if X coordinate of 4 < X coordinate of 3, thumb is open else it is closed. The reverse occurs in case of left hand thumb.

To determine whether a hand is right or left, we will be using a very simple concept. The webcam will mirror our hand and there will be a lateral inversion. Hence, if the pinky finger is on the right of ring finger, it is a right hand else it is a left hand. That means, if X coordinate of point 20 > X coordinate of point 16, it is right hand, else it is left hand.

You can download the code by clicking on the attached file.

NOTE :- the python code will only calculate the number of fingers up in one hand at a time(left or right)

NOTE :- if only one webcam is connected to your computer, you shall write 0 inside the braces of cv2.VideoCapture(). If you are using multiple webcams, write 1 inside the braces.

NOTE :- You need to check the port name of your arduino board. For eg:- the port name of my arduino uno is 'COM4'. You can check yours by opening Arduino IDE ----> click on 'Tools' -----> port . In your python code, write the port name that is being shown in the ports option (Eg :-COM4 or COM3 ).

If we connect the speaker to the arduino using only a resistor, the output sound is very low. Thus, we need an amplification. You can use the LM386 IC to create an amplifier circuit for the speaker but I used an external 9V DC supply to amplify the output sound (I don't have a LM386 IC currently so... ). If you want, you can look up the attached circuit diagram.

Before you run the python code, I suggest you to upload the arduino code first. The arduino code is very simple. It will just receive data from the python and send output according to the data received. Download the Arduino code from the attached file.

NOTE:- make sure that you have connected the arduino to the computer throughout the execution using the USB cable, otherwise arduino won't receive any data.

After completing the above steps, upload the arduino code in your UNO board and then run the python code. Make sure that your webcam is connected to the computer. If you have done every step correctly, you should get a successful output. If you get an error, don't worry. You can comment your error so that I can help you. Always remember that failure is the mother of success. Few weeks ago, I could not even capture a video using openCV ! I came across many errors but I did not give up. So, you should try the same :) . Don't give up !

Cons of the project :-

• Cannot count the total number of fingers shown by 2 hands simultaneously.
• the hand tracking process may get disturbed due to incorrect positioning of the palm.
• the voice from the speaker is not that loud and the sound is a little bit distorted.

If you need any help in eliminating errors in the project, you can comment below your problems. You can also comment on what you think about my project. Thank You !