AI-Migo
Meet AI-Migo: Your Friendly AI-Powered ASL Buddy!
Talking to humans when you can't use words sounds exhausting; that's why a friend would be great. One who doesn't feel
AI-Migo is a user-friendly AI companion that can help you learn and practice American Sign Language (ASL) through real-time hand sign recognition, as well as "talking" your heart out with it. If you are a curious student, language aficionado, or just a computer vision aficionado, AI-Migo makes the learning process accessible in sign language more engaging, fun, and accessible!
With YOLOv11 and a dataset that is specially trained for it, AI-Migo can recognize ASL hand signs from A to Z. Just hold up your hand in front of the camera, and if the model is 60% or more confident, it'll let you know which letter you're signing. It's the same as having a friendly friend nearby—hence the name: AI-Migo (AI + Amigo)! (Nice beautiful name play, I know.)
In this tutorial, I'll walk you through how I made it—from gathering data and labeling it to training a model and visualizing predictions. By the end of it all, you'll be able to make your own smart ASL buddy!
Supplies
1 x 64x64 RGB LED Matrix Panel - 2mm Pitch [ID:5362] = $49.95 / €45.95
1 x Diffused Green 5mm LED (25 pack) [ID:298] = $4.00 / €3.68
1 x 5mm Chromed Metal Narrow Bevel LED Holder - Pack of 5 [ID:2176] = $2.95 / €2.71
1 x Diffused Red 5mm LED (25 pack) [ID:299] = $4.00 / €3.68
1 x Through-Hole Resistors - 470 ohm 5% 1/4W - Pack of 25 [ID:2781] = $0.75 / €0.69
1 x Premium Female/Female Jumper Wires - 40 x 6" [ID:266] = $3.95 / €3.63
1 x Premium Female/Male 'Extension' Jumper Wires - 40 x 12" (300mm) [ID:824] = $7.95 / €7.31
1 x Adafruit RGB Matrix Bonnet for Raspberry Pi [ID:3211] = $14.95 / €13.75
1 x 5V 4A (4000mA) switching power supply - UL Listed [ID:1466] = $14.95 / €13.75
1 x Buzzer 5V - Breadboard friendly [ID:1536] = $0.95 / €0.87
1 x Vibrating Mini Motor Disc [ID:1201] = $1.95 / €1.79
1 x Mini External USB Stereo Speaker [ID:3369] = $12.50 / €11.50
1 x Half-Size Breadboard with Mounting Holes [ID:4539] = $10.00 / €9.20
1 x Premium Male/Male Jumper Wires - 20 x 12" (300mm) [ID:1955] = $3.95 / €3.63
suggestion if u dont want to wire 40 cables: 40-pin GPIO Stacking Header - 15mm pins - 2x20 Female = $2,65 / € 2,29
Total: $139.5 / €128,19
All of these are from the shop Adafruit. Other contributors or different times may have different prices.
BOM file <- be sure to look through this!!
The Matrix Setup
1. Prepare the Hardware
Components Needed:
- Raspberry Pi 5 (powered off)
- Raspberry Pi 5 power cable + power supply
- 64x64 RGB LED Matrix panel
- RGB Matrix Bonnet
- Extra 5V power supply (for the matrix)
- Provided cables (grey ribbon, red & black power wires)
2. Solder the Bonnet
- Under the bonnet are 3 layers of components—only solder the bottom 2, not all three.
3. Connect Power Cables
- Red wire ➜ + terminal on the matrix
- Black wire ➜ - terminal (GND) on the matrix
⚠️ Double-check polarity; reversed wires can damage the matrix.
4. Attach Bonnet to Raspberry Pi
- Align the bonnet’s GPIO header with the Pi’s GPIO pins.
- Press down gently but firmly.
5. Connect Grey Data Cables
Single Matrix Setup:
- Use the IN port on the matrix.
- Plug the grey ribbon cable from the bonnet’s output into the IN port on the matrix.
- Ignore the OUT port unless chaining multiple matrices.
6. Power Up
- Connect 5V ≥ 4A to the bonnet.
- Power the Pi using its standard power supply.
- Turn on the Pi.
7. Test Code: Colour Cycle Script
import numpy as np
import time
from PIL import Image, ImageDraw
import adafruit_blinka_raspberry_pi5_piomatter as piomatter
# Matrix dimensions
width = 64
height = 64
# Set up geometry and framebuffer
geometry = piomatter.Geometry(
width=width,
height=height,
n_addr_lines=5,
rotation=piomatter.Orientation.Normal
)
canvas = Image.new('RGB', (width, height), (0, 0, 0))
draw = ImageDraw.Draw(canvas)
framebuffer = np.asarray(canvas).copy()
matrix = piomatter.PioMatter(
colorspace=piomatter.Colorspace.RGB888Packed,
pinout=piomatter.Pinout.AdafruitMatrixBonnet,
framebuffer=framebuffer,
geometry=geometry
)
# Color cycle
colors = [
("Red", (255, 0, 0)),
("Green", (0, 255, 0)),
("Blue", (0, 0, 255)),
("Yellow", (255, 255, 0)),
("Cyan", (0, 255, 255)),
("Magenta", (255, 0, 255)),
("White", (255, 255, 255)),
("Black", (0, 0, 0)),
]
try:
while True:
for name, rgb in colors:
corrected = (rgb[2], rgb[1], rgb[0]) # Swap R/B channels
draw.rectangle((0, 0, width, height), fill=corrected)
framebuffer[:] = np.asarray(canvas)
matrix.show()
print(f"Showing: {name} – RGB: {rgb} → Sent: {corrected}")
time.sleep(2)
except KeyboardInterrupt:
print("Exiting...")
draw.rectangle((0, 0, width, height), fill=(0, 0, 0))
framebuffer[:] = np.asarray(canvas)
matrix.show()
Key Troubleshooting
No Display?
- Recheck all solder joints.
- Ensure the grey cable is fully inserted in the IN port.
- Confirm power is connected to the bonnet, not just the Pi.
Flickering or Glitches?
- Use a power supply that delivers at least 4A.
- GPIO Conflicts?
- Double-check which pins your matrix uses.
- Avoid using the same pins for other devices.
Advanced Notes
Daisy-Chaining Matrices:
- Connect the OUT of matrix 1 ➜ IN of matrix 2.
- In /boot/config.txt, add:
bash
CopyEdit
dtoverlay=rgb-matrix,chain_length=2
Install Required Library:
- Adafruit RGB Matrix Bonnet Setup Guide (also a visual is found of the soldering component)
Safety Tips
- Always power off before making hardware changes.
- Avoid touching pins while the Pi is powered on.
- Don't overheat solder pads—take breaks during soldering.
8. Video for visuality at the end of the document
Outputs + Breadboard
1. Prepare the Hardware
Components Needed:
- Raspberry Pi 5 (powered off)
- Raspberry Pi 5 power cable + power supply
- 64x64 RGB LED Matrix panel (already attached, powered off)
- Provided cables: 9 female-to-male jumper wires, 1 female-to-female jumper
- 2 LEDS (mine are 5 mm in diameter, and also one is green and the other red)
- 2 transistors, 470 ohm
- Breadboard (i used an half one; you can use a big one if u have those)
2. Decide your GPIO pins
- Go to Adafruit RGB Matrix Bonnet Setup Guide to look at which GPIO's are free or not
- I choose for my GPIO pins:
- GPIO pin 18 -> physical pin 12 (Green LED)
- GPIO pin 25 -> physical pin 22 (
- GPIO pin 19 -> physical pin 35
- Choose your 5V and ground; you need one of them each, and the matrix is using the other. I choose these:
- physical pin 4 (5V)
- physical pin 6 (Ground)
3. Attaching '+' and '-'
I prefer to grab a red and brown/blue cable for this since those are mostly used for '+' and '-'. This is pretty simple to do.
- for 5V Grab your (red) female-to-male jumper wire and attach it to the breadboard. There is this red line at the side and a '+' on top and bottom of it. Attaching your wire means the whole stripe as a '+'
- The same goes for your ground, just that it has a black line against the inward side. Attach your wire, and the whole streak is ground.
4. Attaching the two LEDS
- Look at the two metal pins, as you would notice one is shorter than the other. The smallest one is considered the '-' and the longer one is considered the '+' of the LEDs.
- Make sure your raspberry pi and the matrix don't have their charger in and are powered off
- You take the shorter one and put it in one of your holes from the breadboard where the letters are.
- After that, you grab one of the transistors and attach one pin to the same row as your short pin and the other pin in the '-' line
- put the longest one into another row, attach there a jumper wire to your GPIO pin of choice.
- Repeat that for the second LED, or more times if you use more than one.
5. Attaching the buzzer
- Just like the LEDs, the buzzer has a shorter pin and a longer pin. The longer pin is '+' and the shorter pin is '-'.
- Put the buzzer on your breadboard on two different rows
- By where you put the longest one, in the same row, put a wire towards the GPIO pin you use for your buzzer
- by the shortest one, you just attach a Male-To-Male to it towards your '-' line from your board
6. Buzzer and LEDs test.
buzzer_test.py
import RPi.GPIO as GPIO
import time
BUZZER_PIN = 19 # GPIO pin connected to the transistor base via 470Ω resistor
# Set up GPIO
GPIO.setmode(GPIO.BCM)
GPIO.setup(BUZZER_PIN, GPIO.OUT)
try:
while True:
print("beeping")
GPIO.output(BUZZER_PIN, GPIO.HIGH) # Turn buzzer ON
time.sleep(0.5) # Beep duration
print("STOPPED")
GPIO.output(BUZZER_PIN, GPIO.LOW) # Turn buzzer OFF
time.sleep(5) # Pause before next beep
except KeyboardInterrupt:
print("Stopped by user.")
finally:
GPIO.cleanup()
leds_test.py
import RPi.GPIO as GPIO
import time
# Use Broadcom pin numbering
GPIO.setmode(GPIO.BCM)
# Define LED pins (BCM)
LED1 = 18
LED2 = 25
# Setup pins as outputs
GPIO.setup(LED1, GPIO.OUT)
GPIO.setup(LED2, GPIO.OUT)
print("Blinking LEDs on GPIO 18 and GPIO 25... Press Ctrl+C to stop.")
try:
while True:
GPIO.output(LED1, GPIO.HIGH) # LED1 ON (GPIO18)
GPIO.output(LED2, GPIO.LOW) # LED2 OFF (GPIO25)
time.sleep(0.5)
GPIO.output(LED1, GPIO.LOW) # LED1 OFF
GPIO.output(LED2, GPIO.HIGH) # LED2 ON
time.sleep(0.5)
except KeyboardInterrupt:
print("\nExiting...")
finally:
GPIO.cleanup() # Reset GPIO settings
7. Video for visuality at the end of the document
Assemble the Frame (Laser Cut)
For this step, I used a laser cutter to create the frame and housing for the project. If you're using a similar method, feel free to download and use the SVG files I’ve attached, or you're welcome to design your own to fit your style or dimensions. all together for me would be a simple cube from 455 mm x 330mm
Once all your laser-cut pieces are ready:
- Prepare your parts: Lay out each piece and make sure all parts were cut correctly and align as expected. If something is off, lightly sand or recut as needed.
- Assemble the structure:
- Begin by connecting the base and side walls.
- Use wood glue for strong joints. Hold or clamp them in place for a few minutes so they don't slip.
- For extra strength, you can use small nails or brads to pin the sides together after glueing. Just make sure they don’t split the material.
- Let it dry: Allow the glued structure to dry completely before adding any electronics. This helps prevent shifting or damage while handling.
- You can always paint it, whatever you want. i took inspiration from the older Polaroid cameras with the rainbow stripe
Tip: If your frame includes slots or mounting holes for the LED matrix or breadboard, test-fit them now before final assembly.
Model Coding: Capture Hand Gestures
Now that the hardware is built, it's time to start coding the machine learning model!
I used MediaPipe to track 21 keypoints on my hand and saved the data into .csv files. This is how I trained a model to recognise hand gestures for every letter of the alphabet. You can do it your own way, but I’ve included my full process and code below.
1. Capture Keypoints with a USB Camera
This script opens your camera, cycles through every camera port and records the hand landmarks:
import cv2
import mediapipe as mp
import pandas as pd
import os
# === Try to auto-detect USB camera ===
def find_camera():
print("Checking camera indices 0–4. Press 'y' if it's your USB cam, any other key to try next.")
for i in range(5):
cap = cv2.VideoCapture(i)
if cap.isOpened():
print(f"[INFO] Camera index {i} is available.")
ret, frame = cap.read()
if ret:
cv2.imshow("Is this your USB camera?", frame)
key = cv2.waitKey(0)
cv2.destroyAllWindows()
if key == ord('y'):
print(f"[SELECTED] Using camera index {i} as USB camera.")
return cap
cap.release()
print("[ERROR] No suitable camera found.")
return None
cap = find_camera()
if cap is None:
exit()
# === MediaPipe setup ===
mp_hands = mp.solutions.hands
hands = mp_hands.Hands(static_image_mode=False, max_num_hands=1, min_detection_confidence=0.7)
mp_draw = mp.solutions.drawing_utils
# === Setup for labeling ===
alphabet = [chr(i) for i in range(ord('A'), ord('Z') + 1)]
current_index = 0
LABEL = alphabet[current_index]
data = []
save_dir = "Model_V0.2/data"
os.makedirs(save_dir, exist_ok=True)
# Create column headers
columns = [f'{axis}{i}' for i in range(21) for axis in ['x', 'y', 'z']] + ['label']
print(f"Use 'l' and 'r' to change letter. Current label: {LABEL}")
print("Press 's' to save, 'q' to quit.")
while True:
ret, frame = cap.read()
if not ret:
break
image = cv2.flip(frame, 1)
rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
result = hands.process(rgb)
if result.multi_hand_landmarks:
hand = result.multi_hand_landmarks[0]
keypoints = [coord for lm in hand.landmark for coord in (lm.x, lm.y, lm.z)]
keypoints.append(LABEL)
data.append(keypoints)
mp_draw.draw_landmarks(image, hand, mp_hands.HAND_CONNECTIONS)
# Display current label
cv2.putText(image, f'Label: {LABEL}', (10, 40), cv2.FONT_HERSHEY_SIMPLEX, 1.2, (0, 255, 0), 2)
cv2.imshow("Recording", image)
key = cv2.waitKey(1) & 0xFF
if key == ord('q'):
break
elif key == ord('s'):
if not data:
print("No data to save!")
continue
save_path = os.path.join(save_dir, f"{LABEL}.csv")
df = pd.DataFrame(data, columns=columns)
if os.path.exists(save_path):
df.to_csv(save_path, mode='a', index=False, header=False)
else:
df.to_csv(save_path, mode='w', index=False, header=True)
print(f"Saved {len(data)} samples to {save_path}")
data = []
elif key == ord('l'):
current_index = (current_index - 1) % len(alphabet)
LABEL = alphabet[current_index]
print(f"Switched to label: {LABEL}")
elif key == ord('r'):
current_index = (current_index + 1) % len(alphabet)
LABEL = alphabet[current_index]
print(f"Switched to label: {LABEL}")
cap.release()
cv2.destroyAllWindows()
Each letter you record will get its own .csv file in the Model_V0.2/data folder.
2. Merge All Letter Files into One Dataset
Now combine all the individual .csv files into one big dataset for training:
import pandas as pd
import os
folder = r"Model_V0.2/data"
files = [f for f in os.listdir(folder) if f.endswith('.csv')]
combined = pd.DataFrame()
for file in files:
label = file.split(".")[0]
df = pd.read_csv(os.path.join(folder, file), header=None)
df = df.iloc[:, :-1] # Drop any extra last column
df['label'] = label
combined = pd.concat([combined, df], ignore_index=True)
save_path = r"Model_V0.2/hand_dataset.csv"
combined.to_csv(save_path, index=False)
print(f"✅ Combined dataset saved as '{save_path}' with shape {combined.shape}")
3. Train a Random Forest Classifier
Finally, train your gesture recognition model!
import pandas as pd
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
import joblib
import sys
# Load and clean dataset
try:
df = pd.read_csv("Model_V0.2/hand_dataset.csv")
except FileNotFoundError:
print("❌ Dataset file not found.")
sys.exit(1)
# Convert to numeric
df[df.columns[:-1]] = df[df.columns[:-1]].apply(pd.to_numeric, errors='coerce')
df.dropna(inplace=True)
# Features and labels
X = df.iloc[:, :-1]
y = df.iloc[:, -1]
joblib.dump(list(X.columns), "feature_columns.pkl")
# Split and train
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
clf = RandomForestClassifier()
clf.fit(X_train, y_train)
# Evaluate
y_pred = clf.predict(X_test)
print("\n📊 Classification Report:")
print(classification_report(y_test, y_pred))
# Save model
joblib.dump(clf, "gesture_model.pkl")
print("✅ Model trained and saved as 'gesture_model.pkl'")
You can also try other classifiers like MLPClassifier (Multi-Layer Perceptron) if your dataset gets larger!
The Code
Folder Structure and Setup
You should have two separate folders for this project:
- One for your laptop, where the AI model runs. I called this folder AI.
- One for your Raspberry Pi, called RPI.
Since running a full machine learning model directly on a Raspberry Pi isn't ideal (it's slower and limited in resources), we're going to let the laptop do the heavy work, and have the Raspberry Pi send data to it.
We’ll use Sockets for this. Your laptop acts as the server, and the Raspberry Pi is the client—they talk to each other over the local network.
Also important: Your trained model file (called gesture_model.pkl) stays in the AI folder on your laptop.
AI (Laptop) Folder
This contains:
- The collect_date.py file + the data folder
- Format_csv.py file
- hand_gastures.csv (data merged into one .csv)
- The training_model.py file
- Your trained ML model: gesture_model.pkl
- A Python script (like server.py) that:
- Loads the model
- Listens for incoming data from the Raspberry Pi
- Runs predictions
- Sends the result back
server.py code:
import socket
import struct
import cv2
import numpy as np
import mediapipe as mp
import joblib
import pandas as pd
# === CONFIGURATION ===
PORT = 8000
MODEL_PATH = r"{model_path}\gesture_model.pkl"
# Replace {model_path} with the actual path where your model is stored
FEATURE_COLUMNS_PATH = r"{model_path}\feature_columns.pkl"
# Replace {model_path} with the actual path where your feature columns are stored
# === LOAD MODEL & FEATURE COLUMNS ===
clf = joblib.load(MODEL_PATH)
feature_columns = joblib.load(FEATURE_COLUMNS_PATH)
# === SETUP MEDIAPIPE ===
mp_hands = mp.solutions.hands
hands = mp_hands.Hands(static_image_mode=False, max_num_hands=1)
mp_draw = mp.solutions.drawing_utils
# === SETUP SOCKET SERVER ===
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.bind(('', PORT))
server_socket.listen(1)
print("[INFO] Waiting for Pi connection...")
conn, addr = server_socket.accept()
print(f"[INFO] Connected by: {addr}")
conn_file = conn.makefile('rb')
try:
while True:
# Receive frame size
raw_size = conn_file.read(4)
if not raw_size:
break
frame_size = struct.unpack('>L', raw_size)[0]
# Read frame bytes
frame_data = conn_file.read(frame_size)
frame = cv2.imdecode(np.frombuffer(frame_data, dtype=np.uint8), cv2.IMREAD_COLOR)
# Predict
image = cv2.flip(frame, 1)
rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
result = hands.process(rgb)
prediction = '?' # Default if no hand or error
if result.multi_hand_landmarks:
hand = result.multi_hand_landmarks[0]
keypoints = [val for lm in hand.landmark for val in (lm.x, lm.y, lm.z)]
try:
input_df = pd.DataFrame([keypoints], columns=feature_columns)
prediction = clf.predict(input_df)[0]
except Exception as e:
print(f"[ERROR] Prediction failed: {e}")
mp_draw.draw_landmarks(image, hand, mp_hands.HAND_CONNECTIONS)
# Send prediction back to Pi
try:
conn.sendall(prediction.encode('utf-8'))
except Exception as e:
print(f"[ERROR] Could not send prediction: {e}")
# Draw feedback on image
cv2.putText(image, f'Prediction: {prediction}', (10, 50),
cv2.FONT_HERSHEY_SIMPLEX, 1, (0, 255, 0), 2)
cv2.imshow("Prediction", image)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
except Exception as e:
print(f"[ERROR] {e}")
finally:
conn_file.close()
conn.close()
server_socket.close()
cv2.destroyAllWindows()
RPI (Raspberry Pi) Folder
This contains:
- Your test codes
- Your folder with speaker audio's
- Your folder with videos/gifs that will be displayed on matrix
- A script (like client.py) that:
- Uses OpenCV to capture hand or finger positions
- Sends that data to the laptop via a socket connection
- (Optionally) receives and displays the prediction
code client.py:
import cv2
import socket
import struct
import time
import signal
import sys
import pygame
import random
import os
import threading
import RPi.GPIO as GPIO
from PIL import Image, ImageSequence, ImageDraw, ImageFont
import numpy as np
import adafruit_blinka_raspberry_pi5_piomatter as piomatter
import pygame
# === SETUP PYGAME FOR AUDIO ===
pygame.mixer.pre_init(44100, -16, 2, 512) # frequency, size, channels, buffer size
pygame.init()
pygame.mixer.init()
pygame.mixer.music.set_volume(1.0)
# === CONFIGURATION ===
LAPTOP_IP = '192.168.168.10'
PORT = 8000
AUDIO_FOLDER = "{folder placement}/speaker/data"
# change it to your own path of your audio's
GPIO.setmode(GPIO.BCM)
GREEN_PIN = 18 #change it to your desired GPIO
RED_PIN = 25 #change it to your desired GPIO
BUZZER_PIN = 19 #change it to your desired GPIO
# === GPIO SETUP ===
GPIO.setwarnings(False)
GPIO.setup(GREEN_PIN, GPIO.OUT)
GPIO.setup(RED_PIN, GPIO.OUT)
GPIO.setup(BUZZER_PIN, GPIO.OUT)
GPIO.output(GREEN_PIN, GPIO.LOW)
GPIO.output(RED_PIN, GPIO.LOW)
GPIO.output(BUZZER_PIN, GPIO.LOW)
buzzer_pwm = GPIO.PWM(BUZZER_PIN, 1000) # 1kHz frequency
buzzer_pwm_started = False
# === GLOBAL VARIABLES ===
matrix_pause_event = threading.Event()
client_socket = None
connection = None
cap = None
should_exit = False
last_prediction_time = time.time()
idle_beep_active = False
matrix_audio_playing_event = threading.Event()
WORD_FINALIZE_TIME = 5
# === MATRIX SETUP ===
MATRIX_WIDTH = 64
MATRIX_HEIGHT = 64
matrix_geometry = piomatter.Geometry(width=MATRIX_WIDTH, height=MATRIX_HEIGHT, n_addr_lines=5,
rotation=piomatter.Orientation.Normal)
matrix_framebuffer = np.zeros((MATRIX_HEIGHT, MATRIX_WIDTH, 3), dtype=np.uint8)
matrix = piomatter.PioMatter(
colorspace=piomatter.Colorspace.RGB888Packed,
pinout=piomatter.Pinout.AdafruitMatrixBonnet,
framebuffer=matrix_framebuffer,
geometry=matrix_geometry
)
IDLE_GIF_PATH = "{folder path/faces/idle.gif"
SAD_GIF_PATH = "{folder path/faces/faces/sad.gif"
STEVE_VIDEO_PATH = "{folder path/faces/faces/steve_vidio.mp4"
# change it to your own path of your videos/gifs
def matrix_gif_player():
gif_path = None
frames = []
durations = []
frame_idx = 0
while True:
# Pause GIF animation if requested (e.g., for video or IP text)
if matrix_pause_event.is_set():
time.sleep(0.05)
continue
state = matrix_audio_playing_event.is_set()
new_gif_path = SAD_GIF_PATH if state else IDLE_GIF_PATH
if new_gif_path != gif_path:
gif_path = new_gif_path
try:
with Image.open(gif_path) as img:
frames = [frame.copy().convert("RGB").resize((MATRIX_WIDTH, MATRIX_HEIGHT), Image.LANCZOS)
for frame in ImageSequence.Iterator(img)]
durations = [frame.info.get('duration', 100) / 1000.0 for frame in ImageSequence.Iterator(img)]
if not durations:
durations = [0.1] * len(frames)
frame_idx = 0
except Exception as e:
print(f"[MATRIX] Error loading GIF: {e}")
time.sleep(0.5)
continue
frame = frames[frame_idx]
frame_bgr = np.asarray(frame)[..., ::-1]
matrix_framebuffer[:] = frame_bgr
matrix.show()
time.sleep(durations[frame_idx])
frame_idx = (frame_idx + 1) % len(frames)
def matrix_display_controller():
global matrix_audio_playing
idle_stop = threading.Event()
sad_stop = threading.Event()
while True:
if matrix_audio_playing:
idle_stop.set()
sad_stop.clear()
matrix_gif_player(SAD_GIF_PATH, sad_stop)
else:
sad_stop.set()
idle_stop.clear()
matrix_gif_player(IDLE_GIF_PATH, idle_stop)
time.sleep(0.05)
def play_video_on_matrix(video_path):
matrix_pause_event.set()
cap = cv2.VideoCapture(video_path)
if not cap.isOpened():
print(f"[ERROR] Cannot open video: {video_path}")
matrix_pause_event.clear()
return
video_width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
video_height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
aspect_ratio = video_width / video_height
if aspect_ratio > 1:
new_width = MATRIX_WIDTH
new_height = int(MATRIX_WIDTH / aspect_ratio)
if new_height > MATRIX_HEIGHT:
new_height = MATRIX_HEIGHT
new_width = int(MATRIX_HEIGHT * aspect_ratio)
else:
new_height = MATRIX_HEIGHT
new_width = int(MATRIX_HEIGHT * aspect_ratio)
if new_width > MATRIX_WIDTH:
new_width = MATRIX_WIDTH
new_height = int(MATRIX_WIDTH / aspect_ratio)
x_offset = (MATRIX_WIDTH - new_width) // 2
y_offset = (MATRIX_HEIGHT - new_height) // 2
fps = cap.get(cv2.CAP_PROP_FPS)
if not fps or fps < 1:
fps = 24
frame_delay = 1.0 / fps
try:
while cap.isOpened() and pygame.mixer.music.get_busy():
ret, frame = cap.read()
if not ret:
cap.set(cv2.CAP_PROP_POS_FRAMES, 0) # Loop video if it ends before audio
continue
frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
frame_rgb = cv2.resize(frame_rgb, (new_width, new_height), interpolation=cv2.INTER_AREA)
frame_bgr = np.zeros((MATRIX_HEIGHT, MATRIX_WIDTH, 3), dtype=np.uint8)
frame_bgr[y_offset:y_offset+new_height, x_offset:x_offset+new_width] = frame_rgb[..., ::-1]
matrix_framebuffer[:] = frame_bgr
matrix.show()
time.sleep(frame_delay)
except Exception as e:
print(f"[ERROR] Playing video: {e}")
finally:
cap.release()
matrix_pause_event.clear()
# === CATCH IP ADDRESS ===
def get_ip_address():
import socket
try:
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(('10.255.255.255', 1)) #ip of your raspi
IP = s.getsockname()[0]
except Exception:
IP = '127.0.0.1'
finally:
s.close()
return IP
def show_text_on_matrix(text, duration=60):
img = Image.new('RGB', (MATRIX_WIDTH, MATRIX_HEIGHT), color=(0, 0, 0))
draw = ImageDraw.Draw(img)
try:
font = ImageFont.truetype("/usr/share/fonts/truetype/dejavu/DejaVuSans-Bold.ttf", 12)
except:
font = ImageFont.load_default()
# Center text using anchor (Pillow ≥8.0)
draw.text((MATRIX_WIDTH // 2, MATRIX_HEIGHT // 2), text, font=font, fill=(255, 0, 0), anchor="mm")
frame_bgr = np.asarray(img)[..., ::-1]
matrix_framebuffer[:] = frame_bgr
matrix.show()
time.sleep(duration)
# === IDLE BEEP CONTROLLER ===
def idle_beep_controller():
global last_prediction_time, should_exit, buzzer_pwm_started, idle_beep_active
while True:
if should_exit:
break
idle_for = time.time() - last_prediction_time
if idle_for >= 60:
idle_beep_active = True
# Turn off LEDs during idle beep
GPIO.output(GREEN_PIN, GPIO.LOW)
GPIO.output(RED_PIN, GPIO.LOW)
# Start PWM if not already started
if not buzzer_pwm_started:
buzzer_pwm.start(0)
buzzer_pwm_started = True
# Loud beep
buzzer_pwm.ChangeDutyCycle(100)
time.sleep(0.3)
buzzer_pwm.ChangeDutyCycle(0)
time.sleep(0.15)
# Medium beep
buzzer_pwm.ChangeDutyCycle(40)
time.sleep(0.3)
buzzer_pwm.ChangeDutyCycle(0)
time.sleep(0.15)
# Soft beep
buzzer_pwm.ChangeDutyCycle(15)
time.sleep(0.3)
buzzer_pwm.ChangeDutyCycle(0)
idle_beep_active = False
last_prediction_time = time.time()
time.sleep(0.1)
threading.Thread(target=idle_beep_controller, daemon=True).start()
threading.Thread(target=matrix_gif_player, daemon=True).start()
# === AUDIO RESPONSES ===
# these you can change all you want: "{catogory}: [{audio.files}]
responses = {
"hello": ["hello_1.mp3", "hello_2.mp3", "hello_3.mp3"],
"bye": ["bye_1.mp3", "bye_2.mp3", "bye_3.mp3", "bye_4.mp3"],
"emotion": ["emotion_1.mp3", "emotion_2.mp3", "emotion_3.mp3", "emotion_4.mp3"],
"information": ["information_1.mp3", "information_2.mp3"],
"joke": ["joke_1.mp3", "joke_2.mp3", "joke_3.mp3"],
"funfact": ["funfact_1.mp3", "funfact_2.mp3"],
"off": ["off.mp3"],
"reset": ["reset.mp3"],
"sleep": ["sleep.mp3"],
"compliment": ["complement_1.mp3", "complement_2.mp3"],
"Steve": ["Steve_audio.wav"],
}
# === WORD TRACKING VARIABLES ===
current_letter = None
letter_start_time = 0
current_word = ""
last_letter_time = 0
# === RECONNECT LOGIC ===
def reconnect_socket():
global client_socket, connection
while True:
try:
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect((LAPTOP_IP, PORT))
connection = client_socket.makefile('wb')
print("Successfully reconnected to server")
break
except Exception as e:
print(f"Reconnection failed: {e}")
time.sleep(5)
# === SETUP PYGAME FOR AUDIO ===
pygame.mixer.init()
pygame.mixer.music.set_volume(1.0)
def cleanup_and_exit(signum, frame):
global should_exit, buzzer_pwm_started
should_exit = True
print("\n[INFO] Exiting gracefully...")
try:
if connection is not None:
connection.close()
if client_socket is not None:
client_socket.close()
if cap is not None and cap.isOpened():
cap.release()
pygame.mixer.quit()
GPIO.output(GREEN_PIN, GPIO.LOW)
GPIO.output(RED_PIN, GPIO.LOW)
GPIO.output(BUZZER_PIN, GPIO.LOW)
if buzzer_pwm_started:
buzzer_pwm.stop()
GPIO.cleanup()
except Exception as e:
print(f"[WARN] Exception during cleanup: {e}")
sys.exit(0)
def play_audio(file_name):
path = os.path.join(AUDIO_FOLDER, file_name)
if not os.path.exists(path):
print(f"Audio file not found: {path}")
return
def _play_thread():
try:
matrix_audio_playing_event.set() # Start sad GIF animation
# No need to re-initialize mixer if already initialized at script start
pygame.mixer.music.load(path)
pygame.mixer.music.play()
while pygame.mixer.music.get_busy():
time.sleep(0.1)
except Exception as e:
print(f"Audio playback error: {e}")
finally:
matrix_audio_playing_event.clear() # Return to idle GIF animation
audio_thread = threading.Thread(target=_play_thread, daemon=True)
audio_thread.start()
def _play_thread():
try:
if not pygame.mixer.get_init():
pygame.mixer.init()
pygame.mixer.music.load(path)
pygame.mixer.music.play()
while pygame.mixer.music.get_busy():
time.sleep(0.1)
except Exception as e:
print(f"Audio playback error: {e}")
audio_thread = threading.Thread(target=_play_thread, daemon=True)
audio_thread.start()
def get_response_audio(word): #The can also be changed, the text just means what u will sign it will recognize
text = word.strip().lower()
if text in ["hi", "hello", "yo", "hey", "hallo"]:
category = "hello"
elif text in ["bye", "goodbye", "take care", "later", "peace"]:
category = "bye"
elif text in ["mood", "emotion", "feeling"]:
category = "emotion"
elif text in ["information", "info", "details"]:
category = "information"
elif text in ["joke", "pun"]:
category = "joke"
elif text in ["funfact", "fact", "weetje"]:
category = "funfact"
elif text in ["out", "off"]:
category = "off"
elif text in ["reset", "reboot"]:
category = "reset"
elif text in ["zzz", "mimimi", "sleep", "goodnight"]:
category = "sleep"
elif text in ["compliment", "yapping"]:
category = "compliment"
elif text in ["ip", "address"]:
category = "ip"
elif text in ["steve", "poisson"]:
category = "Steve"
else:
print("No matching response for word:", word)
return None
return random.choice(responses[category])
def is_word_recognized(word):
text = word.strip().lower()
return text in [
"hi", "hello", "yo", "hey", "hallo",
"bye", "goodbye", "take care", "later", "peace",
"mood", "emotion", "feeling",
"information", "info", "details",
"joke", "pun",
"funfact", "fact", "weetje"
"out", "off",
"reset", "reboot",
"zzz", "mimimi", "sleep", "goodnight",
"compliment", "yapping",
"ip", "address",
"steve", "poisson"
]
def trigger_speaker(word):
global current_word, current_letter, letter_start_time, buzzer_pwm_started
print(f"[SPEAKER] Word detected: {word}")
# Reset LEDs/Buzzer
GPIO.output(RED_PIN, GPIO.LOW)
GPIO.output(GREEN_PIN, GPIO.LOW)
GPIO.output(BUZZER_PIN, GPIO.LOW)
if buzzer_pwm_started:
buzzer_pwm.ChangeDutyCycle(0)
if is_word_recognized(word):
GPIO.output(GREEN_PIN, GPIO.HIGH)
audio_file = get_response_audio(word)
if audio_file:
print(f"Playing response audio: {audio_file}")
play_audio(audio_file)
if audio_file == "Steve_audio.wav":
threading.Thread(
target=play_video_on_matrix,
args=(STEVE_VIDEO_PATH,),
daemon=True
).start()
play_audio(audio_file)
else:
play_audio(audio_file)
time.sleep(2)
GPIO.output(GREEN_PIN, GPIO.LOW)
else:
GPIO.output(RED_PIN, GPIO.HIGH)
if not buzzer_pwm_started:
buzzer_pwm.start(100)
buzzer_pwm_started = True
else:
buzzer_pwm.ChangeDutyCycle(100)
time.sleep(0.7)
buzzer_pwm.ChangeDutyCycle(0)
GPIO.output(RED_PIN, GPIO.LOW)
current_word = ""
current_letter = None
letter_start_time = 0
def listen_for_q():
global should_exit
while True:
user_input = sys.stdin.readline()
if user_input.strip().lower() == 'q':
print("\n[INFO] Q pressed, exiting gracefully...")
should_exit = True
cleanup_and_exit(None, None)
def main():
global current_letter, letter_start_time, current_word, last_letter_time, last_prediction_time
global client_socket, connection, cap, should_exit
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 320)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 240)
cap.set(cv2.CAP_PROP_FPS, 10)
signal.signal(signal.SIGINT, cleanup_and_exit)
threading.Thread(target=listen_for_q, daemon=True).start()
last_printed_line = None # Track the last printed line
while not should_exit:
try:
reconnect_socket()
while not should_exit:
ret, frame = cap.read()
if not ret:
continue
_, jpeg = cv2.imencode('.jpg', frame, [int(cv2.IMWRITE_JPEG_QUALITY), 50])
data = jpeg.tobytes()
size = struct.pack('>L', len(data))
try:
connection.write(size + data)
connection.flush()
pred_letter = client_socket.recv(1).decode('utf-8')
now = time.time()
display_text = f"Prediction: {pred_letter}"
countdown_text = ""
if pred_letter != '?' and pred_letter:
last_prediction_time = now
display_text += f" (Hold for: {5 - min(5, now - letter_start_time):.1f}s)"
if pred_letter != current_letter:
current_letter = pred_letter
letter_start_time = now
last_letter_time = now
else:
if (now - letter_start_time) >= 5:
current_word += current_letter
print(f"[WORD BUILDING] Current word: {current_word}")
last_letter_time = now
letter_start_time = now # Reset timer for next instance of the same letter
else:
inactive_time = now - last_prediction_time
if current_word:
countdown_text = f" | Timeout in: {5 - min(5, inactive_time):.1f}s"
if inactive_time >= WORD_FINALIZE_TIME:
print(f"[WORD FINALIZED] {current_word}")
if current_word.strip().lower() == "ip":
ip = get_ip_address()
show_text_on_matrix(f"IP: {ip}", duration=60)
current_word = ""
continue # Skip normal trigger_speaker for this word
trigger_speaker(current_word)
current_line = f"\r{display_text}{countdown_text} | Current word: {current_word}"
if current_line != last_printed_line:
print(current_line, end="", flush=True)
last_printed_line = current_line
except BrokenPipeError:
print("\n[WARN] Connection broken - attempting reconnect")
raise
time.sleep(0.05)
except BrokenPipeError:
print("\n[WARN] Connection broken - attempting reconnect")
try:
if connection: connection.close()
if client_socket: client_socket.close()
except: pass
time.sleep(1)
except Exception as e:
print(f"\n[ERROR] {e}")
if cap is not None:
cap.release()
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FRAME_WIDTH, 320)
cap.set(cv2.CAP_PROP_FRAME_HEIGHT, 240)
cap.set(cv2.CAP_PROP_FPS, 10)
time.sleep(1)
if __name__ == "__main__":
main()
Tip: always do first your server and than your client
You Did It!
If you've followed along this far — congrats!
You’ve successfully built a real-time gesture recognition system using a Raspberry Pi and a laptop running machine learning.
What You’ve Accomplished
- Captured hand or finger movement using a camera and OpenCV
- Sent the keypoint data from your Pi to your laptop using sockets
- Ran an AI model (gesture_model.pkl) on your laptop to predict gestures
- Sent predictions back to the Pi — possibly to control something or display output
All that, and you didn’t overload your Raspberry Pi. Smart move.
Thanks for reading!
If this Instructable helped you, please consider:
- ❤️ Giving it a favorite
- 💬 Leaving a comment or question
- 🔁 Sharing it with someone else learning AI or Raspberry Pi
If you want to suggest something else? Then go ahead! i am just a first-year student and not everything will be that greatly done, i am open to any critic as long as it doesnt make me cry
Github link to every coding and my own code and model