How to Use the Raspberry Pi AI HAT+ for Real-time Object Detection

As someone who loves tinkering with my Raspberry Pi 5 and using it in all sorts of fun projects, I got super excited about the idea of building something with the Raspberry Pi AI HAT+ to integrate AI models without slowing down the Pi. However, when I looked at the documentation, all I could find were examples using the GUI app demos on the desktop OS. This wasn't very helpful for embedding the hardware into standalone projects at home, the office, or where ever I need it.

So I created a reusable module to make getting started and using the AI HAT+ features more accessible for myself and others to use it from anywhere.

To showcase what's possible (and give myself a warning about the expected traffic on my commute to the office), I made a live traffic dashboard that displays real-time images and data from my Raspberry Pi running a YOLO model with the video stream from a USB webcam.

Sign up for a free Viam account, and then sign in to the Viam app

1 - Raspberry Pi 5 w/ active cooler

1 - microSD card to use with your Pi

1 - power supply for your Pi

1 - Raspberry Pi AI HAT+

1 - USB webcam

The active cooler is recommended by Raspberry Pi for the best performance.

The Raspberry Pi AI Kit may still be for sale but it has been replaced by the AI HAT+ going forward, which is why it is recommended here. Either the 13 TOPS or 26 TOPS version will work in this project.

Follow the official documentation for instructions about installing the AI HAT+ on the Raspberry Pi 5.

The Raspberry Pi boots from a microSD card. You need to install Raspberry Pi OS on the microSD card that you will use with your Pi. For more details about alternative methods of setting up your Raspberry Pi, refer to the Viam docs.

Install Raspberry Pi OS

1. Connect the microSD card to your computer.
2. Download the Raspberry Pi Imager and launch it.
3. Click CHOOSE DEVICE. Select your model of Pi, which is Raspberry Pi 5.
4. Click CHOOSE OS. Select Raspberry Pi OS (64-bit) from the menu.
5. Click CHOOSE STORAGE. From the list of devices, select the microSD card you intend to use in your Raspberry Pi.
6. Configure your Raspberry Pi for remote access. Click Next. When prompted to apply OS customization settings, select EDIT SETTINGS.
7. Check Set hostname and enter the name you would like to access the Pi by in that field, for example, test.
8. Select the checkbox for Set username and password and set a username (for example, your first name) that you will use to log into the Pi. If you skip this step, the default username will be pi (not recommended for security reasons). And specify a password.
9. Connect your Pi to Wi-Fi so that you can run viam-server wirelessly. Check Configure wireless LAN and enter your wireless network credentials. SSID (short for Service Set Identifier) is your Wi-Fi network name, and password is the network password. Change the section Wireless LAN country to where your router is currently being operated.
10. Select the SERVICES tab, check Enable SSH, and select Use password authentication. Be sure that you remember the hostname and username you set, as you will need this when you SSH into your Pi.
11. Save your updates, and confirm YES to apply OS customization settings. Confirm YES to erase data on the USB flash drive. You may also be prompted by your operating system to enter an administrator password. After granting permissions to the Imager, it will begin writing and then verifying the Linux installation to the USB flash drive.
12. Remove the USB flash drive from your computer when the installation is complete.

Connect with SSH

1. Place the microSD card into your Raspberry Pi and boot the Pi by plugging it in to an outlet. A red LED will turn on to indicate that the Pi is connected to power.
2. Once the Pi is started, connect to it with SSH. From a command line terminal window, enter the following command. The text in <> should be replaced (including the < and > symbols themselves) with the user and hostname you configured when you set up your Pi.

1. If you are prompted "Are you sure you want to continue connecting?", type "yes" and hit enter. Then, enter the password for your username. You should be greeted by a login message and a command prompt.
2. Update your Raspberry Pi to ensure all the latest packages are installed

With the OS installed, it's time to set up the system packages for the AI HAT+.

Connect to the Pi with SSH, if you're not still connected from the previous step.

Enable PCIe Gen 3.0 speeds for the best performance

1. Open the Raspberry Pi Configuration interface:

1. Select Advanced Options
2. Select PCIe Speed
3. Choose Yes to enable PCIe Gen 3 mode
4. Select Finish to exit the configuration interface
5. Reboot the device

1. Reconnect to the Pi with SSH to continue.

Set up system dependencies

1. Install the "hailo-all" package, which contains the firmware, device drivers, and processing libraries for the AI HAT+:

1. This may take a few minutes depending on your network speed.
2. Reboot the device

1. Reconnect to the Pi with SSH
2. Verify everything is running correctly:

You should see output similar to the following:

Add your webcam

1. Connect the webcam's USB lead to any USB port on your Pi. The webcam transmits video data to the Raspberry Pi through the USB interface.

Now that we have physically connected our hardware components, place them in a spot with a good view of the traffic. In the next section, we'll configure our machine.

Configure your machine

1. In the Viam app under the LOCATIONS tab, create a machine by typing in a name and clicking Add machine.
2. Click View setup instructions.
3. To install viam-server on the Raspberry Pi device that you want to use, select the Linux / Aarch64 platform for the Raspberry Pi, and leave your installation method as viam-agent.
4. Use the viam-agent to download and install viam-server on your Raspberry Pi. Follow the instructions to run the command provided in the setup instructions from the SSH prompt of your Raspberry Pi.
5. The setup page will indicate when the machine is successfully connected.

Configure your webcam

1. In the Viam app under the CONFIGURE tab, click the + icon in the left-hand menu and select Component.
2. Select camera, and find the webcam module. This adds the module for working with a USB webcam. Leave the default name camera-1 for now.
3. Notice adding this component adds the webcam hardware component called camera-1. From the Attributes section of the panel, select a video_path.
4. Click Save in the top right. This may take a moment to apply your configuration changes.
5. At the bottom of the camera-1 panel, expand the TEST section to ensure you have configured the camera properly and see a video feed.

Configure your vision service

1. In the Viam app under the CONFIGURE tab, click the + icon in the left-hand menu and select Service.
2. Select vision, and find the hailo-rt module. This adds the module for working with Hailo Runtime used by the AI HAT+. Select "Add module". Leave the default name vision-1 for now.
3. Notice adding this service adds the computer vision software service called vision-1. From the Depends on section of the panel, select camera-1 from the "Search resources" dropdown.
4. Click Save in the top right. This may take a moment to apply your configuration changes.
5. At the bottom of the vision-1 panel, expand the TEST section to ensure you have configured the service properly and see images from camera-1 with object detection boxes on top.

Configure your detections sensor

1. In the Viam app under the CONFIGURE tab, click the + icon in the left-hand menu and select Component.
2. Select sensor, and find the detections module. This adds a module for capturing object detection data from a vision service. Leave the default name sensor-1 for now.
3. Notice adding this component adds the sensor component called sensor-1. From the Attributes section of the panel, add the following JSON configuration.

1. From the Data capture section of the panel, click "+ Add method". Set the "Method" as "Readings", "Frequency (hz)" as 1. Click "Create data management service" in the banner within the panel to automatically create a Data Management Service called data_manager-1.
2. Click Save in the top right. This may take a moment to apply your configuration changes.
3. At the bottom of the sensor-1 panel, expand the TEST section to ensure you have configured the sensor properly and see a list of the configured labels with the number of detections refreshed regularly.

With all the components and services in place, you can move on to creating a live tele-operations dashboard for your machine!

This step walks through how to use the teleop (or tele-operations) feature of the Viam app.

Configure your teleop workspace

1. In the Viam app, return to the home page.
2. Then under the TELEOP tab, create a new dashboard by clicking Create workspace.
3. Click on "untitled-workspace" to set the name of your dashboard.
4. Click "Select machine" and choose your previously configured machine from the list.

Configure camera stream widget

1. Click "+ Add widget" and select "Camera stream".
2. Notice adding this widget creates a panel called Camera. Select camera-1 from the "Camera name" field, keep the "Refresh type" as "Live".

Configure time series graph widget

1. Click "+ Add widget" and select "Time series".
2. Notice adding this widget creates a new panel called Time series. Set the "Title" to "Traffic" and "Time range (min)" to 30.
3. Under the Lines section of the panel, select sensor-1 for the "Resource name", Readings for "Capture method", cars for "Title", and readings.car for "Path".
4. Add new lines by clicking "+ Add line". Create a line for each remaining label captured by the sensor: bus and person.
5. Click "Save changes" to see the camera stream and data updating live.

At this point, you have created an edge device that can perform real-time object detection and monitor it remotely from anywhere! You can keep building on this project with additional features:

1. Model out a smart city control system by connecting red, yellow, and green LEDs to the Pi for simulated traffic light control.
2. Make your smart home even smarter by integrating with Home Assistant
3. Put it on a rover and make it mobile

The default YOLO-based models can do more than detect traffic! You can find the full list of identifiable objects included in the module repo: https://github.com/HipsterBrown/viam-pi-hailo-ml/

If none of those items suit your project requirements, you can explore the full suite of model types in the Hailo Model Zoo or retrain a model on a custom dataset.

Share more ideas of how you use the AI HAT+ in a Raspberry Pi project in the comments!

This tutorial was originally published on Viam codelabs: Real-Time Object Detection with Raspberry Pi AI HAT