Follow Bot

Project for 16299. A robot that follows a designated person.

Some notes/next steps:

One big mistake I made was not checking if the robot had an encoder, this one doesn't. Sometimes, due to lighting issues or obstacle obstruction, the robot might not be able to see the goal person. I originally wanted to keep track of the last location estimation of the goal and use odometry to continue navigating to that location. However, odometry requires encoders. One next step would be attaching encoders to the robot. This would allow navigation when the goal isn't insight, helpful for obstacle avoidance.

I had originally used YOLO's newest model for person detection, although it was much more accurate than the model OpenCV provides, it's also much slower. Making it not worth using. A potential alternative that would allow the usage of YOLO without the high computation cost would be to send the image to my laptop and do the model running on my laptop and transmit the results back to the pi.

I originally used fastreid, specifically the BOT model, in order to detect which person is the goal person. However it didn't give great results, likely due to a high difference in training data and my use case. A next step would be to fine tune the model for my specific use case so that the robot is able to accurate find the right person to follow.

A next step would be to do some testing with my laptop on the robot and see how well it works with added load.

1. Wave Rover
2. Fisheye USB Camera
3. Raspberry Pi 5
4. USB extension

Some disassembling of the robot is required in order to attach the raspi onto the driver board of the robot. One the raspberry pi is attached, an USB extension is required to attach the camera.

Since I'm using a fisheye camera, I needed to first undistort the images/video I was getting from the camera. This is because if I tried to calculate the distance from the robot to the target without undistorting, the distortion would make the results different from reality.

OpenCV provides some functions that does this for us. First we need to calibrate the camera to get it's camera matrix and distortion matrix by using some pictures of a chessboard taken by the camera. A tutorial can found here.

Then we use this function OpenCV provides to undistort images with the matrixes we found.

Since these matrixes shouldn't change over time, we can keep them for undistorting each frame in the video feed later

I implemented april tag following first as a baseline.

There's an april tag library in python, which does april tag detection in an image for you.

After finding the corner coordinates of the april tag in the image, we can use the solvePnP OpenCV function to estimate the pose of the april tag, this give us the translational and rotational matrixes from the position of the camera to the april tag in the image.

Finally we can find the Euclidian distance from the robot to the tag using the translational matrix.

I implemented 2 PID loops in order to drive the robot.

One loop calculated the power given to the motors based on the x and z axis distances (front back and up down respectively), essentially controlling gas and break. The other loop calculated the power given to the motors based on the y axis distance (left right), essentially controlling stirring. These 2 loops are combined by just adding the results.

I manually tuned the P and D for each loop.

Now that the robot follows an april tag, it's time to implement person detection.

The robot is open source and I found a github for it that already has object detection using a machine learning model that OpenCV provides. I repurposed the code for object detection to get only people.

Detecting the distance from the camera to the person is a bit trickier. Since to use the same method we did for the april tag, we would need to know the height and width of the person. We can estimate the height and width of the person by taking a picture of the person holding an april tag at the beginning. We can then estimate by multiplying the pixel size of the person by the actual size of the tag divided by the pixel size of the tag. Although the result is inaccurate to reality due to the distortion from the camera being much lower down than the center of the person, it still provides us with a good number to work with in distance estimation.

In order to make sure the robot is following the right person, we can compare each candidate person in the frame pixel-wise to the picture taken at the beginning and using the best candidate.

Using the same PID controls, I was able to make the robot follow me and not someone else.