Shape Matching With XArm

This project uses a combination of a camera and a robotic arm with six degrees of freedom to locate holes in a board and place the corresponding shapes back in them. The goal was to write a controller that could do that independently, along with learning about computer vision and policy design for robotics.

1. Hisense xArm 1S
2. Intel RealSense D435 camera (other cameras with sufficient resolution should work as well)
3. Board with cutouts on a contrasting background. The one used for this project was a thick cardboard, with the shapes laser-cut from it.
4. Python code

The board with cutouts should be placed on a surface that it will contrast with (this helps with the image processing that will be done later). If needed, placing a lamp nearby can also provide the camera with a better view. Each piece cut from the board has a small handle for the arm to grip. The arm should be able to reach all of the pieces, and the camera should be able to see the entire board, with a clear view of the board and shape pieces. The camera and the base of the arm must remain in exactly the same places while the program is running.

The arm works using a grid system, with known X and Y points around the area of the board and shape pieces and their respective servo positions. If you run the file main.py, there is an input line giving a pair of coordinates that you need to place the arm at. The motors are not locked at this point, so they can easily be moved to allow the tip of the grippers to be positioned at the given coordinate position. These points will be interpolated to make the arm reach positions between the known ones, so it is imperative that the motions from point to point while calibrating the arm are smooth and move all of the servos. The angle of the grippers should be kept as close to horizontal as possible. The arm itself is controlled using the Python xarm library, which can also be found on Github.

The image processing portion of this project is done mainly with the code in webcam.py and edge_finder.py. The above picture shows a visualization of part of this pipeline. The camera starts by taking a picture of the entire board, which is then cropped and reduced to black and white. The latter part may not be necessary given the quality of the RealSense camera, but it can help to reduce noise when there are fewer pixels in an image. The image is split into two halves, with one being the board and the other the pieces. The location of this split may need to be adjusted depending on where the board is placed, but as long as they remain in the given zone the locations of the board and pieces can be shifted. The edges are then found and kept as a set of points, and K-means clustering used to separate the points into individual shapes. The shapes are then matched across the sides of the board and pieces, with the cluster numbers identifying pairs.

In webcam.py there are two functions to take pictures, while the rest of the processing up to this point is done using the edge_finder and match_clusters functions in edge_finder.py. Edge finding was done using gradient differences, and cluster matching was done by minimizing Chamfer distance, a metric that is commonly used for pattern matching.

At this point, the locations of all the pieces and the holes that they fit in are known, as are the position adjustments needed by the arm to reach those locations. The video above shows this in progress, with the arm picking up and placing the shapes in approximately the spots where they should be.

In the future, this project could be expanded upon by using real-time video streaming instead of discrete pictures to update the locations of the pieces. As it is now, the arm will not adjust its course if there are any changes after it is planned. Another thing to do would be to make a reverse kinematics model for the arm, which would eliminate the need for the calibration step and could be more accurate overall.