AutoToast

AutoToast is a modified toaster that uses computer vision, machine learning, and feedback control which allows you to cook toast to a desired color.

Overview:

A Python program running on a laptop monitors a camera feed and detects when the toast is ready to be ejected.

• ML: It uses machine learning to identify which type of bread is inserted
• CV: It uses Computer Vision to detect how dark the bread is (how much it has been cooked)
• KF: It uses a Kalman Filter to calculate how far along the toasting process the bread is
• When ready, it tells an Arduino to actuate a relay which causes the toaster to eject the toast.

Refer to the block diagram above for further breakdown.

• Toaster with Glass Window
• Arduino
• Relay
• Computer to run python program
• Cardboard
• Wires
• Tape/Hot Glue

First, we need to modify the toaster to allow an Arduino to eject the toast.

For our toaster, there were screws on the bottom that we had to remove for the top black part to come off of the bottom. We also had to snap the wide part of the toaster lever off in order for the two parts to come apart. Be careful when doing this, because there is a wire connecting the top and bottom parts. Remove this wire and the two will come apart.

On the top part, unscrew the circuit board containing the buttons and dial. Solder long wires to the pins of the buttons. Be sure to not use too much heat or the button will come unseated and re-soldering will be more difficult. Bring these wires to the outside of the toaster through the bottom of the level slot.

Plug in the toaster eject button wires into the NO (Normally Open) ports of the relay.

On the control side, plug in the relay to Arduino +5V and GND, and the control pin into a digital pin of the Arduino.

Then plug the Arduino into your computer using a USB cable. This will both power it and let you communicate with it.

Now we need to set the camera up to point at the toaster while its toasting.

First paste an empty red box around the window. This will let the computer vision software identify where the toast is in the image.

A link to the box we used can be found here: https://docs.google.com/drawings/d/1A3dzfQ3Rrwj57LPb8juTV_mUXE60wxeTGvvKuZc7jBI/edit?usp=sharing

Print it out, cut out the middle, and tape to the window.

Next, build a cardboard enclosure around the window that extends approximately 1 foot back. This will isolate the window from extra light as much as possible. Place a horizontal piece of cardboard in the back of the tube at about the same height as the window, and attach the USB webcam to this cardboard piece facing the toaster window.

Plug the USB webcam into the computer.

Download this software in order to change camera settings:

https://support.logi.com/hc/en-us/articles/360024692954--Downloads-HD-Webcam-C270

This will allow you to set the white balance settings for the camera and disable auto-balance. We chose a manual setting of 4220k and this is what our ML model is trained on.

Next, we will go over the python software needed to run on the computer. To run the software, clone the following repository:

https://github.com/VikramBhat00/AutoToast

And pip install the following libraries :

• pyserial
• pathlib
• matplotlib
• numpy
• sklearn
• skimage

Then run newControl.py in the repository . (If this fails, change the serial port on line 15 of newControl.py to the correct serial port for your Arduino).

In order to run everything together, plug in the toaster, put in a piece of bread in clear view of the window, pull the toaster lever down, and run the newControl python file. This will display the toast window on the computer screen and eject when done.

Enjoy your toast!

Below is a breakdown of the different parts of the code.

The computer vision component of the code returns the average darkness of the toast window.

The computer vision pipeline is as follows:

1. Isolate the Red Parts of the image
2. Use HSV Thresholding to find colors with hue range 0-10 and 160-179
3. Turn red area into a binary image, red corresponds to 255 and everything else 0.
4. Erode this binary image to remove very small extra red regions
5. Run Contour Detection on binary image to find the red box, and draw the largest contour with the thickness set to cv2.FILLED. This will draw another binary image of just the white filled box on a black background. We will call this a mask.
6. Convert the original RGB image from the camera to grayscale
7. Bitwise and the gray image with the mask. This will yield only the gray region inside the box (presumably the toast).
8. Take this gray region and sum the grayscale values of all the points and divide by the size of the mask .
9. Return this average value as the 'doneness' of the toast.

This code is implemented in the "CV.py" file.

The communication component of the code sends a message from the python program to the Arduino over Serial.

The message format is defined as follows:

#Message Format  Start byte    Rotation Degrees  Toast Stop indicator  End Byte
#									(0-180)			    (0, 1)
msg = bytearray([	254, 		       0,               0, 			     255])

Where there is a start byte, end byte, and 2 bytes of data. Currently, the rotation degrees is ignored and only the stop indicator is used. When this message is sent with a 1 in index 2, the Arduino actuates the relay and the toast is ejected.

The function write_message(Arduino, rotate, stop) is used to send messages of this format.

This code can be found in "communicate.py"

The Learning module identifies the type of bread present in the toaster so the software can select the appropriate model to use for the Kalman Filter.

• An SVM (Support Vector Machine) detects the type of bread in the toaster
• Support Vectors are points in a distribution of data that are able to define the decision boundary within that data
• For this project, we trained the SVM to recognize the difference between Whole Grain and Wheat Flour bread (firm brown vs soft white).

To train the svm, we took 55 images of each type of bread within the toaster and varied the orientation, direction, and position of the bread slightly in each photo to get enough diversity for a robust SVM.

The code to recognize the type of bread is contained in the "ML.py" file.

KF (Kalman Filter):

The Kalman Filter adaptation allows us to estimate our toast done-ness state as a percentage of completeness. We originally used basic checking of the toast grayscale color returned from the CV pipeline to do this, but to add a bit more complexity, we decided to try and implement a Kalman filter.

The state is represented as the percent "done" a piece of toast is.

The measurement involves combining the time taken thus far with the array of readings gathered in this run.

These are compared to a model generated for the particular type of bread thats being toasted, of desired colors over time . The function that matches the current readings to a location in the model can be seen above as well, the "fitting_function() " method.

The graphs in the above images can be seen, matching the read-in camera sensor values to the dotted model.

The Kalman filter estimates done-ness and the controller ejects toast when it has reached what we found by popular opinion to be the 'ideal' shade of toast. If you prefer a lighter or darker color, you can adjust the threshold for ejection on line 59 of "newControl.py" :

    if state_est >= 0.99: #Reached done state
 

Changing this percentage will yield different colors of toast!

All equations adapted from here:

https://www.kalmanfilter.net/kalman1d.html

The math can be seen in the above image.

The code that implements the Kalman Filter can be found in "kalman_filter.py"

Enjoy your automatic toaster! Here are some images of bread that we toasted to varying degrees of done-ness!