Making Your Alexa Smart-Home Magic With Smart Wand Remote

Unfortunately, I was born a muggle. Unfortunately, we are all born muggles. There is no magic. There are no potions. There are no flying brooms. Unless we make them.

This instructable will walk you through the steps of inventing magic, or at least the steps of making a wand that can control Alexa-enabled devices. For this project, we will be using an Arduino Nano BLE 33 and an ESP32. The Arduino Nano will be installed inside the wand and will detect gestures. Depending on the gesture, the Arduino Nano will send a signal to the ESP32 though BLE (Bluetooth). The ESP32, depending on the signal received (aka, depending on the gesture) will perform the magic (i.e. make an HTTP request, turn on an LED, etc.).

**AT LEAST A LITTLE EXPERIENCE WITH PROGRAMMING IS HIGHLY RECOMMENDED

**EVERY PHOTO IS ORIGINAL

• ESP32
• Arduino BLE 33 (You can use the Arduino Sense or the cheaper version.)
• 5V portable battery (The Arduino uses so little power that some portable batteries will cut off power to the device, so a battery with no amperage cut-off limit is necessary.)
• Computer with Arduino IDE installed
• Phone with the Alexa app installed
• Amazon Smart Plug or another device connected to Alexa
• Some structure for the wand itself (This can be 3D printed, fabricated with cardboard, or created with another approach.)
• 2 micro USB cables (I used a shorter cable to connect the Arduino board and the battery inside the wand. A cable with data lines is necessary.)

Overview:

We will be using Google's Tiny Motion Trainer to complete this step. This will create a model that will enable the wand to detect your wand movements with TensorFlow, which is a machine learning program.

Sub-Steps:

1. Download the code here to train the model.
2. Extract the .ZIP file.
3. Open the extracted directory.
4. Open the "arduino" directory.
5. Open the "tf4micro-motion-kit" directory.
6. Double click the last file in the folder. (It should be the .INO file type for the Arduino IDE.)
7. Plug your Arduino 33 into your computer with a micro USB cable.
8. Click the -> arrow in the top left to flash the board. (If the IDE fails to find the board, go to tools and change ports.)
9. Go to Google's Tiny Motion Trainer on a Bluetooth-enabled computer. Click on "Launch Experiment". Connect to your device by following the instructions given.
10. Click on "Capture Your Data". Create a label for each wand movement you plan to incorporate into your "magic". Create a model for each movement by clicking on the corresponding label and waving your board. Do each movement at least 20 times for a better accuracy rate. If you plan to 3D print a model and put the board inside, you should do that before this step. The board must be in the same location relative to your hand during training as it will be in the final product. For my first version, I taped the board and a portable battery to a ruler.
11. Once you have finished your training, click on "Train your model". This process only involves the computer finishing your model. Once you have high accuracy (at least 95%), you can test your model by clicking "Test Your Model". If your gestures are not being recognized correctly, you should repeat sub-step 7.
12. Once you are satisfied with your model response, you can download the model in the bottom right corner.

Overview:

Now that you have trained your model, you need to incorporate the model into your project. I merged the example code that Google's Tiny Motion Trainer provides with an example "weather station" for the Arduino 33 Sense I found online. The weather station allowed me to connect to the ESP32 over Bluetooth and to send the ESP32 "Notifications" containing the number corresponding to the gesture performed. Because I did not edit all of the weather station variable names and Bluetooth advertisements, my example code still carries a weather-station theme.

Sub-Steps:

1. Copy my code here (to the clipboard).
2. Again, I merged Google's example with a weather station example from online. Thanks to both sources for their examples.
3. Open your example code from Google's Motion Trainer (NOT the one you used prior to training your model).
4. Open that same file AGAIN, and save it as a new file. This will be YOUR version. Put YOUR version on one side and the ORIGINAL on the other side of your screen. They should be duplicates of one another.
5. Inside YOUR .INO version (the one you duplicated), paste the code you previously copied from my GitHub. Now, you should have your original file and your new version both visible.
6. Scroll though YOUR ENTIRE .INO file code and make the changes I've specified. I've added comments in places where you must personalize the code. Look for "Hey Change Me!" and follow the given instructions.
7. Save the project file and then click the -> to flash the code to your board once you have finished sub-step 5.

Overview:

In this step, you will program the ESP32 to receive the BLE signals sent by the Arduino 33 board. You will also process the signal sent to determine which gesture was performed and then perform the "magic".

Sub-Steps:

1. Copy my code here (to your clipboard).
2. Notice that I edited a Random Nerd Tutorial file. They don't support me, but I thank them for the example they offered.
3. Open a new file in the Arduino IDE.
4. Paste my code into the file.
5. Scroll through and find every occurrence of "Hey Change Me!". Follow the instructions to make the required changes.
6. Save the file, but don't flash it to the ESP32 yet.

Overview:

In order to control Amazon's Smart Plug with a webhook (HTTP request), you need to sign up for this third-party skill named Voice Monkey. Voice Monkey creates virtual doorbells in your Amazon devices. Alexa routines will, when triggered by the virtual doorbell being clicked (by the webhook), turn on a smart plug, a smart light, or any other Alexa device. The photo above shows my "monkeys". 0A turns the light on, 0B turns the light off, and so on.

Sub-Steps:

1. Open the Alexa app.
2. Click on "More" -> "Skills and Games" and search for "Voice Monkey".
3. Enable Voice Monkey and create an account.
4. On a computer, open Voice Monkey and sign in.
5. Click "Manage Monkeys" -> "Add Monkeys" and add two monkeys for every gesture you trained in step 1. (I trained two gestures and therefore added four monkeys.)
6. I named my gestures "0" and "1" in Google's Tiny Motion Trainer. I created "0a" to turn a device on and "0b" to turn a device off with one gesture, and then I created "1a" and "1b" to turn another device on and off with another gesture.

Overview:

In this step, you will be finishing the code for the ESP32 and then uploading the code to the board.

Sub-Steps:

1. Open the ESP32 file you created in step 3. Scroll to the "if" statements that are conditioned based on the status of your devices.
2. Replace the string "WaitForStepFive" (as it is shown in the photo above) with your monkey API calls. (Go to "Playground" on Voice Monkey and copy and paste the API calls one at a time into your code.)
3. Save the file.
4. Click -> to upload the code to the ESP32.
5. I was unable to upload my file with the Over the Air (OTA) update partition included. (My file was too large for the default partitions.) You can resize the partitions within the ESP32 by selecting the ESP32 Dev Module as your board ("Tools" -> "Boards"). I selected my partitions with "spiffs" but without OTA. Here is a great video explaining this process.

Overview:

In this step, you will use the Alexa app to associate the virtual doorbell rings (your HTTP requests) with tactile magic effects such as turning on a smart plug, a smart light, or any other Alexa device.

Sub-Steps:

1. Open the Alexa app.
2. Click on "More" -> "Routines".
3. Click the "+" button at the top right.
4. For "When this happens", click "Smart Home" and then click the appropriate Voice Monkey doorbell created in step 4.
5. For "Add action", click on "Smart Home" and then click the desired device (a smart plug, a smart light, or any other applicable device).
6. Repeat steps 3 through 5 for as many monkeys as you have created.
7. Power up the Arduino 33 board in your wand.
8. Plug in the ESP32 (the Arduino 33 must already be on before the ESP32 is powered).
9. Wave your wand!

You have a lot of options for fabricating a wand structure. For example, you can 3D print a wand, or you can put the board inside a staff or cardboard tube. So far, for my wand, I've covered the Arduino 33 battery and the board with spray-painted newspaper, as shown in the photos above. I will eventually make a 3D print for my wand structure.

When practicing your magic, please note that, because the ESP32 takes a few seconds to make the HTTP request, just wave your wand for a little longer in a motion that will not trigger a new gesture. For example, I trained an L-shaped gesture to turn my lamp on and off. If I draw an entire triangle with my wand, including the hypotenuse as an extra motion to fill the delay, the magic happens as soon as I finish waving my wand.