IoT Motion Controlled Servos

Secure and reliable real time data streaming is essential for IoT. I've seen plenty of demonstrations involving applications or "push button here, LED on over there" type hardware, but a friend and I wanted to make something that was more interactive... a way to almost feel the data stream as you manipulate it. So, we decided to build a motion controlled "thing" that mimics your hand movements and displays colors based on finger positioning. Hopefully, with my help you'll be able to build your own!

A conceptual overview and some more in depth look at some of the source code can be found on the PubNub blog, the complete source code is available in this GitHub Repo, and a separate Instructable detailing the LED matrix driver circuitry has been written.

There are a lot of pieces to this puzzle, and I will try to list out the major things.

Parts and Pieces

• Raspberry Pi - I am using a V1 Model B+
• Leap Motion Controller
• 5V Power Supply - There are a lot of power hungry parts here, better make it 5A!
• I2C Enabled PWM Driver - Adafruit has a nice one.
• 4 x Micro Servos - Cheap TowerPros will work.
• 4 x Micro Servo Mounts - I modified these from Lynxmotion
• LEDs and Current Limiting Resistors - For visual effect, but not necessary.
• Project Enclosure - This one is laser cut by Ponoko, more on that later.
• Screws, bolts, nuts, zip ties - and other such means of fastening things together
• Cable Wrap - To keep the wires neat and tidy (on amzaon).
• Power Jack, Switch, and Cable - So as to get power from the wall to the circuitry (on amazon)!
• On-Off-On Slide Switch - Used to select Servo output modes
• Small Push Button - Used as an R Pi reboot / shutdown button
• RGB LED + 2 x 470 ohm resistors - Reboot / shutdown indication
• Stand-Offs - To Mount componentes, These cheap ones on amazon are great.
• Hook Up Wires - So as to connect one thing to another

The RGB Matrices sitting on top of the servos are using a custom built driver. This part of the project is fairly advanced, and a separate Instrucable has been written for it.

Software and Services

• PubNub - Data Streaming Service. Free Sandbox mode for developers!
• Java SE - JDK 8 - Be sure to get the correct version for your operating system
• Leap Motion Visualizer and Java SDK - Again Be sure to get the correct package for your operating system
• PubNub Java SE SDK - This needs to go into the project libs directory
• Java IDE - Use your favorite, such as JGrasp, NetBeans, IntelliJ, or Eclipse
• Project Source Code Repository - Hosted on GitHub

Raspberry Pi Setup

• PubNub Python Library - This is an in depth guide for setting up PubNub on the R Pi
• Internet Connectivity - This can be WiFi or Ethernet. If you can ping Google, then you are good to go.

Tools and Such

• Computer - Writing code, viewing of the Instructables, etc
• Electric Drill + Drill Bits/Drivers - Drilling Holes, Driving screws, etc.
• Dremel Rotary Tool - For cutting holes in the enclosure... may not be necessary for you.
• Hot Glue Gun - Is there any other way to stick something to something else?
• Soldering Iron + Solder - Don't breath the fumes!
• Laser Cutter - Again, I used a service, but you can DIY if you have your own machine laying around.
• Screwdrivers, Pliers - Stuff you should have by now!

Like I said, there are lot of pieces to this puzzle! At the top level, there are three main components:

1. Computer with Leap Motion Controller - publishing data to the internet
2. Raspberry Pi in "the box" - Subscribing to data from the internet
3. PubNub - the Communication layer securely connecting these things.

The computer part is simple enough, but in "the box" there is a bit more going on...

1. Raspberry Pi - Primary Controller, talks to other parts using I2C Bus
2. ATmega328p Matrix Driver Circuit - Receives commands via I2C Bus, controls 2 8x8 RGB matrices
3. TLC5916 Based LED Sink Circuit - Controlled by ATmega328p, Sinks LED matrix current
4. Adafruit PWM Driver - Receives commands via I2C bus, drives 4 servos
5. 5V, 5A power supply - Brings everything to life

To start, we need to set up the user computer to use the Leap Motion Controller.

Java Setup
We will be using the Java SDK for the Leap Motion controller, so you will need to install the Java Development Kit which includes the Java Runtime Environment. This is pretty straight forward, just choose the correct package for your operating system.

Rather than create an installation package, you will be running the source code directly from a Java IDE. This allows you to play with the code yourself, modify things, and learn! Once you have the JDK installed, you need a good IDE (that is, integrated development environment). Some common ones are Eclipse, IntelliJ, and NetBeans. Use your favorite, or pick one at random. Setting up libraries is a bit different in each, so you will have to refer to the documentation on how to setup a new project!

Java Source Code

There is only one java file needed to run the code, but you will also have to install the Leap Motion and PubNub Java libraries... The GitHub project repository contains the necessary java file. I'll talk a bit about the code in the next step.

Leap Setup

Setting up the Leap can be a bit tricky, and the process is different for different systems. Rather than step you through how I did it, I will direct you to the appropriate Leap Motion articles. You can always ask for help if you get stuck somewhere.

1. Download the SDK, Drivers, and Applications from Leap Motion.
2. Install the Leap Application, which also installs any necessary drivers and processes.
3. Run the Leap Motion Diagnostic Visualizer to make sure Leap is working.
4. Read through the Java SDK Documentation, particularly setting up a project.

PubNub Setup

The last step is to download the PubNub Java SDK. The documentation page talks about how to add the library to your Java Project.

If you are not interested in the java code, you can skip this step; however, you will at least need to put your personal PubNub keys into the code for it to work! The source code can be viewed on GitHub.

The following imports are required. If these result in any errors, you probably don't have the SDK libraries installed correctly.

import java.io.IOException;
import java.lang.Math;
import com.leapmotion.leap.*;
import com.pubnub.api.*;
import org.json.*;

It is crucial that you make your project implement Runnable so that we can have all Leap activity operate in its own thread. We begin by setting up the project main, an implementation of the Runnable interface and initializing of global variables we will be using later. The main things to take note of are the global variable "CHANNEL" as well as a pair of keys declared in main.

public class LeapToServo implements Runnable{
	public static final String CHANNEL = "leap2pi";

	// ... A Bunch of Code ...

	public static void main(String[] args) {
		String pubKey = "Your_Publish_Key";
		String subKey = "Your_Subscribe_Key";

		// ... A bit more code ...
	}
}

These strings are the only values that you will have to alter. The channel name can remain the default "leap2pi" or can be some other alpha-numeric value. Everything using PubNub for communication talks over a channel, so you want to use a unique name to prevent cross talk between various projects! The publish and subscribe keys are unique personal identifiers given to you by PubNub when you register for an account. Keep them safe. Keep them private. These keys prevent other people from talking on your channel(s). Take note, there might be a few places in the code that specifically use the string "leap2pi" instead of the constant "CHANNEL," and all of these instances should be changed.

The Leap Motion captures about 300 frames each second. Within each frame, we have access to tons of information about our hands, such as the number of fingers extended, pitch, yaw, and hand gestures. The values returned by the Leap for pitch and yaw (wrist pan and tilt) are in radians, which doesn't help us too much. We want to convert these values to a format accepted by the PWM driver in the box. We first convert radians to degrees, and then degrees to an acceptable servo PWM value between 150 and 600. This range corresponds to the typical working range of a servo (500 - 2000 us) as represented by the 12 bit servo driver.

The basic equation is 2500us / (2^13 -1) * (Servo Min/Max Pulse Width), so [500, 2000]us maps to [150, 600]... roughly... In the code, these conversions are handled by the following methods.

public static int radiansToAdjustedDegrees(int radians){}
public static double pitchDegreeToPWM(double degree){}
public static double yawDegreeToPWM(double degree){}
public static int normalizeDegree(int value){}

Also, please excuse the incorrect servo and PWM terminology in the code... not everyone who worked on this project was an expert in these subjects!

The best way to ensure your code is working is to load the PubNub Debug Console. From here, you can enter the name of your channel as well as your publish and subscribe keys. Hit "Subscribe," and your published data should show up in the messages box as you move your hands over the Leap Motion Controller with the Java code running.

With the Java code working, it's time to setup the Raspberry Pi to subscribe to that data and use it to drive the LEDs and servos. You must have already configured a Raspberry Pi with a working internet connection; this can be WiFi or Ethernet, but it's up to you to get that part done! If you need help, PubNub wrote a great article a while ago. Also, you can do any of these steps directly on the Pi using a monitor and keyboard, or remotely using SSH. Either way is fine, but the end goal of the project is to have a stand-alone, headless, setup that automatically runs the necessary files on boot.

The first step is installing the PubNub Python SDK.

Open a terminal, and install the following:

1. Python: pi@raspberrypi ~$ sudo apt-get install python-dev
2. pip: pi@raspberrypi ~$ sudo apt-get install python-pip
3. PubNub: pi@raspberrypi ~$ sudo pip install pubnub

Well, that was simple enough! Now we need to get a copy of all the files found in the Pi directory of the GitHub repository. The easiest way to do this is to clone the repo using git, and then get rid of all of the other stuff we don't need:

1. > sudo apt-get install git
2. > git clone https://github.com/pubnub/LeapMotionServoBots.git
3. > cp -ar LeapMotionServoBots/Pi leap2pi
4. > rm -rf LeapMotionServoBots

Now, the contents of the leap2pi directory should be identical to the files found in the Pi directory of the GitHub repo. Finally, we want the python scripts to run at boot. There are two python scripts we want to run:

• servo.py - The code which subscribes to PubNub and drives the LEDs and servos
• shutown.py - A shutdown button monitor to turn the Pi off.

This is simple enough, we just need to edit a Linux system file as the root user. First, make sure the scripts are executable, then open the rc.local file for editing.

1. > chmod +x leap2pi/servo.py
2. > chmod +x leap2pi/shutdown.py
3. > sudo vi /etc/rc.local

You can use whatever editor you like, but I prefer vi. The last line of the file should be "exit 0" which allows for the clean exit of the file and the initialization of the Bash terminal. We can add any number of commands here, as long as they also run the final "exit" command.

Insert the following, just above the "exit 0" command:

python leap2pi/servo.py &&
python leap2pi/shutdown.py &&

Save the file, exit, and on boot, the Pi will automatically run those scripts. The "&&" parts make sure that every command is run. If you leave that off, the bash terminal will never load, so you will be locked out of your Pi indefinitely!

The Pi can be used to directly drive servos, but this isn't easy. There is one dedicated PWM channel, but we need 4. It can be faked in software, but again, this isn't the easiest thing to do. Being a computer, the Pi is much better at higher level processing than low-level control. You can certainly use any PWM driver for this step, but I am using this 16 channel, 12bit driver form Adafruit. Yes, it is overkill, and yes, it is easy enough to program an ATtiny24 to command 4 servos while communicating on the I2C bus. I encourage you to create your own, but that is beyond the scope of this guide!

There are plenty of online tutorials to get you started with this board, and you will need a few additional source code files. These can be found in the Pi directory of the source code repository - the files named "Adafruit ..." will tell the Pi how to talk to the PWM driver.

Connecting the Pi to the board is easy. The Pi's I2C lines are found on Pins 3 and 5, but you also need to enable the I2C communication channel if you have not done so already. One important input pin is labeled "OE." This is an active-low "output enable" pin. The servos will only be enabled when this pin is held low. It can be connected directly to ground, or driven from another of the Pi's IO pins. In this project, I am actually driving it from the ATmega328p matrix driver circuit which also connects to the R Pi using the I2C data lines.

In the diagram, a servo is attached to channel 1 of the PWM driver. In the project, set up the servos like so:

• Channel 0 is Left Yaw (Pan)
• Channel 1 is Left Pitch (Tilt)
• Channel 2 is Right Yaw (Pan)
• Channel 3 is Right Pitch (Tilt)

The python code to control the servos is pretty straight forward, as long as you understand Python. Personally, I'm not so much a fan of this language, but that is beside the point. You can read along with this copy of the servo.py file on GitHub.

Just like in the Java code, you need to put your personal PubNub publish and subscribe keys in here, as well as the name of your communication channel.

The Pi will do the following on boot:

1. Reset the AVR matrix driver circuit.
2. Initialize PubNub with your keys
3. Subscribe to the PubNub channel "leap2pi"
4. Loop forever, checking an output mode switch (more on this in a minute)

Underneath the hood, the PubNub library is handling all of the hard work. This one subscribe call takes care of everything, we just need to specify a few callbacks - these are functions that are called when an event takes place.

#Subscribe to subchannel, set callback function to _callback and set error fucntion to _error pubnub.subscribe(channels=channel, callback=_callback, error=_error, connect=_connect, reconnect=_reconnect, disconnect=_disconnect)

Callbacks regarding the connection should be obvious (connect, reconnect, etc), but the one that does the most work is the aptly named "callback" callback. This function actually does something with the message we receive from the subscribed channel. As mentioned above, the box has a couple of output modes selected by a slide switch connected to a few IO lines.

• MIRROR - The bots will mirror your movements; hence, your left hand inversely controls the right bot
• DISABLED - The servos will be unresponsive
• CLONE - The bots will clone your movements; hence your left hand directly controls the left bot

The logic for these modes as well as the I2C driving statements are handled in the _callback function.

The only other item of note in this file is the use of GPIO pin 4 as an output. This pin is driving the gate of a MOSFET which connects an array of blue LEDs to ground. This pin is enabled on a connection to PubNub, so the LEDs act as a connection indicator. This is a very important aspect of headless setups - there needs to be some indication to know we are successfully connected to the internet!

This script is a lot easier to follow than the last, but is just as important. Since the raspberry pi is a fully functional computer, it should be properly shut down. Just killing power to it can result in drive failure, data loss, and memory corruption.

To combat this, I installed a simple push button on the back of the box with an RGB LED indicator. Holding the button for at least one second will turn the LED blue. This will command the Pi to reboot when the button is released. Holding the button for an additional couple of seconds will cause the LED to turn red, signalling a full shutdown.

This functionality is handled entirely in the shutdown.py script. In a "forever loop," the following takes place:

1. Sleep for 0.25 seconds
2. Check for a button press (low state on the pin)
3. Repeat forever
   1. On Button press, Sleep for 1 second
   2. Recheck the pin to see if the button is still being held in
   3. If button is still held in, we might want to reboot... If not, keep checking!
   4. Set RGB LED as blue
   5. Sleep for another 2 seconds
   6. Recheck the pin to see if the button is still being held in
   7. If button is still held in, we want to shutdown!
   8. Set RGB LED as red
   9. If not, reboot!

To build the physical box, we had our friend Eric create a couple of vector files in Illustrator (see the attached files). These were sent to a laser cutting company (Ponoko) which sent us a bunch of laser cut pieces of wood and acrylic. The pieces came back great, but I had the joy of assembling them.

If you are careful, you can drill pilot holes in the acrylic (as well as the wood) and use thin screws to mount the pieces together. If you don't want to go that route, then feel free to build any sort of enclosure that you want. The standoffs from the parts list should be used to keep the circuit components firmly mounted to that base. I also cut a few additional holes in the back of the box to account for the power inlet, push button, and RGB led.

The most important thing is the mounting of the servo rigs. As they move around, the servos will cause any light weight mounting frame to fall over or jump around. It is crucial that they be firmly mounted to a sturdy surface - we chose the lid of the box!

The servos themselves were mounted to the Lynxmotion aluminum frames, but I did have to bend the tabs in a bit to get the servos to fit as they are slightly smaller than the intended servos for these mounting pieces.

And that about does it! I know I glossed through a few parts, but I think a lot of this will have to be customized, especially building the actual box and servo mounts. Get creative, and do let me know if you have any questions!

Motion-controlled Servos with Leap Motion & Raspberry Pi from PubNub on Vimeo.