Augmented Reality Fidget Spinner

In this Instructable we are going to make some fidget spinners in Augmented Reality and Virtual Reality with a DIY Hololens of sorts. This augmented reality tutorial will show you how to get the Leap Motion communicating with your mobile phone in AR and VR so you can use your hands to interact with 3D objects using a Google Cardboard style VR headset and your mobile device. We will use Unity 3D to create the 3D world and the Vuforia plugin to create the AR and VR apps that we will use.

In order to follow along with this video you will need a leap motion, a google cardboard style VR headset and a mobile phone. The leap motion can be purchased for about $80 bucks, you can also get a dev kit with a mounting bracket but I just have mine taped to my headset. You also need a VR headset with a window that allows the rear facing camera to be exposed. Mine cost about $10 from Amazon.

So the first thing we will do is get the leap motion tracking our hands on our desktop to make sure it works, then we will get hands showing up in mobile VR so we can interact with some virtual reality fidget spinners, then with a few extra steps we will be able to make the same project work in augmented reality so we can play with some fidget spinners in AR as well.

First lets test out the leap motion on our desktop.

Download the SDK from here: https://developer.leapmotion.com/sdk/v2

We are going to use an older SDK so it will work on both Windows and Mac.

Open up the installer and install the leap motion SDK to your computer. Open up the resulting app and click on the visualizer to get your hands tracking for the first time.

Plug in your leap motion and you should have something like what is pictured above.

Before we move on download the assets for Unity here: https://github.com/leapmotion/LeapMotionCoreAssets

Also, now would be a good time to download Vuforia for Unity if you don't already have it.

Vuforia can be had from here: https://developer.vuforia.com/downloads/sdk

Unity 3D can be downloaded from here: https://store.unity.com/products/unity-personal?_g...

Don't forget to install the packages for Android or IOS depending on what mobile platform you plan on using.

First of all the Leap Motion does not offer an easy way to communicate with your mobile phone. It hooks up to your computer with a standard USB cable and does not have bluetooth functionality or anything like that. The only option we have is to send UDP packets from our computer to our mobile device by opening up a socket and sending the data we need via WIFI.

Luckily, I already created this functionality for us, so let me walk you through getting it set up.

First of all download the Unity project from my GitHub account here: https://github.com/MatthewHallberg/DesktopServerLe...

Drag the FidgetStuff folder onto your desktop because we will need its contents later.

Open up the HandController.cs script from the assets folder and you will see a string at the top called "ip".

Change this ip address to the address of your mobile phone on your wireless network. Make sure you computer is also connected to the same network.

Go to file, build settings, and hit build and run.

Now you have the desktop application that sends the positional and rotational data of your hands from the leap motion to your mobile phone.

Start a new Unity project and switch your build platform to Android or IOS.

Save the scene before we forget.

Drag the "FidgetStuff" folder into the assets folder of your project.

Import the Vuforia plugin to your project.

Now we need to set up our image target because we need a fiducial marker in order to display our fidget spinner in AR. We are not actually going to print it out in this case, but rather display it on our computer screen so that when the camera on our phone recognizes the image it knows to augment the model with respect to that image.

So, go to the Vuforia website and create a new license key from the dev portal, copy it to your clipboard.

Create a new image target database and add the fidget spinner jpeg from the fidget stuff folder. Set the width to 5 and name it whatever you want. Download this image target database as a Unity Package and drag it into your assets folder as well.

Normally we would create a mobile VR scene in Unity with the Google VR plugin but Unity introduced native VR integration into their latest version (5.6) and it does not yet seem to work for IOS. In order to get this project to work the same for both Android and IOS we are just going to use the Vuforia plugin to create our VR app (this will also make it easier when we transition to AR).

First of all drag the "ARCamera" prefab into the scene from the Vuforia, prefabs folder. Click on Vuforia Configuration in the inspector and change the mode to device tracking, choose video see through.

Change the digital eyewear setting to Vuforia, Cardboard v1. Enable device pose tracking and change model correction mode to head. Uncheck enable video background.

Paste your license key into the box at the top.

Check the box to load your image target database, and click activate.

First we need to import some assets for the leap motion so go to the Leap Motion Core Assets folder that e downloaded and navigate to the Leap Motion folder. Hold command or ctrl and highlight the Textures, Materials, Models, and Prefabs folders. Drag all 4 of those into your assets folder in Unity. This might take a few minutes.

Lets create the ground so right click in the hierarchy and create a 3D object, plane.

Drag in any darker material from the "Materials" folder and set the Y position of the plane to -.80.

Now we need to get a skybox so we have a nice background. Go to the Unity Asset Store tab and search "Cope Sky Box." Download and import the first listing.

Back in the scene view go to Window, lighting, and change the skybox to "Mossy Mountains" or any of the other ones we just downloaded.

Right click in the hierarchy and create and empty game object. Call it "HandController." Drag this new game object on top of the "ARCamera" game object, making it a child.

So we have our hand control game object but it's not going to do anything yet. Drag the HandControlMobile.cs script from the fidget stuff folder on top of the hand control game object. This add the script as a component.

The script will have two empty slots for the left and right hands. Go to the Leap Motion folder and under prefabs their will be a "human hands" folder. Inside there find the "peppermediumcutleft" and "peppermediumcutright" prefabs. Drag them into the scene and add a box collider to each game object. Scale it accordingly because this is what is going to detect the collisions with the fidget spinners. Try to make the box colliders cover the entire hand. Hit apply in the upper right hand corner of each hand to save the changes. Drag each hand into the appropriate slot of the hand control game object.

Drag the fidget spinner 3D model from the "FidgetStuff" folder into the scene. It will have 3 child game objects so you can go to the materials folder and drag on a different material to each child object.

Add a rigid body component to the fidget spinner and freeze all rotations and positions except for the z axis (this will allow it to spin). Set the mass equal to 0 and uncheck use gravity. Change interpolate to extrapolate.

Scale it to .05 on the x, y, and z. Move it to somewhere just outside the ground plane. Set the x rotation to 145.

Add 3 box collider components and make each one cover each end of the fidget spinner.

Finally we just need to add some C# code to our fidget spinner in order to keep it from changing positions and to make it spin a little faster.

Right click any where in the assets folder and create a new C# script and call it FidgetControl.cs

Add this code:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class FidgetControl : MonoBehavior {

	private Vector3 originalPosition;
	float xAngle;
	private Rigidbody rb;

	void Start(){

		originalPosition = transform.localPosition;
		xAngle = transform.eulerAngles.x;
		rb = GetComponent<Rigidbody>();
	}

	void Update(){
	
		transform.localPosition = originalPosition;
		transform.eulerAngles = new Vector3 (xAngle, 0, transform.eulerAngles.z);
	}

	void OnCollisionExit(Collision col){

		rb.AddForce (col.transform.position);
	}
    }

Save this script and drag it on to your fidget spinner object, adding it as a component.

Duplicate your fidget spinner and create as many as you want, moving them around accordingly.

Now to get it on your phone go to file build settings and click add open scenes.

If you plan to build for Android, make sure you have usb debugging enabled on your phone, plug it in to your computer and hit build and run.

DO NOT CHECK ENABLE VR!! Native VR integration with Unity will not work with this method.

If you are building to an iPhone or iPad make sure you have Xcode downloaded. Create a free apple developer account here: http://www.idownloadblog.com/2015/12/24/how-to-cre...

Hit build and run. This will bring up Xcode, just choose your team and click play to get it onto our phone!

Go back to Unity and create a cube in the scene. This is what is going to represent our hands for right now because I could not think of a better way.

Change its scale to 2, .5, and 2.

Drag it into your assets folder to create a prefab, then drag the prefab into both slots of the hand controller script on the hand controller game object (replacing the hands).

Go to the AR camera configuration and check enable video background.

Go to its child camera and change the clear flags from sky box to solid color.

Drag the image target prefab from the Vuforia, prefabs folder into the scene. Drag one of your fidget spinners on top of this image target, making it a child. Delete the rest.

Change this fidget spinners position to -1.219, .62, and .14. Change its x rotation to 145. Finally, change its scale to .02, .02, and 02.

Click on the image target and activate the image target database. Choose the image you want to track.

Delete your ground plane and you should be ready to build this out to your phone for mobile AR!

Start the desktop server application on your computer and put your phone inside your VR headset. Pull up the fidget spinner image on your computer. Make sure your computer is connected to the same network as your phone. As long as your leap motion is plugged in, you should now have hands in augmented reality and you should be able to interact with the fidget spinner!

Keep in mind the desktop application will not run in the background so it always has to be slightly in front of the image target in the foreground.

Thanks for looking! I would love to see what everyone does with this so please leave a link to your own projects in the comments!