Lets Make an Augmented Reality App for MEMES!

In this Instructable we are going to make an augmented reality app for Android and IOS in Unity3D that uses the Google API to search for memes. We’re gonna use Vuforia ‘s ground plane detection in Unity so this mobile app will work for most Android and IOS users. Using Vuforia will also allow us to have the pictures anchored in one location so we can walk through this field of pictures and the objects will stay where they are.

We are also going to test out the new IBM Watson API so we can do these searches with our voice and leverage their natural language processing.

So the bad news is neither of these API’s are totally free, but the good news is they are both free to try. The google custom search API gives you 100 free searches per day, and the IBM Watson API gives you the first month free.

In short, this app will get our speech from the microphone in Unity, send that to the IBM Watson servers, which will return to us the text. We will then take that text and send it to the Google servers that will return us a list of image URL’s in JSON form.

To get the Watson API going you need to first get your credentials from their site. Go to Console.bluemix.net, create and account, and log in. Go to your IBM account and navigate to cloud foundry Orgs and create a new space. Now go to your dashboard and click to browse services, add the speech to text service because that’s what we are going to use. Choose your region, organization, and space and create the project. Now you will see your API credentials at the bottom.

Download Unity if you don't already have it and import the IBM Watson SDK from the asset store in Unity. We can test this out by creating an empty game object and call it IBM Watson and add the example streaming script. This script is already set to record audio from unity and send it up to the Watson servers for processing.

For now we are just going to use this example script because we have a lot more to do but maybe next time we can go deeper into the Watson stuff because I would like to do something with the Vision API.

This script is looking for a UI text object so lets create a new UI button this will give us the text that we need, we will use the button later. Set the canvas to scale with screen size and resize the button a little bit. Anchor it to the bottom left. Drag that text into the empty slot. Open up the script and lets add our IBM Watson credentials, find where the "resultsField" text is used and set it to only "alt.transcript" because we are going to use this text to search google. Now before we can test this out we need to make the text size itself dynamically so whatever we say will fit inside the box. Go back to the text and set it to best fit. Type in some text to test it out. Now when we click play our words will be transcribed to text from the Watson Text to Speech API.

The next piece we need to do is get the Google custom search api set up to use in Unity. At a high level we will be making an HTTP request from Unity to the Google servers which will return us a response in JSON format.

So go to the Google Custom Search JSON API set up page, click to get an API key and create a new app. Keep this open. Now we can go to the control panel. Put in anything for the sites to search, name it whatever, and click create.

Click control panel and lets make some modifications: we want to mainly search memes, and turn on image search. Under sites to search switch that to entire web. Click update to save everything.

Now find the google api explorer and go to the custom search API. This will allow us to format the JSON response we get from Google. So put in anything for the query for now, paste in your search engine ID, put 1 in for the filter so we don't get duplicates, put in 10 under num because thats the maximum number of results we can return at a time, put in image for search type because thats all we want to return. Put 1 in for start, and finally under fields put in "items/link" because for each item returned we only want the image link. Now when you click execute you will see that we get 10 nice image links returned.

Now we have to get these pictures into Unity.

Lets get Vuforia working so we can leverage their ground plane detection. Save your current scene and go to the build settings. Switch your platform to Android or IOS and if your on IOS put something in for the bundle identifier, add a camera and microphone usage description. Under XR settings check Vuforia augmented reality supported.

Now in the scene delete the main camera and add a Vuforia ARCamera. Go to the configuration section and change tracking mode to positional. Uncheck all the databases cause we don't need them.

Now add a plane finder and we need to override its default behavior because we want to deploy the ground plane stage only once so lets find the Deploy Stage once script on the Vuforia website. Bring that script into Unity and put it on the plane finder, removing the old script that was there. Change the mode to interactive and make sure the "OnInteractiveHitTest" function gets called on that Unity Event. While we are here let’s set the button we made earlier to active once we have found the ground plane, set its default state to inactive. Now put a ground plane into the scene and change it to mid air because we want all the pictures floating in the air. Drag this ground plane into the empty slot on the plane finder.

Before we start putting all these pieces together we need to create a prefabricated game object that we can instantiate every time a picture is loaded. So create an empty game object under the ground plane stage and call it "picPrefab". Create a quad as a child of that and scale it by 2, rotate its y by 180 degrees so that the parents forward vector which is shown as a blue arrow is the front of the quad.

Create a new script called "PictureBehavior" and add it to our picPrefab.

Now drag this pic prefab into your assets folder and this is what we are going to put each picture on.

Our "PictureBehavior" script should look like this:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class PictureBehavior : MonoBehaviour {

	public Renderer quadRenderer;
	private Vector3 desiredPosition;

	void Start(){
		//look at camera
		transform.LookAt (Camera.main.transform);
		Vector3 desiredAngle = new Vector3 (0, transform.localEulerAngles.y, 0);
		transform.rotation = Quaternion.Euler (desiredAngle);
		//force into air 
		desiredPosition = transform.localPosition;
		transform.localPosition += new Vector3 (0, 20, 0);
	}

	void Update(){
		transform.localPosition = Vector3.Lerp (transform.localPosition, desiredPosition, Time.deltaTime * 4f);
	}

	public void LoadImage(string url){
		StartCoroutine (LoadImageFromURL (url));
	}

	IEnumerator LoadImageFromURL(string url){
		WWW www = new WWW(url);
		yield return www;
		quadRenderer.material.mainTexture = www.texture;
	}
}

Now lets drag in the reference to the quad renderer from our "picPrefab".

We only have two scripts left to make, so lets create a C# script called GoogleService.cs and PictureFactroy.cs.

Inside "GoogleService" paste this code that makes our request:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;

public class GoogleService : MonoBehaviour {

	public PictureFactory pictureFactory;
	public Text buttonText;
	private const string API_KEY = "PUT API KEY HERE!!!!!";

	public void GetPictures(){
		StartCoroutine (PictureRoutine ());
	}

	IEnumerator PictureRoutine(){
		buttonText.transform.parent.gameObject.SetActive (false);
		string query = buttonText.text;
		query = WWW.EscapeURL (query + " memes");
		//delete old images
		pictureFactory.DeleteOldPictures();
		//save camera forward vector so we can move around while objects are getting placed
		Vector3 cameraForward = Camera.main.transform.forward;
		//we can only get 10 results at a time so we have to loop through and save our progress changing the start number after every 10
		int rowNum = 1;
		for (int i = 1; i <= 60; i += 10) {
			string url = "https://www.googleapis.com/customsearch/v1?q=" + query 
				+ "&cx=011535004225295624669%3Afeb1gwic6bs&filter=1&num=10&searchType=image&start=" + i + "&fields=items%2Flink&key=" + API_KEY;
			WWW www = new WWW (url);
			yield return www;
			pictureFactory.CreateImages (ParseResponse(www.text), rowNum,cameraForward);
			rowNum++;
		}
		yield return new WaitForSeconds (5f);
		buttonText.transform.parent.gameObject.SetActive (true);
	}

	List<string> ParseResponse(string text){
		List<string> urlList = new List<string> ();
		string[] urls = text.Split ('\n');
		foreach (string line in urls) {
			if (line.Contains("link")){
				string url = line.Substring (12,line.Length-13);
				//filtering by png or jpg doesnt seem to work from Google so we do it here:
				if (url.Contains (".jpg") || url.Contains (".png")) {
					urlList.Add (url);
				}
			}
		}
		return urlList;
	}
}

Inside PictureFactory.cs put this code to create all of our pictures and loads their textures from a URL.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class PictureFactory : MonoBehaviour {

	public GameObject picPrefab;
	public GoogleService googleService;

	public void DeleteOldPictures(){
		if (transform.childCount > 0) {
			foreach (Transform child in this.transform) {
				Destroy (child.gameObject);
			}
		}
	}
	
	public void CreateImages(ListurlList, int resultNum, Vector3 camForward){
		int picNum = 1;
		Vector3 center = Camera.main.transform.position;
		foreach (string url in urlList) {
			Vector3 pos = GetPosition (picNum, resultNum, camForward);
			GameObject pic = Instantiate (picPrefab,pos,Quaternion.identity, this.transform);
			pic.GetComponent ().LoadImage (url);
			picNum++;
		}
	}

	Vector3 GetPosition(int picNum,int rowNum,Vector3 camForward){
		Vector3 pos = Vector3.zero;
		if (picNum <= 5) {
			pos = camForward + new Vector3 (picNum * -3, 0, rowNum * 3.5f);
		} else {
			pos = camForward + new Vector3 ((picNum % 5) * 3, 0, rowNum * 3.5f);
		}
		return pos;
	}
}

Create empty gameobject called GoogleService and put the "GoogleSerivice" script on it.

Drag the "PictureFactory" script onto the ground plane stage because all of our pictures are going to be created as children of this game object.

Drag in the appropriate references in the inspector, do the same thing for the google service.

The last thing we should need to do is make sure our "GetPictures" function gets called. So lets go to the "onClick" event of our button and call it from there.

Now we can click play and test this out. Make sure to enable the ground plane stage and the button. Say a word, and click the button to perform the search on that text!

Now to get this app on your phone, plug it in and go to File->Build Settings. Hit build and run!

Let me know in the comments if you have any questions!