Animating Action Figures in Augmented Reality!

In this instructable we're going to use free software to take an action figure, import it into a 3D modeling program, clean it up, animate it, and then use Augmented reality to view the figure. This can be a super fun thing to show a kid...their toys actually dancing or moving on a phone or tablet as if they're in the 'real world'. You can even scale a child's toy up to their size! But here's the most important thing...we're going to use free software in a clever way to do this, and it's going to be much easier than you can ever imagine. If you want to do this...you can do this.

Note: I'm entering this into the multi-discipline contest! Please vote if you think this is worthy!

This process has some parts that are better explained through text, and some that are better explained through media. I've uploaded photos where I can with teal lettering to highlight parts of the program that you'll want to be aware of for that step. I've also made a youtube video (above) that I recommend viewing as it shows the steps in action (though it glosses over some of the nitty gritty, and I've gone into much more technical details here as something optional you can read through). I designed this instructable with 2 viewers in mind: 1) Those that want to have fun and just make stuff dance and 2) those that want to learn the technology behind it.

The other important thing is that this is going to cross over several different disciplines. So when you've completed this, you'll know skills that can be used for 3D modeling, game creation, art, 3D printing, animation, or just making fun videos!

Real quick, the basic steps we're going to do are:

1) Photogrammetry - This is basically how you capture photos of an object and get it into a 3D model

2) Cleanup - When you import the model, we'll want to remove things that aren't related to the 3D model itself

3) Animating - We're going to use cloud based software to automagically do this, but I will explain what rigging is, and a bit of how this works

4) Viewing / Augmented Reality - This is were we output our final model and then load it on our phone/tablet to view it as if it's in the real world! You'll be able to take video of the model or snap photos as well.

I don't want to get too wordy for this one, as the fun is in the making, so let's get to it!

The main things you'll need are easily accessible:

1) A camera - A cell phone camera will work great for this

2) A PC - To import the models we use Meshroom, which works in WIndows and Linux. I haven't found free photogrammetry software for MAC, if you do please comment below and maybe I can adjust this to add that in.

3) An action figure that is a humanoid with posable arms - You can do half of this instructable without the posable humanoid, but if you want to animate it it's required that the figure has 2 arms, 2 legs, and a head. Being able to pose it in the shape of a "T" is ideal.

4) The following software:

For Photogrammetry - MeshRoom (https://alicevision.org/)
To Optimize the mesh: - Meshlab (http://www.meshlab.net/)

For Cleanup / Decimation- Blender (https://www.blender.org/) Rigging / Animation - Mixamo (https://www.mixamo.com/#/)

To view it in Augmented Reality - Aryzon AR Studio (https://www.aryzon.com/ar-content) - Available on IOS and Android

5) To view the figure in Augmented reality, you'll need a phone or tablet that works with Aryzon AR Studio. I believe most modern ones will (I used an iPhone 7Plus). If you don't have a phone, this will still show you how to import and animate the figure, which can be entertaining in itself :)

The first thing you'll want to do is photograph the character. These photographs will be sent to an application to build a 3D model of the action figure. To that extent, you have to take photos of every angle of the figure. And since the application that analyzes the photos needs to match things up in the photos, you have to be careful to make sure there are no strong shadows and things are evenly lit.
To do this, place the character on a table, and pose it with it's arms out, legs and arms straight, standing upright. Place it on a flat surface that you can walk around the character and take photos. I've found that a kitchen table, or outdoors on a patio table during a cloudy day, work well. Try to use the lighting in the room as much as possible to keep things evenly lit...adding a light to one side or the other will hurt the results. Don't move the character while taking photos, and don't worry about what is in the background of the photos...we'll crop that out later. Also don't stress this too much...if you get photos that can't be used or miss spots, I'll show you how to go back and add more photos in step 2. But please note..you don't want to move the character until you've completed step 2 if possible! Ok, to keep things simple, here's a bullet list of things to keep in mind. Also please note...this is probably the toughest part of this, so once you're past it you're in good shape!

• I usually do 3 full body passes, of about 20 shots each, slowly changing angles around the character. I do some from a slightly higher angle, some from level, and some from lower angle
• I usually do a pass of just the lower and upper body to get about 20 more pictures
• I then do 10 shots or so close up on details like the face, or any part of the boy that has something intricate
• Try to keep your camera set so that it doesn't blur any part of the photo. If you're using a DLSR, open up the F stop/aperature to 5 or 6. (Thanks to Josef Prusa for that tip!)
• Try not to let a shadow from you or the camera come over the object. Try to avoid harsh shadows or bright spots
• Note: Shiny (Chrome or reflective) objects sadly don't seem to work. If you have a shiny object, you can try to add something like chalk dust to it to take a way the shine, but your results may vary
• I set my character on a textured floor...I didn't put paper or anything behind or under the character. I think this helps the software orient the photos...so if you have graph paper or a granite counter top (Something with a pattern) I felt like that helps greatly
• The number of pictures isn't important, but getting all the angles and elevations is. I took about 120 photos to get a decent image from an 8" action figure.

This step has photos which I used to show some examples, but the last one is zoomed out to show what the setup looked like...just an action figure on a kitchen counter!

Ok at this point you should have a ton of photos. You'll want to copy them to your computer into a single folder. Protip for iOS users...I used an app called simpletransfer (https://rambax.com/simpletransfer/)...this let me transfer them wirelessly over my network to my PC. In the end the important part is to have the photos in a folder, and to be ready to take more photos and transfer them if needed. Android users can likely move them via SD card, or you can always use the cable transfer method.

In the intro I have links to the software. For this step we're going to use Meshroom. Download and launch Meshroom, and then please look at the image for this step.

Meshroom has 4 sections...the photo upload section on the left side (labeled 'step 1'. The middle has a photo viewing section that will reflect whichever photo is selected. The right side is where the results show up. There are 2 main results we care about:

1) Point cloud - A point cloud is like a bunch of floating dots that are show in X, Y, and Z axes. If you don't know much about 3D graphics, just think of them as like a 'connect the dots' version of the 3D image we're trying to create. It takes much less time to generate the point cloud as the full image, but it's close enough for us to get a feel for how the computer is doing with the images we've given it. If it's not doing well, we can just try to give it more! (Please see 'dot cloud' image in this step for an example)

2) Final textured mesh - A "Mesh" is a 3D figure in computers. A Texture is like the 'paint' or 'skin' on top of it. So if you view a raw mesh it will typically look colorless. The Texture, when added on top, is what we usually see in games. This takes much longer to generate, but the good news is this app is doing that work for us :) (See the 'textured' image in this step for an example)

Before beginning save the project and keep in mind where you save it. I'll refer to this location throughout the instructable as the 'mesroomcache' location (and that's likely what the folder will get named).

The first thing you want to do is to copy your photos over to the right pane. They'll show up in a list (First image above) and the second pane will show the currently selected image. The first thing we want to do is generate the point cloud, and see if we should gather any more images. One thing to note...on my PC the point cloud takes about 5 to 20 minutes to generate, based on the number of photos. The full workflow takes 2 - 6 hours. So to save time, we generate the point cloud, which is relatively quick, and then examine it to see if we need more data before running the full pass.

To run half of the work flow we should talk about the fourth and final pane at the bottom. This consists of boxes which represent each piece of the workflow. The only thing you have to worry about here is that one is called "Structure from Motion". Right click on it and select 'compute" (There's an image above showing exactly where to hit.

The progress bar will appear at the top, when it dissapears you should have a point cloud in the right pane. Zoom with the mouse, and rotate with the left mouse button. Also if you hold down the middle mouse button you can 'pan'. Those controls should allow you to get into your model and look at the dots. See the image in this step which shows the two controls on the right...one is for camera size, and one is for dot size. Lower those both so that the cameras get out of the way and you can actually see the points. Then look to see where there are gaps. This is where you'll want to take more pics.

Also, the photo icons on the left pane will have new icons in the upper right of each picture...either red or green. If any are red, then that photo was rejected for some reason. Those are ones you'll likely want to retake to recapture the detail.

At this point...recapture any photos as needed, import them to your PC, and then simply drag them to the left pane...this will add them to the list. Don't worry, the work you've done up till now is still relevant...just right click again and go to 'compute'. You shouldn't have to wait nearly as long as the first time, and you should see the details fill in. When you fee like you have a decent amount of details, then click the green 'start' button at the top.

This takes a VERY VERY long time...hours....sometimes overnight. I typically run this overnight and take a peek in the morning. Click on 'texture' node in the bottom pane, then look at the left pane...if you have multiple entries there, click the eyeball to filter out the point cloud (I have an image on this step showing what this looks like). It may take a second, but you should get a textured version...it's going to have more than your action figure, and will likely have the table it's on, and anything around it. The only thing we care about is the action figure...look at the details and make sure everything is captured. If not...add more pics, and repeat!

When you have a figure that looks complete you're ready to move to the next step...were we delete all the crud around the figure. The good news...you're past the second hardest part, and the rest is relatively fast!

Save your project in case you need it later. Look at your meshroomcache location where you saved the project, you should see a folder which matches each 'step' of the way...look in the texture folder for your model...it's the one that ends in .obj. OBJ is a file format representing a 3D model, and we'll be working with that from here on out for the most part.

If you want to understand this deeper, read on (but this is optional!)

The OBJ file is the 3D model. There will also be a .mtl file, which is a material file. You can open that in notepad, and you'll see that it references any png files that are part of the texture. If you look in the same folder you should see those png files (PNG is a lossless image file format). If you open those PNG files they may look a bit scary...but really they're the flat texture that gets applied to the OBJ file. You can imaginge having a box (the model) and wrapping paper (the texture)....basically when that flat square is wrapped all around the model, the texture lines up and it appears as if the entire model is painted.

It looks like this:

newmtl TextureAtlas_1001

Ka 0.6 0.6 0.6
Kd 0.6 0.6 0.6
Ks 0.0 0.0 0.0 d 1.0

The Ka, Kd, and Ks specify how the texture should react to lighting...things like selectivity. So the app not only examined our photos and generated a 3D model, but it also figured out how to generate a texture file and properties of that texture. It may not have gotten them right, but just the fact that it did all this is honestly a marvel of how far computer science has come. This is thanks to the awesome work of the Alicevision.org project.

For further reading of this format and to understand how this works, I highly recommend Paul Bourkes description: http://paulbourke.net/dataformats/mtl/

Ok, so the good news is that this step is made really straightforward and easy. Now that the 3D model you've generated, you'll notice there is a bunch of stuff that isn't the model....and we want to just delete ALL of that stuff! So we'll fire up the next free program, Meshlab.

Go to your meshroomcache from Step 2 and open up the object file via the File->Import Mesh menu in Meshlab. (note: in 3D programs you usually use 'import' to pull in a mesh, and then use open/save to save the project. So we'll be using import a lot from here on out)

The controls for Meshlab are the same as Meshroom's 3D model view:

• Middle Mouse button down = Pan (move up/down left/right)
• Left click to rotate the model
• Mousewheel to zoom in/out

Now that you can move around, rotate your model so you can get a good look at things. What I recommend is that you go for a top down view to start, and zoom out so your model is in the middle. This process is gradual...you are trimming pieces away in as big of chunks as you can, and just keep whittling away at them, zooming in and working closer and closer to the model.

In the photos for this step you'll see two arrows...these are the two menu items at the top you'll want to use for this step. Select the one of the left (Select faces in a rectangular region).

TIP: once you do this...you can't rotate the model or move anything until you've selected something! If you feel like you get 'stuck' then select something small and then you automatically go back into move/rotate mode

Select a bunch of faces, you'll see them highlight in red. Stay clear of your model, and if you do hit it, just select again. The selection is rectangular. You don't have to get everything in one shot. Once it's selected, you'll want to delete it by hitting the button shown by the second arrow in the photos (it's labeled: Delete the current set of selected faces, and those not referenced are not deleted" in the tooltip).

Once you click that dlelete button, you should see everything highlighted in red disappear. Repeat this process deleting around your model. Once you have most of it cleared out, zoom in, delete more, rotate as needed, and repeat. In the end try to get it so that just your model is there.

Select 'export mesh' and keep the defaults when you're done. Next up we'll simplify the mesh and get it ready for animation!

Ok the next program we'll load is Blender. Blender can be intimidating, and I'm not great with it. More good news...we're going to use simple features, and even if we screw up things are pretty forgiving.

Quick tutorial to get around...movement in blender:

• Middle mouse button down and move - Rotate view
• Mouse wheel - Zoom in / out
• If the 'move' icon is selected (Please see image for this step..the icon with 4 arrows) - Left mouse button down + movement Moves the model

• If the 'rotate' icon is selected - Left mouse button down + movement rotates the model

I've uploaded an image of the default blender UI and labled the sections we care about so that you can easily and quickly find them. We need to do a few things in this step:

1) The model is probably not 'centered' where we'd want it, and is likely off to the side or sideways/upside down. We're going to orient it in the 'middle' facing forwards just to make the next step easier.

2) The model consists of 'faces' which are the flat sections that make up the model. When two faces meet that's called an 'edge'. When multiple faces meet at a point it's called a vertex (verticies as a plural). The auto-animation software we're using works best if the number of faces in the model is under 200,000. The model, even with all of the stuff we cut out, is likely over that, so we're going to fix that.

To prepare, delete the default square, lights, and anything else you can delete in the upper right corner (labeled in the first image for this step). You'll likely delete a camera, a light, and the cube.

Import your mesh just as you did in step 3...just this time in blender (File->Import->Wavefront (.obj). Select the OBJ file you've been working with.

Now we can get to work. From the object menu at the top, select 'set origin' and then 'set geometry to origin'. This will move the mesh to 0,0,0 where we want it. You can then use the move/rotate tools from the left hand tool bar to move and rotate your character so it's facing along the green line.

For the next part click on the wrench on the right side (please use the image attached to this step for a guide). Select 'add modifier' and then select 'decimate' from the tons of options that appear. You should see the decimate modifier populate...and as part of it is a 'ratio' slider. Here's how we figure that out...

Look at the number of 'faces' shown in the lower right corner of Blender...it'll show the number of faces. We want that number to go under 200,000. So if it's 400,000, then we need a 50% reduction...so the ratio would be .5.. Calculate your reduction ratio (it doesn't have to be exact, we just need to get under 200,000...) and then click on ratio and type it in. After a second you'll see the number of faces update...if it's under 200,000, you're golden. If not just keep lowering the .ratio. When you have the number you want, hit 'apply'. You'll see the menu disappear, but the faces should be the new updated lower number.

Excellent, export the obj one more time and lets get ready to animate (The most fun part!)

I'll run you through this part quickly, but as a second portion of this I'll take the time to describe what exactly is happening, as it's good stuff to know if you get into this.

For this final portion, we're actually going to upload the model to a free website called "mixamo" which was a company acquired by Adobe. So to start navigate to http://www.mixamo.com

Signup

Note that it requires an account, but you can sign up for free. Since it's a cloud service it's handy to have an account that you can use to store your files and load them later.

Rigging

Once you've logged in, you'll see animations on the left, and a menu on the right. This app is super simple..just click 'upload' and then once again find your .obj file that we've been using throughout.

The first thing it'll do is analyze it...if you get an error...well that bites. There's not much that can be done other than to post here or on a reddit forum and hope for some help. I've done all of the things I know how to do in the previous steps to get the model ready for mixamo. It does have a link for some helpful tips, please read through them.

Otherwise, you'll get a window showing the model, ready to 'mark'. The first thing I'd usually do is deselect 'symmetry'. When a model is made from scratch, it's usually symmetrical, but since we're using photogrammetry, it's usually not quite even.

Labeling the model

Next up drag the 'wrists' 'elbows' 'groin' and 'knee' circles over to your model to match up each joint. The second image for this step shows an image of me going through this process to give you an idea.

From the drop down at the bottom I usually select '25 bones' (The simplest skeleton) just because the fingers don't usually come out in photogrammetry. If your finger joints look good though, give it a shot! Worst case you just redo this step.

Ok...then click next, and wait and hope....it usually takes 2 -3 minutes or so....and if everything worked, you'll see an image of your model looking around. If you're here...CONGRATS!!! You've done something amazing, and you're about to have, I hope...a TON of fun!

Once you have the model looking around, click on the animations on the left. The search bar lets you filter it down to superhero moves, dance moves, combat, etc. There are hundreds of animations ready to go from the Thriller dance to zombie moves, from super hero martial arts to just walking around. Have some fun watching your model move around. Note that the texture is likely not going to show up, but that's ok, we'll fix that next.

Exporting / Saving

Once you have your model animated, then select "Download' and then select "FBX" (not the FBX for unity unless you're going to make a game next). Download this model into it's own folder, then go to your meshroomcache (where you got the obj) and copy any PNG texture files into the FBX folder as well.

FBX files should have the textures embedded, but for some reason I haven't had luck with that. So if you copy the textures into the folder with the FBX, then ZIP those files (in Windows you can right click and "add to archive", though I find using 7Zip is good here). This zip file you'll want to upload to a cloud service such as onedrive for the next step.

Some technical notes:

"Rigging" is the process of adding 'bones' to a 3D model. When 3D models are animated, it's often through the use of a rig. The rig consists of bones which have joints, and constraints which describe things like how far and in which direction a joint can bend. For instance, if you bend your elbow....there's limits to which way and how far. The bones that make up the skeleton in a 3D model are very similar. RIgging can be a very complex thing to do for a model...and yet this software just did it for us by analyzing the model and guessing where the bones should go!

Animations for 3D models are usually just frame by frame descriptions of where the joints and bones for a rigging should go. So you don't necessarily apply an animation to a model, but to the skeleton. Then the model reacts based on where the bones are and how it needs to move.

This is really handy because you can have a library of animations captured that describe a humanoid (head, arms, legs, torso) and then apply it to thousands of different models...be it a ninja or a monster, a dancer or a super hero.

So what's happened here is Mixamo has rigged our model for us, and then shown us a library of animations which we can apply to the skeleton. When you apply them, you see the result. When you save it as FBX, it saves not only the model, but also the skeleton, and the animation. this is how we can load this into Augmented reality and see our model move.

One last neat note...if you load your model into Unity or Unreal, you could build a video game with your model, as they know how to apply animations to models. The logic may go "if the joystick is moved up, then play the walk forward animation." "if the A button is pushed, play the 'jump' animation. In this case Unity/Unreal would hold the animation files. And that's your gateway into game programming or even movie making with this technology! :)

Get the app

Forthis portion, I've been using AR Studio. I've tried 3 or 4, but had issues finding one that supported zip files and showed the textures. And without the textures, it's just not nearly as cool!

I got it from: https://www.aryzon.com/ar-content. It's free to use, but has unlocks if you pay. For this the free version won't restrict us in any meaningful way.

Once you've installed the app, click on the 'library' button on the bottom, click + to add to the library, and then add your file. When I click 'browse' I see Microsoft Onedrive, Dropbox, iCloud Drive (this is on iOS), and 'on my phone' as options. So you'll want to get that zip file from the last step onto one of those, and then add it to your library.

Once it's in the library, click 'AR view' and enjoy! You should see a textured version of your model dancing around the area you placed your phone. If you click the camera icon you can then record a video of that and share it to amaze your friends/insta follower/whathave you.

This instructable came from me stringing together a few ideas and thinking 'wow, if I could scan a kids toy and make it dance that'd be cool'. Then while trying to do that, I found out I could show it in AR, and have the toy dance next to itself...or scale it up and work as lifesize. That was pretty wild to me as a concept.

But also, like may concepts...along the way I solved a bunch of important problems that I've shared here, which result in skills that you can take forward and do amazing things with.

Photogrammetry could be used when traveling to try to capture a sculpture or statue. It could be used to create a 3D model that may be 3D printed.

The animation work could start from a predefined model or from one created for you. You could apply the auto-rigging to any 3D model that is humanoid..so if someone made a character for you or you got one from the tons of free repositories, you could animate it.

The rigged models are incredibly useful in game creation.

The cleanup/decimation work we did in Blender can help for came creation, movie making, 3d modeling, etc.

So out of this goofy instructable are some great skills. Also, we just used some amazing cutting edge work in image analysis and artificial intelligence as well as leveraged a cloud service to do things that would've taken hours or weeks by professionals using thousands of dollars of software just ten years ago (and maybe even 5 years). Of course doing it by hand would return better results, but what we've accomplished in this instructable should not be undersold.

Thankyou for reading and taking this journey with me. I hope you learned and became aware of a bunch of fun tools and techniques along the way. Please let me know in the comments if you hit any issues and please share what you create, no matter how goofy! I'm "Farrojoe' on instagram and "geektoolkit" on youtube!