Make a Holographic Dance Party With Your Friends

I've had a funny little hobby for the past couple months -- I like to 3D scan my friends, then animate their scans and make them dance, and then I like to put all my friends into a holographic dance club inside one of the Looking Glass holographic displays that my company makes, because YOLO.

In search of the je ne sais quoi of holographic dance parties, I copied a page from the magnificent Oliver Garcia-Borg's playbook and added squishy physics interactions to the characters.

Want to join the dance party? This is actually really easy to do, and just takes about ten minutes once you're used to it. While I was learning this, I had to stumble my way through a bunch of 3D modeling and animation steps that were new and foreign to me, and this is my way of condensing everything I learned into a few simple steps. Ready? Put on some scanning music (mine is invariably despacito), and let's go!

p.s.

tl;dr? Just want to start playing with some dancing people? Grab an example program here and just go bananas!

Here's the basic steps to start from scratch and get a friend dancing in a hologram. I'll go into more detail later in the instructable, but this'll give you the big picture

• First, 3D scan your friend.
• Get the 3D model and make sure it has less than 50,000 polygons
• upload the model and texture to mixamo, and add rigging and dance moves there. This step is really fun, and if you don't have a holographic display handy, this is a fun place to hang out for a while.
• Get the FBX file from mixamo that has both the rigged model and the moves
• Bring the FBX file into unity. Fix up the textures and set up the animation
• Bango -- you've got your own holographic dance party

Here's some stuff you'll need:

3D scanner -- for scanning people, I really like to use the Structure sensor clipped onto an iPad, running the itseez3D scanning app. I do a lot of scans, and the reason I like this is that it's very portable, doesn't need a comuter, and it captures faces better than any other scan software I've tried. The structure sensor is medium-expensive -- if you already have an ipad, it'll run you about $400 for the sensor. If you want cheaper, you can use a kinect ($20 at gamestop nowadays) and Skanect Pro software, or for the ultimate cheapness (with associated frustration of working with lots of fiddly little pictures), you can use your phone or other 2D camera with some free 3D photogrammetry software like autodesk's own ReCap 360 (the descendant of 123Catch, which I used and thought was pretty o.k.), or just google 'free photogrammetry software' and see where the fates take you.

Good news! Everything else is free!

meshlab -- this is an open-source 3D model viewing and editing software that we'll use to reduce the number of polygons in a scan to something manageable

Mixamo -- it's an adobe title for rigging and animating 3D models -- everything's on a website, and you'll need to make a free account to use it

Unity -- If you just want to look at your model in boring ol' 2D on a webpage, you don't need this, but if you want to use your scan as a character in a custom video game or put them into a hologram, get Unity (free for hobbyists/indie game devs) and you're off to the races.

--------

I'm working on a release of holographic dance party software that doesn't require you to fiddle about in Unity. If this is more your speed, let me know in the comments and I'll see how fast I can go.

The first step is to get a 3D model of the person you want to animate. My scanner of choice is a structure sensor and the itseez3D ipad app running on an ipad mini. This isn't the cheapest scanner out there -- about $400 in hardware, not counting the ipad, and $7 to export a scan. I like it because it takes really good scans of faces, it gives pretty clean output, and it's portable and easy to use.

There are cheaper scanners, all the way down to some free photogrammetry software that take a series of phone pictures and try to spit out a 3D model. Cheap comes at a cost -- cost like warped geometry, artifacts, or blurred faces.

The tricky thing with all these scans is that they take about a minute -- you have to move all the way around your subject, taking images from all sides. People tend to move subtly over a minute -- especially if their arms get tired, and that gives you weird jumps and discontinuities in your scan. You want to get this done as quickly as you can.

Don't have a scanner? No problemo! Here's a model (of me? aw, shucks) from my scanner -- you can download it from sketchfab (be sure to download the original 'obj' model format, not the 'gltf' format) and follow along with the rest of the steps. Otherwise, carry on!

Here's the basics for scanning someone:

Put them in a big room with a bunch of open floor space and indirect lighting -- you want even illumination from all sides and no hard shadows. People walking in and out of your scene will mess up your scan, so keep everyone still while you're doing your scan. Make enough room for yourself to get all the way around your subject without having to bump into stuff.

Put your subject into an 'A' pose (see the title photo). Feet shoulder width apart, arms just far enough out to be distinct from the torso -- the wider the arms, the harder it is for people to hold still for a minute. If they're wearing glasses, take 'em off -- glasses come out badly in a scan.

Here's a walkthrough video showing me scanning the glorious Mr. Tschudko:

If you're using itseez3D, be sure that your scan is set to 'high-res' (click on settings, then flip the 'highest quality' toggle on). When you're done, tap your scan and click 'send to cloud for processing', It'll upload it to the cloud, then crunch your images, and in about 5-10 minutes, will send you back a 3D model.

If you like that scan, tap 'export' and my favorite way to get it out is to send it via email as an 'obj' file -- it's important that it's an OBJ, other formats will confuse the next steps.

If you're using another scanner, that's fine too -- just get a textured OBJ file onto your computer.

By default, itseez3D spits out models with too many polygons for mixamo or unity to handle. They come out of itseez3D with 200,000 polygons, and we have to shrink it down to 35,000 polygons.

To get started, open up Meshlab, and go to File->Import Mesh and point it at your OBJ file.

Once your model is up, go to Filters -> Remeshing, Simplification, and Reconstruction and select "Simplification: Quadratic Edge Collapse Decimation (with texture)"

Set the target number of faces to 35,000 and check 'preserve boundary of the mesh' and 'preserve normal'

Click 'apply' to finish and reduce the number of polygons in the mesh.

It won't look like much of anything when it's done -- that's the point. Go to File->Export Mesh As and select 'Alias Wavefront Object (*.obj)'' as the file type. Zip up the .obj and the model_texture.jpg file, and you're all set.

If all these instructions are finicky, try following along with this screencast:

Now you're all set to upload your model to mixamo -- mixamo is some free adobe software that can rig and animate a 3D model.

Rigging is what brings a 3D model to life -- the scan we just took is just a rigid object -- I can spin it around or make it bigger and smaller, but it's not really expressive unless I can move the limbs and joints. Rigging is a way of telling the computer where the model's bones are, and as the bones and joint move, deforming the skin of the model. Mixamo is a semi-automated way of taking a 3D model of a person and rigging that model. Once it's rigged, Mixamo has a library of animations that you can put onto the model with a click.

To start, go to mixamo.com and make a free account. Click on 'upload character' and upload the .zip file that we made in the last step.

Mixamo will walk you through the steps of rigging your character, or you can follow along with my video:

When you're done, you'll have your character rigged in Mixamo, and you can just click on any one of the hundreds of animations in mixamo's library and the character will do that animation.

If all you're looking for is a tiny nibble on the delicious twizzler of 3D animation, well then welcome to hog heaven! But I'd be remiss if I didn't mention the wonderful holographic world of endless possibilities that lies beyond. Step on in!

Have you ever played with a hobbled 3D environment, where you quickly run into arbitrary rules that you can't change? If that bothered you, Unity will be a delight -- it's an interactive game engine where everything is controllable, either by hand or by code. It's free for hobbyists and indie devs, and you can download it here.

I built a simple scene with music and a stage for a dance party -- you can download the whole unity project here, and it's a good template to get your own people on stage.

This is elaborate enough that I'm not going to write out all the steps -- if you just want to get to the action, you can just start playing around inside the unity project. If you want a walk-through, follow along with the video!

A wise man once said, "what is dancing without music", shortly before taking off his shirt, swinging it around his head like a helicopter, and launching himself bodily from the stage.

Point taken, sensei. In that same unity project, I have another scene in Scenes/Dancy Party that has the basics of an interactive dance party -- music, animations, and simple controls to change characters and animations. You can try it out just by hitting 'play' -- the up and down arrows change the dancing character, and the left and right arrow keys change the dance.

It just takes a minute to add your character to the party, as well as your own dances and music. Again, there's a series of fiddly little steps, but it's pretty painless to do. Follow along with the video! If you get stuck, leave a comment!

aaaand now, it's time for a SHAMELESS PLUG!

Every day, I wake up and I think about how to make wonderful things that blur the line between reality and fantasy. I started working on this project because I make holographic displays with my company, and while I was staring at a hologram on my desk, I thought it would be fun to have an animated dance party with my friends in a hologram on my desk. Like a bacchanalian diorama.

That's what got me started playing with this world of scanning and animation. As it so happens, the unity project you've been playing with also works with our holographic displays. If you have a Looking Glass or Holoplayer on your desk, just start the unity scene (or if you just want to see the damn thing work, download an executable) and hit the 'enter' key to toggle the holographic scene off and on. It shows a 2D scene by default because we live in a dreary, humdrum world where most people, oddly enough, don't have a holographic display on their desks. If that world feels unjust to you, help us change it!

You'll notice in the scene hierarchy that there's a game object called 'holoplay'. Inside there, there's another game object called 'HoloPlay Capture' -- this looks like a green box. Anything inside that green box will show up in the hologram. By moving the holoplay capture object around, you can frame different parts of the scene, zoom in or out and adjust exactly how the scene looks in the Looking Glass.

There's a more in-depth guide on how to use the SDK online here.

The cool thing about blending virtual and real worlds is being able to break the rules of the real world. I really like playing around in this crazy land, and I hope you do too. If you have any questions, or want to show something you made, put 'er down in the comments!

!!

--alex