Kinect Hacking (Article)

When Microsoft released it's Kinect system for the Xbox 360 it was hoping to give a boost to the console and sell plenty of games. It certainly hit that target with some brisk sales, but the real surprise was that the Kinect's powerful technology also became a useful tool for programmers. Within a week of its release, the Kinect had become a fantastic high-tech DIY toy with a life of its own and it was all thanks to a worldwide network of hackers and a bounty of cash.

This article is one in a series of Instructables articles about DIY technology. The full list can be seen here.

For many years, video game controllers were straightforward: push a joystick to move and hit a button to jump or fire. Successive generations of gaming consoles added more buttons, another joystick, and a d-pad or two, but for the most part gamers were still moving sticks around and pushing on buttons. The evolution of the peripheral had a slow and steady pace.

In 2006, Nintendo shook this up with its release of the Wii console. In addition to the buttons and joysticks, the controllers could also detect motion. So instead of button pushes for actions, gamers were mimicking what was happening on the screen. Swings of the controller became bowling, baseball, and even dancing. These intuitive controls enabled lots of non-gamers to play, even becoming popular with senior citizens, and pushed sales of the console past 70 million worldwide.

Two groups were paying close attention to the Wii: Microsoft and hackers. Microsoft saw the potential in freeing gamers from controllers and started the Kinect project, originally known as Project Natal. With the Kinect, a pair of cameras record video and depth and are able to combine the information to precisely see the player's body position. Now no controllers were needed at all and gamers could play just by moving around.

Meanwhile, hackers saw the Wii's controllers as wonderful tools that were dying to be put to use for things besides games, such as head-tracking, virtual whiteboards, and controlling a 15-ton robotic arm. One such hacker, Johnny Lee, made such cool use of the Wii that he was hired on by Microsoft for the Kinect.

The Kinect looks like a freaky wide webcam. In fact, there are two cameras in it: an RGB camera and an infrared camera. The RGB camera works like any webcam would, picking up the visible light in the room that it's in. The real work starts to happen with the infrared camera.

With an infrared camera the Kinect can see the room it's in no matter what the lighting is like. This is important since many folks like to play games well into the night. To make sure that it has something to read there is also an infrared projector that shines an array of dots into the room. The infrared camera ses these dots and through a process called "structured light" the Kinect can create a depth map.

This video shows the infrared projection into a room.


With this depth information, the Kinect can create a skeleton of the player that has 20 joints. It's all done thanks to lots of R&D on Microsoft's part, analyzing terabytes of data to get a system that works without players even thinking about it. The jointed skeleton then allows for the Xbox 360 to have games that use players motions and gestures fo a more immersive experience.

So when Microsoft's Kinect was about to be released with its amazing potential to read users' movements, it was inevitable that people would want to use it for their own purposes. It was such an exciting idea that Adafruit Technologies put up a $1,000 bounty for the first person to create open-source drivers for the Kinect. With those, others would have the keys to the Kinect to create their own uses for it.

Microsoft didn't take kindly to the bounty offer and issued a statement saying, Microsoft does not condone the modification of its products. Upon hearing this, Adafruit Industries doubled the bounty to $2,000 and then $3,000 soon after. The race was on around the world, open to anyone with a Kinect and a computer and Microsoft wasn't happy about it.

Where the Wii hacks had helped to nurture a global network of hackers, the hunt for the Kinect drivers took it even further. Adafruit Industries helped out by buying a USB analyser to see all the information that was coming out of the Kinect and posting it online. Chat rooms lit up and updates were constantly being shared.

With so much attention it shouldn't have been surprising that open-source drivers were released on Nov. 10, just six days after the Kinect went on sale. Even more amazing, the winner was Hector Martin, a programmer who wasn't even in the United States, but northern Spain. Since the Kinect went on sale in the EU on Nov. 10, he had made the drivers on the same day that he bought it. Shortly afterwards, Adafruit Industries tested it out and declared him a winner: the search was over.

With the keys to the kingdom out there, Microsoft had a change of heart. In an interview Microsoft's Alex Kipman and Shannon Loftis said that the Kinect interface was left unprotected "by design and they're inspired" by community finding new uses. Whether this was the Kinect team's own desires or Microsoft giving in to the inevitable is up for debate. What's clear is that there's nothing they could do about it, but enjoy the new uses for their own device.

And now there are tons of Kinect hacks that people have created. New videos of cool uses are popping up almost daily. Invisibility cloaks, virtual puppetry, turning people into puffy 3D characters, and so much more. After putting up the $150 for the Kinect, the main limitation is how much creativity and time people can put into it. Do you have an idea that could use this? Then maybe you'll want to make a Kinect hack of your own.

DIY Kinect Hacking - Guide by Lady Ada
OpenKinect on GitHub - Share code
OpenKinect Google Group - Discuss
How You Become the Controller - Xbox engineers explain how the Kinect works