Kinect: A Powerful Interface Between our World and the Virtual

If you remember my very first article on H+, you might remember this quote: “With the upcoming release of “Project Natal”, the Emotiv Epoc neural headset, a variety of video glasses such as Limus’s see-through glasses, and a computer or console game, basic virtual reality is now a reality.”

It’s called the Kinect now, and with all the news recently about its being ‘hacked’ I thought it was worth revisiting that original concept and cover it a bit more in depth and explain why the Kinect is so very much more than a “mere” console game controller.

If you are not yet aware of this fact, the Kinect is a combination Lidar Camera device. It uses infrared light like a radar, tracking how long it takes a beam of light to reflect off of the body to make a 3d “depth map” of the space in front of the device. This, combined with the stereo microphones and the full color camera, enables the Kinect to know exactly where your body is at any given second, and to thus control your onscreen “you”.

As a console controller, the reviews have been a little critical, mainly highlighting that the Kinect more or less requires you to be exactly 6 feet away to function correctly, that the menus and such are not the easiest to use, etc. Basically, for all its sophistication, the Kinect is less than the ideal console controller. Which, to be honest, I expected. Game makers really don’t know how to use the whole body in a “casual” game, and “immersive” games like Shooters can’t really make use of the Kinect without two additional elements, video lenses, and an omnidirectional treadmill.

But console gaming is not really the point, not for those who recognize the device for what it truly represents. Despite threats of litigation from Microsoft, a bounty was offered within hours of the Kinect’s release for a means to use the Kinect on a PC. Within days, it was claimed. The result?

Well, one of them is that Kinect is suddenly opened up to a whole slew of new uses other than Microsoft’s rather limited use as a game controller. Willow Garage, a robotics company, has used them to replace their previous robotic vision sensors with the Kinect:

Note that not only are they using the Kinect as a vision system for the robot, they are using one as a telepresence unit for a manipulator arm. Another thing to note is the video footage of the “world view” of the Kinect. Like one of those pin tables that you can impress your hand on, the Kinect sees in straight lines, and by the speed of the return of each of those “lines” build a 3D model of the world quickly.
But Willow Garage is not the only group connecting the Kinect to a robot, as MIT is also playing with them:

As you can see, the robot can not only map it’s environment, it can recognize humans in it, and take hand commands. I got this video from an article over on the Singularity hub. The reason I mention this is because I was struck by an offhanded comment the author made about “Don’t expect Kinect sensors strapped to quadrotors anytime soon.” It made me laugh because I had already seen this video:

This is from the STARMAC project at UC Berkley. As it shows, weight is certainly not the issue, though battery life with current Lithium Ion batteries probably is.

However, the really interesting thing is that this is a gigantic step towards the subject of my recent Quadcopter article (Fly Your POV Around with Your Own Personal Quadcopter, The Remote Telepresence Unit.

While currently no power supply exists that would enable a true RTU, these videos show rather well that the Kinect can be used to not only give the RTU sight, but that it could enable both control, and the ability to use telepresence via manipulators on the RTU.

Yet even this is just part of the value of the Kinect, because MIT is also working on a gestural interface:

Note that the top shows the screen of the computer, and the bottom shows exactly what the Kinect is seeing. Note the fine detail of the figure, even though only his hands are being actively tracked. This is much closer a realization of the true power of the Kinect than it’s marketed use. Imagine using this to control a virtual body, and an Emotiv Epoch to track fine details like facial expressions, and tracking your area of focus in a virtual world, or even a virtual user interface for your personal computer.


And then think about that pair of wraparound video lenses, and that omnidirectional treadmill…

Yeah, the Kinect is a game controller, but more than that, it is an interface between our world that of the virtual, a means to seamlessly connect the digital and analog. It may not be the Matrix, and it may not be the best controller for every kind of game, but for what it truly is, the Kinect is well named.



    IEEE Spectrum has pulled together videos about Kinect Hacks for a Top 10 list.

Leave a Reply