Sign In

Remember Me

Real Virtuality

Augmented reality (AR) is a technology that has made truly extraordinary steps moving from science fiction to reality in the last year alone, and seems poised to explode into a variety of new applications in the imminent future.

The basic idea is to take video and overlay it with virtual 3-D objects, text, or sound in realtime, either on a handheld screen (such as a mobile phone or Nintendo DS) or a head-mounted display. The educational, entertainment, artistic and business applications are nearly limitless. The technology allows a blurring between the real and virtual on a scale that’s never been seen before.

In one recent research project, Tobias Lang and Blair MacIntyre at Georgia Tech used a head-mounted display and a Second Life client to overlay objects and avatars from Second Life into the real world. The head-mounted display included a camera, to take video from the perspective of the user; a computer, to engage with Second Life and reprocess the video; a location sensor, to give the computer data about where the user is looking; and an internal projector, to project the finalized image onto the eyes. For now, most augmented reality “stages” are stationary. A small piece of patterned paper serves as the “fiduciary marker” to tell the head-mounted computer where to project the virtual scene.

A very simple example of augmented reality would be the yellow “first down” line seen on broadcasts of football games. This helpful graphic serves as a sort of memory enhancer for viewers. Instead of having to imagine where the first down line is, they get to see it automatically, outsourcing the cognitive task to the computer. Watching football might not be the most intellectually challenging activity, but the basic principle of outsourcing cognitive tasks obviously has potential applications beyond sports broadcasts.

To get an idea of how much augmented reality has started to explode just in the year 2009, chew on these stats: looking at Google Trends, searches for “augmented reality” exploded to four times the five-year average in 2009, starting from just the average level at the beginning of the year. Games Alfresco, a leading AR blog by Ori Inbar, founder of Pookatak Games, recently nominated 18 augmented reality games for its “Games Alfresco Hall of Fame Award.” Thirteen of the games were made in the last half of 2008 or the first half of 2009, including some by big names like Sony (Eye of Judgment, Invizimals), IBM (Seer Android Beta), and McDonald’s (Do the Dip). Most of these games are somewhat crude, but the fact that AR games are being made at all shows that the era is dawning.

The simplest augmented reality games use mobile phones with cameras. Nokia smartphones are among the most popular platforms in the currently limited AR gaming market. One AR game for select Nokia phones, Fanta Virtual Tennis, lets you play tennis with a friend using a virtual ball and your phones as racquets. The “court” consists of a piece of paper with designated fiduciary markers (location markers) so that the software in the phones can determine their spatial orientation. Another interesting AR game for Nokia and the iPhone, Kweekies, is a Pokémon-like game where players customize animal avatars and send them into battle.

Wikitude

Other AR games involve moving around in the real world instead of just playing other people on a stationary AR stage. Wi-Fi Army for the GPhone allows “real-life first person shooter” action, where your phone is your “gun” and crosshairs are superimposed on realtime video. The goal is to locate other players in the real world and “shoot” them by taking pictures of them with your phone camera. The GPhone then compares captured pictures against a database of known player faces and awards points for successful “hits.” There is a global leaderboard and the number of players is unlimited. Wi-Fi Army was the first game to be developed for Google’s Android operating system, and one of the only games in the world that simultaneously makes use of camera, Wi-Fi, and GPS functions on cell phones.

The mobile operating system Google Android is starting to emerge as a prominent competitor for Nokia in the augmented reality application space. One application for Android called Wikitude is an AR travel guide that displays information associated with prominent landmarks on G1 phones. It recently became available on the iPhone 3GS. Development kits like ARToolKit have hit the mainstream in the last couple of years, making it easier for developers to build applications across any platform with the essentials: a camera, screen and sufficient computing power. After over a decade of research, AR is finally hitting consumer markets and becoming feasible for extensive development.

Augmented reality (AR) is a technology that has made truly extraordinary steps moving from science fiction to reality in the last year alone, and seems poised to explode into a variety of new applications in the imminent future.

Because of their ubiquity, many emerging AR games make use of the humble webcam and everyday PC as a platform. The July 2009 issue of Popular Science bills itself as the “first interactive 3-D augmented reality magazine cover.” Focusing on the future of energy, the cover features an image of windmills with a skyline in the background. When the user visits a specific web page and holds the magazine up to a webcam, the live video feed on the computer screen is overlaid with a 3-D pop-out image of windmills with the Golden Gate Bridge in the background. Blow into your microphone and the windmills spin faster. Trying it out for myself, I found the 3-D cover exciting, but noticed that even slightly turning the magazine away from the webcam resulted in the 3-D image disappearing. Still, it’s an early version, and exciting that it works at all.

Many AR blogs and experts refer to the French company Total Immersion Software as the world’s leader in augmented reality. Their technology certainly is impressive. Demo videos for Total Immersion clearly show that the company is way ahead when it comes to AR graphics. Their flagship application is RealWorld®, which allows non-programmers to quickly build and test 2-D/3-D geospecific simulations. The program is marketed for “time-critical applications where lives are on the line — first responders, counterterrorism professionals, warfighters, healthcare workers, the intelligence community and security forces.” Clients include the Department of Homeland Security and the Department of Defense, including the Army, Navy, Air Force, Marines, and the Special Operations Command.

Total Immersion Software hasn’t just received the attention of the military and the government. A wide variety of software and entertainment companies are also getting involved. Total Immersion is one of eight French companies that Microsoft has chosen to support software innovation in Europe. Three major TV networks have used their technology to project virtual objects into their programming. Disney Imagineering is testing the technology for use at theme parks. The company also has a contract with Boeing to use the technology in its development of new planes. AR is a favorite with FedEx’s CEO Fred Smith, who has used the technology to send virtual packages flying around the room during in-house presentations.

Real VirtualityBoth Sony and Microsoft have been pursuing augmented reality. In late 2009, Sony is scheduled to release its EyePet game for PlayStation 3, which uses the PlayStation Eye camera to create a virtual pet in augmented reality for the gamer to interact with. Another game, Eye of Judgment, released in 2007, comes with physical game cards and uses an augmented reality playing mat and camera to project 3-D characters battling it out on the TV screen. Most cards are associated with summoning creatures.

Microsoft has not released any AR games yet, but at the 2009 Electronic Entertainment Expo in June, they announced the ongoing development of a “controller-free gaming and entertainment experience” for the Xbox 360 console currently called “Project Natal” after Natal, Brazil. The Project Natal sensor device includes an “RGB camera, depth sensor, multi-array microphone, and custom processor running proprietary software.” The software will allow advanced gesture, facial, and voice recognition. If the gamer is close enough to the sensor, it can even map the movement of individual fingers. Scheduled for a rollout in late 2010, Project Natal could initiate a new era of augmented reality gaming, if it lives up to its promise.

As is practically always the case with emerging technologies, the most exciting applications of AR are either research projects, still in the developmental pipeline, or purely conceptual. Dozens of big companies are currently developing AR applications but have not deployed them yet, including Mini, Toyota, Nissan, BMW, Lego, Ray-Ban, and Holition (jewelry). The potential of using augmented reality to take online shopping to its next stage has been widely recognized in the retail industry, with the falling costs of high-resolution webcams setting the technological stage for widespread adoption. In five to 10 years, online shoppers looking at jewelry, sunglasses, hats, and even clothing may use augmented reality to “try on” virtual products before they buy.

A key application of augmented reality that has only just begun to be probed is virtual “x-ray vision.” Using digital eyewear like the Vuzix Wrap 920AV, recently revealed at 2009 CES in Las Vegas, engineers and maintenance technicians might be able to “look inside” machines without ever taking them apart, allowing for better analysis of complex systems and faster repair. Combined with microsensors, augmented reality could provide an intuitive way to analyze the features of systems that are not immediately obvious through visual scrutiny alone. In the further future, augmented reality instruction manuals could replace paper manuals entirely, giving the user all the information they need without even requiring them to take their eyes off the product.

Digital eyepiece-enabled augmented reality seems like a way we can have our cake and eat it too.

The most awe-inspiring visualizations and presentations of augmented reality are completely fictional. Generation X was introduced to augmented reality by shows and movies like Star Trek, Terminator, and Robocop while Generation Y often associates the technology with the “scouters” in Dragon Ball Z. In Dragon Ball Z, digital eyewear with augmented reality is used for communication, mapping, object tracking, even evaluating the strength of opponents. Augmented reality has been a staple in Japanese animation for decades, appearing in series including Gundam, Neon Genesis Evangelion, Voices of a Distant Star and Martian Successor Nadesico. Augmented reality plays a central role in the award-winning series Denno Coil, which the major Japanese newspaper Mainichi Shimbun said could impact future technology in the same way that the book Snow Crash helped inspire the development of Second Life.

One of the most extensive fictional portrayals of augmented reality appears in Vernor Vinge’s book Rainbows End (2006). The world in the book includes a ubiquitous augmented reality that has replaced conventional screens as the primary medium for accessing the Internet and communicating with others remotely. The characters in the story use contact lenses to view augmented reality, removing them only to sleep. By combining augmented reality graphic overlays with haptic feedback and robots, the story presents a world where the line between the virtual and the real is thoroughly blurred. Numerous AR worlds are available for any user, but the most popular are built in collaborative units called “belief circles.” In the novel, entire parks are used exclusively as augmented reality playgrounds.

Perhaps the most exciting potential of augmented reality is that it has the ability to bring people back together after modern technologies have done so much to break us apart, spatially if not socially. Using augmented reality to surf the web and do computer work without having to remain seated in a chair could improve the public health by leaps and bounds and save hundreds of billions of dollars in health care costs. Despite the best efforts of pro-exercise government programs and kids’ TV shows like LazyTown, obesity is more of a problem than ever. Thirty-two percent of the children in the United States are overweight or obese. Digital eyepiece enabled augmented reality seems like a way we can have our cake and eat it too: benefit from the entertainment and educational abilities of computers while also experiencing the sunlight and physical activity of outdoors.

Michael Anissimov is a writer and futurist in San Francisco. He writes a blog, Accelerating Future, on artificial intelligence, transhumanism, extinction risk, and other areas.

%d bloggers like this: