For years now, technology companies have promised us that “augmented reality” — technology that allows you to look at the world under a layer of superimposed computer-generated imagery — was just around the corner. It’s been a compelling promise. The ability to place and manipulate digital objects in analog settings is so appealing, both as entertainment and as a pragmatic solution to real problems, that Facebook, Apple, and Google have all prominently announced or demonstrated their own AR programs over the last few years. But of those three, Apple will be the first to roll out augmented reality to hundreds of millions of phones. And it’s going to be huge.
Apple’s “ARKit” (the development tools used to create augmented-reality apps and environments) is a characteristic synthesis of its tightly integrated software and hardware. It uses a combination of your iPhone’s camera lens and its internal gyroscope and accelerometer to calculate, within centimeters, where and at what angle your phone is being held. This let’s your phone detect horizontal planes (i.e., floors and tables), to precisely and accurately layer objects or information on the screen. This version of AR, which uses only a single camera lens, won’t allow for fancy things like occlusion — if a digital object goes behind a table, ARKit isn’t sophisticated enough to realize it should make part of that object disappear. But it does mean that when iOS 11 goes live later this fall, the iPhone 6s and every iPhone newer than it, along with most iPads sold in 2016, will be able to run augmented-reality apps.
Since ARKit’s announcement in June 2017, videos of several extremely tantalizing demos have been circulated online. Over the last week, I’ve spent some time looking at various AR-enabled apps created with ARKit, some created in just a few weeks. A few were less than impressive, but some were a stunning vision of how AR could quickly become as much a part of everyday life as smartphones and touchscreens have become.
AR for Kids
On the entertainment side, there were two apps that particularly impressed. One, meant for children, was based on Eric Carle’s classic children’s book, The Very Hungry Caterpillar. A plushy toy crawled around, munching on apples or pears or strawberries dropped by the player. Anyone who’s spent time around a small child knows the fascination smartphones already hold for them — but the ability to suddenly turn their living room into a small world of fruit trees, caterpillars, and eventually, butterflies was stunning. Barry O’Neill, Touch Press CEO and creator of the app, said children tended to interact with their parents a lot more while using the app, versus the zombie stare that screen time can sometimes induce.
AR for Games
Another game, Arise, played out a bit like a game of Monument Valley or Echochrome. After scanning for a flat surface to place the game on, players physically circle a tower, using perspective to turn what looks like a gap into a bridge a small soldier can walk across. The mechanics aren’t new, but having the player move a phone screen around to control the camera is. Similarly, a Walking Dead AR experience essentially had you scan for footprints and point your phone at zombies (er, “walkers”) to shoot them — not groundbreaking stuff, but fun in the context of suddenly seeing zombies shambling through your living room or local park.
AR As Tools
Finally, there was a demo from Ikea, where you could scan a room and then begin to place down furniture. AR interior decorating has been around for a while, but the speed and ease of use of Ikea’s app stood out. If a sofa didn’t look quite right, you could swipe through a couple of different designs — or different colors — until you got what you wanted. The ability to plop down a virtual armchair and ottoman, and then walk around the room to see how it looked from the entryway or a sofa you already bought, was impressive; it’s the type of thing I wish we’d had when recently buying furniture for our home.
Early Days
The demos shown were very impressive, but phone- and tablet-based AR is very much in its infancy. Many of the apps shown were built in a matter of weeks. But with hundreds of millions of iPhones and iPads capable of running these AR apps relatively soon, I think the vocabulary of AR is going to expand at a tremendous pace, the same way our use of touchscreens and smartphones has wildly shifted.
Early iPhone games mainly attempted to either re-create gamepads on screen, or use your pointer finger as a mouse. Then someone realized how satisfying pulling back, releasing, and destroying buildings was, and Angry Birds was born. Or that carrying a camera around in your pocket all the time meant that you could have a running feed of an idealized version of your life, and you get Instagram. By using ARKit and putting it on so many iPhones all at once, Apple has a good chance at allowing developers to be the first to come up with that sea-change moment.
