Cheap AR
AR is the new VR that never was.
Virtual Reality promised to create engrossing environments to fool the senses and transport the participant to amazing places and magical experiences. There are a few efforts still moving in that direction, but have drifted away from the head tracking helmets and input gloves used in earlier efforts and more into an externalized (from the perspective of the participant) infrastructure: surrounding the subject with lots of screens and integrating with more natural input modes. High-speed 3D has been a nice addition too, but the cost and inherent limitations of the infrastructure are still a limiting factor.
Enter Augmented Reality, stage left. This has been an interesting evolution, and is still in its early stages of development as the mechanics and utility of features are worked out. It started, interestingly enough, as part of VR directly - incorporation of real details into virtual environments and the reverse. It's migrated away from the high-cost infrastructure and the gargoyle-like wearable computing that promised to make it portable into handsets - now ubiquitous technology with increasingly powerful computing performance. These are capable of motion-matching the surrounding environment and overlaying positional, geographic, or interactive elements onto a camera-captured realtime view, or in some creative instances literally projecting an interface onto something else and measuring interaction that way. One of the common themes of the camera-augmented view is the attempt to create a 3-dimensional feel: the view has essentially been downsampled to 2D, so stereoscopy isn't an option, and instead other standard 3D presentation mechanisms are used such as shading, occlusion, distance cues, etc., positionally represented based on the relative angle between the view and the virtual object. This is an interesting hack of modest utility, and when combined with viewer tracking can create some very convincing effects.
This morning I saw an unlikely (or at least unexpected) implementation of the concept on my way in to work that has started me thinking. A 3D rendered scene (very simple, a few gift-wrapped packages) was shown on a digital billboard (in 2D of course, though given the distance the stereoscopy is essentially uniformly columnated anyway), and had a virtual camera shift on the display which corresponded roughly (and I do mean roughly) with the anticipated perspective shift of a driver moving at average speed on the freeway. There were some aspects that made it less convincing, such as the vertical offset being mismatched (the camera was looking slightly down on the package, as opposed to the driver's lower perspective) and the horizontal shift speed being off, but the gimmick caught my attention enough to get me thinking about the possibilities - especially the integration of time/condition aware presentation for lighting and shading, and what could be done with very little work to make it far more immersive. I'm not even sure that their use of the effect was intentional or if they were just happening to use fancy graphics while I was coincidentally driving by in a way to perceive it as such. No matter what, it was nifty and I'm sure we can expect to see more of it.
Persistence
Not too long ago I was fortunate enough to pick up an iPod Touch (bonus had come in from last year's work). The device has been remarkable, smaller than it looks and packing a nice little portable media punch. It also happens to be a fairly robust mobile computing platform, as evidenced by the strength of community development for the device even prior to the release of the official SDK. Being a developer myself, I was intrigued enough to give that part of it a shot.
Rather than put together an Intel hacintosh or risk the wrath of my wife by messing with her new Intel iMac (used for her wedding photography) as required to run the X Code IDE/SDK, I decided to use Jiggy's JavaScript based framework and IDE. While this surely introduces some functional limitations and performance overhead, it was also a very painless way to slap together a proof-of-concept to get a good idea of the problem space I was approaching.
Given that the device contains a set of accelerometers (used by the iPod/iPhone user interface to determine landscape and portrait layout), I thought it would lend itself well to a Persistence of Vision hack - which is to say, using a thin portion of the display, flash certain bits of it on and off as the device is shaken back and forth. The flashes correspond to certain spaces of the visual field during the swipe, and as a result of the mechanics of the human eye (which takes roughly 1/60 of a second to clear out the stimulus) appears to create many little spots floating in the air which can be used to write letters, display images, etc.
I first encountered the effect at a novelty store when I was about 9, when I saw a clock which consisted of a sturdy base and a spring-loaded wand which, when flicked, would flash it's LEDs as at waved back and forth and created the illusion of floating time. While several devices have picked up the gimmick since then, I've never actually bothered to own any of them. Now that the opportunity was right in front of me with a low barrier of entry, why not give it a shot?
First came the math. Working with my wife we were able to determine that a good hand-wave average was roughly 16", making a full cycle (back and forth) every 1/2 second or so (11 over 5 seconds). In order to achieve sufficient resolution to be able to display any letter of the alphabet discernibly, while still keeping the overall number of cells sufficiently low (so as to have low logical and processing requirements) we chunked this 16" section into 6 letters per wave, with each letter broken into 5x6 segments. 6 letters per wave is pretty low, but the idea was to have a word persist only through a single wave - as soon as the accelerometer detected a significant shift in direction it would load the next word, allowing sentences to be spelled out sequentially.
( 6 letters * 5 columns = 30 columns ) / 16" / 0.25 sec (single wave phase) = 120 columns / 1 second, or 120hz switching. In order for a line to pass through 1/2" of space during the course of a single wave, it would need to be on for only 0.008333... seconds. That's for a perfect line - I would of course be switching a region rather than a mathematical construct, so I would in theory need to divide that 120th of a second by the width of the region in order to achieve the target resolution. I fudged this part, and for the sake of the proof-of-concept development decided to have the region correspond to the width of the column itself - 1/2" square, which means that if everything worked perfectly a total of 1" would be illuminated, a 1/2" of virtual overlap with itself (area of continual brightness) giving a linear fall-off to either side. So the letters would look a little fat and maybe just a touch blurry - but that's acceptable to start with.
However: most displays only refresh at a rate somewhere between 70-90hz. If I'm trying for 120hz it means that each cell would fall quickly out of sync, ending up only partially drawn or have some frames of display skipped altogether (compounded by the actual switching rate of the LCD). The second component of the problem is the brightness of the display itself - in order to sufficiently invoke that perceived persistence of stimulus, a certain intensity is required. If intensity is not available contrast may be substituted, but this means it will only ever work in a darkened environment.
Despite the math working against me I decided to give it a go. Setting the interior timer to 120hz is easy enough, but I suspect the time to navigate the DOM logic to the desired cells and switch their states runs slower than that anyway, plus whatever overhead the Jiggy framework required to assist in doing that. I set up the code to follow this plan, draw a column of cells and switch them over the course of fractional sections to the different mapped columns which comprised an entire letter table (in this case a letter is an array of 5 different numbers 0-63 which corresponds to a 6-bit binary display indicating which cells to turn on and off - makes for very tight book-keeping and rapid addressing, and binary logic is just fun anyway so why not?). My test consisted of letters A through E sequentially set up in memory in a per-wave array (all in a single wave) with events tacked to the accelerometer which would catch significant departures from one direction to the next and determine whether that array was being walked forward or backward, even capable of reversing mid-display.
And it almost worked. In the end I had to simplify the display to at most 2 letters, and it only works if 1) the room is dark and 2) one covers the persistent battery-level display at the top of the screen (otherwise it wrecks the PoV effect). Photographing this is especially difficult when one is both the photographer and demonstrator, compounded by doing it in a dark bathroom shot "through the mirror" without being able to effectively test the composition. In the end I got a few good shots and many not-so-good - shown here are 2 images (ISO 80 f/2.0 @ 1/25 sec, the display simplified to just the letter 'A') combined and slightly enhanced for contrast. This shows the difficulty of keeping in sync with the animation - something that with enough tuning of the accelerometer event code and a steadier hand would become easier, but which I don't plan to pursue due to the aforementioned global limitations. The predicted linear fall-off is visible, as are the very slight gaps between vertical blocks on the display.
Overall I'm very pleased with the exercise: it taught me more about a really fun toy, gave me a chance to play with some simple science, and allowed math to Save the Day. Everything worked almost perfectly the first time and I come out richer in knowledge for the effort. The fact that any of this at all pleases me just reinforces the fact that I'm in the right line of work - I'm pretty sure most of the world would find this pretty dull. Hurray for geeks!