In the last article, Pokémon Go And The Future In Front Of Your Face, I painted a picture of augmented reality that sounds like science fiction. Smart contact lenses, information floating in front of your eyes, walking on a virtual reproduction of the surface of Mars – admit it, you know things move quickly but you mentally put it in the category of things that won’t affect you, right?

The best technology doesn’t call attention to itself. It enhances our world in ways that feel natural. It turns out that there is an example of augmented reality that you’ve accepted routinely for almost twenty years. It’s an interesting story.

In 1998, the NFL broadcast the first football game that showed a yellow line on the field marking the first down line. It’s been used routinely since then. You don’t give it a second thought.

It was hard to get excited about key plays watching televised football in the 90s because there was no good visual feedback about whether a play had been successful. Occasionally the sideline marker was visible but frequently we had to wait for refs or announcers to tell us whether a team had gotten its first down.

When Sportvision inserted the yellow line into the first broadcast, it was immediately apparent that it solved a problem in an elegant way that now seems natural and necessary. We are so used to the yellow line that it is disconcerting to go to live games where there’s no line marker. A Reddit comment about a childhood memory: “I was so confused my first time going to a Bears game. There were no announcers and there was no first down line or anything. I had never seen a game without those things so I just assumed they would be there.”

The yellow line is an example of augmented reality. It is digital information inserted on top of a view of the real world. As it happens, it’s on television instead of on a phone or some high-tech goggles, but that yellow line is one of the first widespread applications of augmented reality. It’s been in front of you for eighteen years.

The yellow line gets better when you look behind the scenes. That is one fabulous yellow line.

See how the yellow line and the blue line in the photo above look like they’re painted on the ground? The line doesn’t move as the cameras pan and zoom. The players are running on top of the line. Now that I mention it, you’re thinking, hmm, interesting. I wonder how they do that?

Glad you asked.

Each year, crews do detailed laser surveys of each stadium to create a very precise 3D model of the field. The laser surveys even catch the slight contour of the field that helps rain water run off.

During the game, each camera is constantly sending back data about its pan, tilt and zoom positions for every frame. Computers place the line on the field, adjust its perspective, and hold it steady by analyzing each frame on the fly and adjusting it so the line appears to be rock solid.

Meanwhile, all of the colors on the field get sampled before the game – every distinct shade of green or brown in the grass given the lighting conditions. Then information is fed into the computers about all the colors for the players and the refs – uniform colors, helmets, skin tones, shoe colors, football colors, flags. The field is treated as a giant green screen but far more difficult to implement – most green screens are required to be very uniform in color. The Sportvision computers put the line on the field and remove it whenever any part of a player covers it. Again, each frame is analyzed on the fly.

All of that computing is done before the live feed is sent to your television. It delays the live feed by less than one second. You’re not watching the live game, you’re watching a data stream that has been altered by real-time computer algorithms.

When Sportvision implemented the first yellow line, it required six people and a 48-foot semi filled with servers and video equipment to make it work. Now much of it is automated and much has been added, including the blue line of scrimmage, arrows, and descriptive information about the situation (“3rd & 5”). And an interesting thing to know – not all of the ads on the billboards and screens inside sports fields are visible to the people in the stadiums. Some of those are inserted on the fly using the same technology.

Vox has a nice short video here explaining more of the technology and the background (an early experiment in hockey games using information from sensors embedded in the puck).

Just this year, the NBA began using similar augmented reality technology to highlight the three-point arc on the fly during pro basketball play.

The America’s Cup sailing competition embraced augmented reality a few years ago because it had traditionally been a difficult sport to follow, even for passionate fans.

Augmented reality has been used to enhance broadcasts from the last few Olympics, like the banner floating in the water in the above picture of Olympics swimming in 2008. Now that you’re alert for it, you can expect to see augmented reality all over your screens as you watch the upcoming Olympics. Augmented reality technology will be used in other ways related to the Olympics as AR begins to be adopted more widely. The US Olympic Cycling Team is training with augmented reality glasses that can display speed, heart rate, distance traveled, and many other things. (You’ll be able to buy a pair for yourself later this year.) Olympics coverage will also feature more than 100 hours of virtual reality coverage for anyone with a Samsung smartphone and Samsung Gear VR headset.

The science fiction future is all around us, hidden in plain sight. Keep your eyes open as digital information begins to overlay your view of the real world – useful, elegant, seductive, and fun.