We’ve been talking about new devices for AR and VR as if they’re just engineering challenges. All the companies have to do is invent some cool batteries and lenses and poof, there’s a revolution.
There’s another problem and it’s just as difficult as the high tech stuff. Let’s call it the input problem. How do you give commands to the system? How do you write things down?
It seems trivial, like you should just be able to, I dunno, wave your hands or something? Most discussions of AR and VR ignore the input problem.
I don’t have any answers. I just want you to be aware that this is an interesting question.
When you want to do something on your phone, you take out your phone and hold it in your hand. You tap on an icon to launch an app. If you have to type words, you use your fingers on a little keyboard on the screen.
At your computer, you move a cursor with a mouse or touchpad and use a keyboard for words.
When you’re wearing a virtual reality helmet, you can’t see your hands. You can’t use a keyboard. Have you ever tried to type with your eyes shut? It doesn’t work. And you don’t know where the mouse is. I know. I have a lot of experience knocking things over while I’m flailing around trying to locate the mouse with VR goggles on.
When augmented reality glasses reach the goal of being self-contained, you’ll see a digital overlay on the world with nothing in your hands. How do you switch from one effect to another? How do you start a conversation with someone? How do you type?
Above is a typical promotional picture of a business meeting in virtual reality. Do you see what’s missing?
Right – there are no computers. The picture shows nothing more than a chat session. It’s a Zoom meeting but no one in the meeting can see their hands or look at anything else on their laptop. That’s not how business meetings work.
Here’s another stock photo showing a woman in a VR meeting with her laptop and phone in front of her. She would have to tip up the goggles so she can see her keyboard. That is a non-starter in business.
Other VR promotional pictures show people in a meeting with virtual keyboards in front of them. Typing on a virtual keyboard when your fingers only feel a flat desk is almost impossible.
There’s something else missing in that photo. VR goggles use handheld controllers, one in each hand, to position your hands and give you buttons to click on things. To use a keyboard/mouse in VR, you’d have to lay the controllers to one side. To other people in the meeting, it would look like you set your hands to the side and your arms would go slack. That’s weird.
You can’t keep the controllers in your hand and do anything with text by pointing at a virtual keyboard. You already know what that experience is like. It’s like entering a login name and password on a smart TV. It’s awful.
Think about augmented reality. Imagine that you’re wearing superpowered glasses that can display arrows pointing to your destination, and pop up a window with the hours that a business is open.
How do you put in the address you want to visit? Speak out loud? Sure, maybe, but there are limits to how much you want to talk in public to something only you can see, and voice commands are notoriously imprecise and misinterpreted.
Gesture with your hands – literal hand-waving? Sure, maybe, but it will take a long time for that to become intuitive and socially acceptable. Plus, that doesn’t work for typing. We need some kind of feedback from our hands to type well, either from pressing keys or feeling our finger tap a phone screen. Gestures can be used to click things that have already been created but they’re terrible for entering new information. The picture at the top of the businessman typing in space – nah, that doesn’t work.
Oh, and another problem. We can’t keep our hands up too long. There’s a scene in the movie Minority Report where Tom Cruise stands at a transparent computer screen and slides things around with his hands. Cool! But in the real world, our hands get tired if we have to hold them up for more than a couple of minutes. That’s why we have handrests on keyboards and rest our hands on our lap during extended phone sessions.
Possible solutions to the input problem
Lots of people are working on possible solutions to the input problem. No one knows the answer. Here are a few possibilities in addition to voice and gestures.
Your brain Headline from an article in The Verge five years ago: “Facebook is working on a way to let you type with your brain.” The research labs at Meta are trying to set up a brain/computer interface, even if it’s just a Yes/No mouse click that you can do by thinking about it. Fortunately they’re aware that no one wants implanted brain electrodes. Instead, a team of 60 scientists and engineers from UC Berkeley, UC SF, Johns Hopkins, and other labs are trying to come up with a cap that can pick up neural activity. Sure, this is science fiction, but isn’t all of this just SF coming true?
Wristbands Mark Zuckerberg was confident enough about “neural interface EMG wristbands” to show them off to a potential fashion eyewear partner a few months ago. They can track the nerve signals from your fingers so you can type on a virtual keyboard. Yeah, I know, more science fiction, right? There are several variations in the labs already. There’s still more science to be done but they’ve moved forward to questions about marketing and privacy issues that would come up if Facebook had access to lots of neural data. “That might include incredibly fine variations in typing patterns; overall levels of bodily tension; and any biometric information captured by fitness tracking sensors, augmented reality glasses, and other tech that could be integrated with the bands. (Facebook Reality Labs notes that it has a “neuroethics program” examining the privacy, security, and safety implications of AR and neural interface tech.)” Here’s more information about Meta wristbands.
Keyboard emulation A few years ago Logitech showed off a fist sized device hanging off one of its keyboards. The “Logitech Bridge” aligns the keyboard in virtual reality so you can position your hands correctly on the real keyboard with a helmet on. Apparently it was quietly shelved and never came to market but it gives you an idea of the range of solutions out there.
Eye tracking and blinking VR goggles will include inward-facing cameras that can track eye movements; AR glasses might watch every twitch of your iris. Several companies are working on eye tracking that judges your intention by where you look. If you look in the upper right, say, it brings up a command menu; staring at a menu option and blinking would be the equivalent of a mouse click. Tobii, for example, says its technology “leverages eye movement to enable object control and intuitive aim. For usability and ergonomics, eye tracking delivers attentive UX, a capability that makes 3D interfaces sensitive to user attention.”
Virtual reality keyboard passthrough VR headsets have cameras that can pass the view from outside into what you see in the VR world. Meta is doing something very clever in Horizon Workrooms, its virtual reality business environment. If you look closely at the above picture, you’ll see a barely visible outline of a translucent laptop in front of each person in the meeting. Each person can see their own laptop screen using a remote desktop program, while everyone else just sees the translucent ghost.
The clever bit is the rectangle on the desk in front of each person, with video passed through from the real world. You can see your hands on your own keyboard and mouse without interfering with your view of the rest of the room.
That doesn’t answer the other questions. Using a keyboard and mouse means putting down the controllers that are used for everything else in the VR experience, which means switching in an unintuitive way between typing and other tasks. A clever keyboard/mouse hack doesn’t address anything about AR in the real world outside the office. But one puzzle at a time. I’ll be curious to see how this works in practice with Meta’s latest VR equipment.
