While exploring different ways of introducing guests in to a virtual reality world, I have started to like the metaphor of the headset as a blindfold. Bringing guests into a space, thoughtfully, with visual restraint in VR, feels important. First, on a practical level, it can help eliminate some of the potential nausea of a mismatch between the real and the virtual worlds. Second, it forces the experience to focus on other sensory inputs like sound, haptic feedback, and room-scale space. This toolkit of inputs is something I have explored in more depth over the past few months because my current project is a sound-first VR world. In exploring this, I have looked at interactions that have have limited visual components, and more specifically, experiences or games designed for the blind. What are the important mechanics, controls, and types of feedback if you are not leading with visual elements?
The first area I ventured was into the realm of games designed for the blind. While this genre still feels like it is in its infancy, some fair attempts have been made to explore audio-based mechanics. In Nightjar, you are stranded on a spaceship in the dark of a black hole that is threatening to engulf you. You must navigate your way around the ship and escape with sound cues only (and the help of Benedict Cumberbatch). This game, and others like it, have just a few simple keyboard inputs that help you move through the world. This is a good reminder that straightforward, intuitive UI is encouraged in this setting.
Although navigating with binaural audio cues is challenging enough, the game Three Monkeys is interesting because it has found ways of including interactions like hunting and fighting. The tone of the game has shifted: now we are in a standard adventure game. Many games with a “blindness” mechanic also tend towards the horror genre, which speaks more to the connotations of darkness for those of us with sight. Darkness does not need to be a negative; it can be calm, or mysterious, or full of potential. And, sound gives us the pallet with which we can begin to aurally “visualize” those spaces.
In fact, all of the key “visuals” in these games are created through soundscapes in binaural audio. They are often beautifully rich and seem to bring the world to life. That means the sound design should be responsive and as flawless as possible. At times, in Three Monkeys, some of the sound effects become too artistic and not functional enough. One of the birds you are hunting at the very start of the game is difficult to differentiate in timbre from a flute and it often reads as part of the music of the world. When sound creates and defines a virtual space, it must be both precise and discrete. If we hope to use sound as our primary form of feedback we may need to be more pragmatic in its design.
Casual games are excellent for exploring new modes of interaction.Those that I have mentioned demonstrate that sound can be its own interaction when used carefully and clearly. But do these concepts transfer to a VR space where there are a different set of rules? It seems like a strange platform for the blind because we are so used to the VR space being about unreal and vivid visuals.
Nevertheless, a handful of research groups have started exploring this area. In some extreme cases, VR gifts people with sight-impairments a new form of vision. For debilitatingly nearsighted people specifically, VR allows them to see what appears to be “deep space” and distant horizons, within the range of a few inches (Durbin, 2016). Coming at it from a very different direction, De Montfort University is actually looking into how binaural and ambisonic sound can help to sculpt spaces for the blind. Their research group is exploring the use of VR employing interactive spatial audio as a means of helping the blind mentally map unfamiliar rooms (DMU, 2011). This means that, for people with new jobs or situations, they might be able to experience and learn the office or building without needing to be there in person. This suggests that spatialized audio itself can help us to reconstruct real spaces virtually.
Several years ago, a group from my own program (CMU’s Entertainment Technology Center) did a project on interactive stories told through spatialized sound. The team, Sonology, used body tracking along with binaural audio to help people navigate the world of a narrative. Impressively, people were led through the room space with audio cues alone. Story objects were built from sounds effects and the interactions were reinforced with audible feedback. I think the project is an interesting demonstration that sound can successfully craft room scale virtual worlds.
There is still a piece missing here: the hands-on experience. Museums are a group that is focusing on making their primarily visual experiences more accessible and inclusive. The rule in these spaces is usually “don’t touch.” But, the Andy Warhol Museum in Pittsburgh is very interested in devising more accessible exhibits. They have put a lot of time into creating 3D printed and sculptural versions of Warhol’s pieces. One of my favorite tactile experiences was an interactive exhibit based on Andy Warhol’s Silver Clouds. In this piece, pillowy balloons float lazily around the room. Just walking through this space feels unique; it pulls you into a unique and separate world. Most high-quality museums recognize the importance of tactile experiences. They are now hosting tours which allow guests to interact with sculpted replicas of famous works. Artists are using more texture and leaning into materials.
The tactile world is slowly being made available to us in virtual reality as well through haptic feedback. The Vive controllers allow for vibrations of varying strength. For example, there is a mini-game in The Lab (made by Valve) in which you fire arrows down on an attacking army. he sensation of tension when you draw an arrow, and the “thwack” you get when you release the bowstring, are all surprisingly authentic feelings. Other more immersive devices are on the horizon. The Haptx glove, for example, uses an exoskeleton to return feedback as pressure and even supports thermal and textural stimulus using smart textiles. It is easy to imagine that these steps forward in haptics could quickly transform a blind VR experience into one that is experimentally very rich. If you can feel and hear rain falling on you, then you have the key pieces of that experience. The visuals seem relatively less important in this case.
Virtual worlds are not purely visual realms. They are daily becoming more robust, striving to engage our other senses. A blind VR experience that still feels vivid is on its way. But, even with the technology available to me at this moment, I believe we have the capacity to create a world that can be rich, interactive, and not focused heavily on visuals. Games and experiences for the blind remind us that sound has the power to be its own mechanic. And, that when used clearly, sound can paint worlds, sculpt objects and layout UIs. This works well, even more successfully perhaps, when we fit our sounds to room-scale space. And, if we can add some haptic interaction, then we are really beginning to fill our virtual world. The blind experience reminds us how much weight and nuance our senses bring to our perception of the world, real or virtual. Learning to be artists of sound and touch will make our virtual worlds more whole.