We’re living in a day and age where technologically, the sky seems to be the limit. Yet, our communication with computers remains quite crude. Machines have to infer our intentions from what we explicitly tell them, via keyboards, controllers, or voice-commands. To our machines, that’s all we are: mere semantics, an entity that provides them with a stream of cold instructions. That’s vastly different from how we obtain information from other humans. We rely on much more than semantics. We are looking for subtle cues in somebody’s face, voice, and body language. In other words, a substantial amount of the information we obtain from others is emotional. What if our machines could do the same: sample information about their user’s emotions, beyond what they gather from keystrokes? The development of systems that can recognize, interpret, process human emotion has been summarized as affective computing.
Using affective computing, Abele and I will develop a virtual training environment for the Dutch police, that will assist the officers in regulating their responses to threatening situations with the help of biofeedback (for more details have a look at our project page). We feed the player's heart rate, brain activity, and breathing rhythm to an algorithm that is trained to use these signals to infer the player's emotional state (Nastasia’s wrote about the link between physiology and emotions in her latest blog post). The algorithm can detect subtle signs of distress that the player might not even be aware of. As soon as a player shows signs of distress, the game alerts them by changes in the environment, say, by dimming the ambient lighting. This kind of environmental adaptation is nicely illustrated in Elke's biofeedback game MindLight.
With biofeedback we establish a more holistic communication with our machines: we extend our input to the game beyond semantics, beyond telling the game where we want to go, or which item we’d like to pick up next. Now, additionally, games adapt to how we feel. Equipped with this additional channel of information, games can teach us how we respond to challenges and danger. They can help us listening to our bodies.
https://www.shutterstock.com/nl/g/vladystock (header image)