Interaction with Affective Computers

Once we begin to explore applications for affective systems, interface design challenges and novel strategies for human-computer interaction immediately begin to suggest themselves. These design challenges concern both hardware and software. In terms of software, the human interface to applications can change with the increased sensitivity that the affective sensing/recognizing/understanding system will bring to the interaction. In terms of hardware, the design challenges present themselves even more immediately.
For example, various bio-sensors and other devices such as pressure sensors may be used as inputs to an affective computing system, perhaps by placing them into mice, keyboards, chairs, jewlery, or clothing, things a user is naturally in physical contact with. Sensing may also be done without contact, via cameras and microphones or other remote sensors. How will these sensors evolve into the user's daily life? Will sensors be embedded in the user's environment, or will they be part of one's personal belongings, perhaps part of a personal wearable computer system? In the latter case, how can we design these systems so that they are unobtrusive to the user and/or invisible to others? In either case, consideration for the user's privacy and other needs must be addressed.

A number of our research projects impact upon issues in interface design; these are listed below.

Research projects related to affective interfaces

Affective Avatars: Virtual Reality avatars that accurately and in real time represent the physical manifestations of affective state of their users in the real world.
AffQuake: AffQuake is an attempt to incorporate signals that relate to a player's affect into ID Software's Quake II in a way that alters game play. Several modifications have been made that cause the player's avatar within Quake to alter its behaviors depending upon one of these signals. In StartleQuake, when a player becomes startled, his or her avatar also becomes startled and jumps back. Quake changes the size of the player's avatar in relation to the user's response as well, representing player excitement by average skin conductivity level, and growing the avatar's size when this level is high.
Affective Carpet: This deformable surface made of conductive cloth and conductive foam detects continuous pressure with excellent sensitivity and resolution, and re-expresses the applied pressure musically.
Affective Jewelry and Accessories: Wearable jewelry and other clothing designs with embedded sensors for sensing physiological changes associated with emotions.
Affective Social Quotient (ASQ): This project aims to help autistic kids learn about social-emotional cues. It consists of short digital videos that embody one of several emotions (currently happy, sad, surprise, and anger) and a set of physical "dolls" linked by infrared to the system. The system knows which dolls correspond to which clips, so that the child can explore emotional situations by picking up dolls with certain emotions, or the system can prompt the child to pick up dolls that go with certain clips.
Affective Tangibles: People naturally express frustration through the use of their motor skills. The purpose of the Affective Tangibles project is to develop physical objects that can be grasped, squeezed, thrown, or otherwise manipulated via a nat ural display of affect. Current tangibles include a squeeze mouse, affective pinwheels that are mapped to skin conductivity, and a voodoo doll that can be shaken to express frustration. All of these physical manipulations are converted to bits via a new i nterface we have designed for facilitating such forms of communication.
Affective Touchables: Physical objects that sense affective parameters through being held or touched, and communicate the emotions abstractly through sight, sound, or haptic changes
Affective Tutor—the Learning Companion: We are interested in constructing an agent that senses affective states like boredom, anxiety, and engagement, and adjusts its response to the user in accord with the user's state. This would be aimed at learning situations where the agent acts as a kind of mentor, occasionally supporting the user in his or her otherwise self-propelled exploration. It also provides an opportunity for learning about the role of human emotions expressed during a learning situation.
Bruzard: A software platform for enabling the computer's expression of emotion via a 3D, expressive animated character.
Computer Response to User Frustration: A human-computer interaction agent was designed and built to support users in their ability to recover from negative emotional states, particularly frustration. The agent uses social-affective feedback strategies, including a kind of "artificial empathy." Its effectiveness was evaluated against two control conditions in a 72-subject study. Behavioral results showed the agent was significantly more effective than the control conditions in helping relieve frustration levels.
Conductor's Jacket: A jacket with an embedded sensor net worn by a conductor helps extend the conductor's ability to express emotion and intentionality.
Detecting Driver Stress: We outfitted a car with cameras and physiological sensors and used real-time signals to recognize driver stress as represented by road conditions, questionnaire responses, and third-party observations.
Expression Glasses: A wearable device which allows any viewer to visualize the confusion and interest levels of the wearer.
The Galvactivator: A wearable device which maps your skin conductivity to a glowing red LED. Set the baseline, and then use this as a device to learn about and communicate your body's response.
Frustration Detection: Results of using HMM's to try to detect episodes of likely frustration in a human-computer interaction, using physiological signals of skin conductivity and blood-volume pressure.
Frustration Experiment Design: A psychological experiment designed to elicit a "frustration response", and to synchronize computer events with multiple channels of physiological data Used to collect data for the Frustration Detection study, above.
Interface Tailor : The Interface Tailor is an agent that attempts to adapt the system in response to affective feedback. Frustration is being used as a fitness function to select between a wide variety of different system behaviors. Currently, the Microsoft Office Assistant (or Paperclip) is one example interface that is being made more adaptive. Ultimately the project seeks to provide a generalized framework for making all software more tailor-able.
Learning and pattern recognition: Computers can potentially learn patterns of behavior (physiological and otherwise) that depend on the user (personality, goals, preferences, etc.) and on his or her situation (work, commute, too much caffeine, etc.). We conduct basic research in Bayesian learning theory, pattern recognition, and other aspects of modeling that form the crucial underpinnings of our efforts to build robust, flexible, and adaptive systems.
Mood Interfaces: This project explores graphical interfaces in which physiological signals drive the visual display, allowing users or their conversational partners to engage in a computer-mediated dialogue of emotional expression, viewing a graphical representation of expressions of their current physical/emotional state.
Orpheus, the affective CD player: A digital music delivery system that plays music based on your current mood, and your listening preferences -- whether, for example, you want to hear sad music when you are sad, or music to change your mood.
PalmPilot Interface for Affective Wearables: The PalmPilot provides an easy-to-use diagnostic and annotation interface for affective wearable computers.
Sentic Mouse: A modified computer mouse that includes a directional pressure sensor for aiding in recognition of emotional valence (liking/attraction vs. disliking/avoidance).
SmartSHELL: If a system is smart enough to know when a person is going to be at a "burned out" performance level, can it help motivate the person?
TurnStyles: A personalizing conversational speech interface agent designed for affective communication, which adjusts speaking style to the user.
Robotic Computer: A personal computer with an active, robotic, monitor, which expresses and responds to social-emotional cues.

Want to email us? For answers to frequently asked questions and for our email address, please visit the FAQ and email page. Thanks!