Highlighted Projects:

Affect as Index
Affect as Index is a tool that takes group physiological data as input, aggregates it across different demographic dimensions, and attaches them to media content. Users can review videotaped or prerecorded events by clicking on points of interest in a physiological graph. This software addresses two challenges: the difficulty of expressing and sharing emotions with others, and the laborious task of monitoring interpersonal interactions within natural settings. For the former, groups interested in discussing shared and dissimilar emotions evoked during experiences can use this tool to place context around their dialogue. For the latter, "meaningful moments" observed within natural interactions can be marked and superimposed on the physiological data collected. In this way, affect and observations of affect can be used to index group-level significant moments that occur within volumes of video data.

Affective-Cognitive Framework for Machine Learning and Decision Making
Recent findings in affective neuroscience and psychology indicate that human affect and emotional experience play a significant and useful role in human learning and decision-making. Most machine-learning and decision-making models, however, are based on old, purely cognitive models, and are slow, brittle, and awkward to adapt. We aim to redress many of these classic problems by developing new models that integrate affect with cognition. Ultimately, such improvements will allow machines to make smarter and more human-like decisions for better human-machine interaction.

Affective-Cognitive Product Evaluation and Prediction of Customer Decisions
Companies would like more new products to be successful in the marketplace, but current evaluation methods such as focus groups do not accurately predict customer decisions. We are developing new technology-assisted methods to try to improve the customer-evaluation process and better predict customer decisions. The new methods involve multi-modal affective measures (such as facial expression and skin conductance) together with behavioral measures, anticipatory-motivational measures, and self-report cognitive measures. These measures are combined into a novel computational model, the form of which is motivated by findings in affective neuroscience and human behavior. The model is being trained and tested with customer product evaluations and marketplace outcomes from real product launches.

Auditory Desensitization Games
Persons on the autism spectrum often report hypersensitivity to sound. Efforts have been made to manage this condition, but there is wide room for improvement. One approach - exposure therapy – has promise, and a recent study showed that it helped several individuals diagnosed with autism overcome their sound sensitivities. In this project, we borrow principles from exposure therapy, and use fun, engaging, games to help individuals gradually get used to sounds that they might ordinarily find frightening or painful.

Customer Measurement Using Bluetooth
We are exploring innovative use of cell-phone Bluetooth technologies for consumer research and customer measurement. We have developed a small, portable, Bluetooth base station that can monitor consumer activity in a retail space and also enable new interactive services. This Bluetooth hub also serves as a network gateway for other wireless sensors in the local area.

Emotion Communication in Autism
People who have difficulty communicating verbally (such as many people with autism) sometimes send nonverbal messages that do not match what is happening inside them. For example, a child might look calm and receptive to learning, while having a heart rate of over 120 bpm and being on the verge of a meltdown or shutdown. This mismatch can lead to serious problems, including misunderstandings such as "he became aggressive for no reason." We are creating new technologies to address this fundamental communication problem and enable the first long-term, ultra-dense longitudinal data analysis of emotion-related physiological signals. We hope to equip individuals with personalized tools to understand the influences of their physiological state on their own behavior (e.g., "which state helps me best maintain my attention and focus for learning?"). Data from daily life will also advance basic scientific understanding of the role of autonomic nervous system regulation in autism.

Emotional-Social Intelligence Toolkit
Social-emotional communication difficulties lie at the core of autism spectrum disorders, making interpersonal interactions overwhelming, frustrating, and stressful. We are developing the world's first wearable affective technologies to help the growing number of individuals diagnosed with autism—approximately 1 in 150 children in the United States—learn about nonverbal communication in a natural, social context. We are also developing technologies that build on the nonverbal communication that individuals are already using to express themselves, to help families, educators, and other persons who deal with autism spectrum disorders to better understand these alternative means of nonverbal communication.

Evaluation Tool for Recognition of Social-Emotional Expressions from Facial-Head Movements
To help people improve their reading of faces during natural conversations, we developed a video tool to evaluate this skill. We collected over 100 videos of conversations between pairs of both autistic and neurotypical people, each wearing a Self-Cam. The videos were manually segmented into chunks of 7-20 seconds according to expressive content, labeled, and sorted by difficulty—all tasks we plan to automate using technologies under development. Next, we built a rating interface including videos of self, peers, familiar adults, strangers, and unknown actors, allowing for performance comparisons across conditions of familiarity and expression. We obtained reliable identification (by coders) of categories of smiling, happy, interested, thinking, and unsure in the segmented videos. The tool was finally used to assess recognition of these five categories for eight neurotypical and five autistic people. Results show some autistics approaching the abilities of neurotypicals while several score just above random.

Externalization Toolkit
We propose a set of customizable, easy-to-understand, and low-cost physiological toolkits in order to enable people to visualize and utilize autonomic arousal information. In particular, we aim for the toolkits to be usable in one of the most challenging usability conditions: helping individuals diagnosed with autism. This toolkit includes: wearable, wireless, heart-rate and skin-conductance sensors; pendant-like and hand-held physiological indicators hidden or embedded into certain toys or tools; and a customized software interface that allows caregivers and parents to establish a general understanding of an individual's arousal profile from daily life and to set up physiological alarms for events of interest. We are evaluating the ability of this externalization toolkit to help individuals on the autism spectrum to better communicate their internal states to trusted teachers and family members.

FaceSense: Affective-Cognitive State Inference from Facial Video
People express and communicate their mental states—such as emotions, thoughts, and desires—through facial expressions, vocal nuances, gestures, and other non-verbal channels. We have developed a computational model that enables real-time analysis, tagging, and inference of cognitive-affective mental states from facial video. This framework combines bottom-up, vision-based processing of the face (e.g., a head nod or smile) with top-down predictions of mental-state models (e.g., interest and confusion) to interpret the meaning underlying head and facial signals over time. Our system tags facial expressions, head gestures, and affective-cognitive states at multiple spatial and temporal granularities in real time and offline, in both natural human-human and human-computer interaction contexts. The system is being made available on multiple platforms, including portable devices. Applications range from measuring people's experiences to a training tool for autism spectrum disorders.

Frame It
"Frame It" is an interactive tangible-digital puzzle game intended as a play-centered teaching and therapeutic tool. Current work is focused on the development of a social-signals puzzle game for children with Autism that will help them recognize social-emotional cues from information surrounding the eyes. In addition, we are investigating if this play-centered therapy results in the children become less averse to direct eye contact with others.

Gestural Control of Guitar Audio Effects
Emotions are often conveyed through gesture. Instruments that respond to gestures offer musicians new, exciting modes of musical expression. This project gives musicians wireless, gestural-based control over guitar effects parameters. For example, with this system, a guitarist can manipulate any style of pitch bending (from subtle vibrato, to whole step bends, to two-octave dive bombs) just by moving the headstock of the guitar.

Girls Involved in Real-Life Sharing
In this research, a proactive emotional health system, geared toward supporting emotional self-awareness and empathy, was built as a part of a long-term research plan for understanding the role digital technology can play in helping people to reflect on their beliefs, attitudes, and values. The system, G.I.R.L.S. (Girls Involved in Real-Life Sharing), allows users to reflect actively upon the emotions related to their situations through the construction of pictorial narratives. The system employs common-sense reasoning to infer affective content from the users' stories and support emotional reflection. Users of this new system were able to gain new knowledge and understanding about themselves and others through the exploration of authentic and personal experiences. Currently, the project is being turned into an online system for use by school counselors.

Heartphones
We are developing wearable sensors that measure cardiovascular parameters such as heart rate and heart rate variability (HRV) in real time. HRV provides a sensitive index of autonomic nervous system activity. These sensors will be capable of communication with mobile devices such as the iPhone and iPod Touch.

iCalm (TM): Comfortable, Wearable, Wireless Bio-Sensing
We are developing a tiny wearable wireless sensor platform that allows comfortable, long-term sensing of physiological information coupled with low-cost connectivity to consumer devices including mobile phones and the XO laptop. This platform has many applications, including health monitoring for outpatients or the elderly, communication of affective information for people who are non-speaking or otherwise interested in sharing this information, education for individuals who want to learn about their own internal physiological changes during daily life, and customer experience data gathering in mobile situations.

Infant Monitoring and Communication
We have been developing comfortable, safe, attractive physiological sensors that infants can wear around the clock to wirelessly communicate their internal physiological state changes. The sensors capture sympathetic nervous system arousal, temperature, physical activity, and other physiological indications that can be processed to signal changes in sleep, arousal, discomfort or distress, all of which are important for helping parents better understand the internal state of their child and what things stress or soothe their baby. The technology can also be used to collect physiological and circadian patterns of data in infants at risk for developmental disabilities.

Machine Learning and Pattern Recognition with Multiple Modalities
This project develops new theory and algorithms to enable computers to make rapid and accurate inferences from multiple modes of data, such as determining a person's affective state from multiple sensors—video, mouse behavior, chair pressure patterns, typed selections, or physiology. Recent efforts focus on understanding the level of a person's attention, useful for things such as determining when to interrupt. Our approach is Bayesian: formulating probabilistic models on the basis of domain knowledge and training data, and then performing inference according to the rules of probability theory. This type of sensor fusion work is especially challenging due to problems of sensor channel drop-out, different kinds of noise in different channels, dependence between channels, scarce and sometimes inaccurate labels, and patterns to detect that are inherently time-varying. We have constructed a variety of new algorithms for solving these problems and demonstrated their performance gains over other state-of-the-art methods.

Mechatronics and Prompt-Assisted Typing Aids
People on the autism spectrum face a number of challenges, including motor movement issues that can cause limbs to cease activity. Circumstantial evidence suggests that autonomic nervous system influences related to stress and overload may arise from and contribute to these problems. We propose allowing individuals to monitor several physiological parameters to see if there are patterns that recognize or predict the onset of their individual motor problems. We plan to develop new, wearable technology to treat these problems via the use of tiny, vibrotactile devices carefully placed at the joints. We hypothesize that some methods of touch-feedback and vibration at the joints may enable individuals to recover motor functioning during episodes of intermittent loss. We are also exploring the development of personally controlled devices that facilitate finer motor movement for augmenting communication as needed for assisting in typing or pointing.

Objective Self: Understanding Internal Responses
How can technology help us understand ourselves better? Measuring the physiological arousal of children with sensory challenges such as ASD and ADHD, tools were developed to help children understand and control what makes them overexcited. Using iCalm hardware, children in therapy sessions measured their arousal while eating, throwing tantrums, playing in ball pits, and making challenging choices. Beyond progressive findings in the field of occupational therapy, this research is a basis for bio-information technology: tools to help children, their parents, and their teachers better understand what is going on in their bodies in a comfortable, affordable, and adaptable way. With future work, technology will be developed to help children understand and control their own internal state. In addition, this project will go beyond children’s therapy—helping adults in various settings including business and home life.

Passive Wireless Heart-Rate Sensor
We have developed a low-cost device that can wirelessly detect a beating heart over a short distance (1m) and does not require any sensor placed on the person's body. This device can be used for wireless medical/health applications as well as security and safety applications, such as automobile/truck drivers as well as ATM machines. We have also created a small battery-powered version of this sensor that can be worn on a person's clothing but does not require touching the person's skin.

Prediction Game and Experience Sharing Market for Forecasting Marketplace Success
We have developed a novel market game, Prediction Game and Experience Sharing (PreGES, pronounced PreGuess), that harnesses people's collective prediction and experience sharing to forecast success or failure of new items (e.g., products, services, UI designs). Companies can register their new items on this market (as a testbed) to ask for collective opinions. In each PreGES trial session, participants makes their own best predictions on other people's overall opinions about the new items to get incentives (e.g., real opportunities to experience the items) and have fun in gambling-like games. As a participant’s guess (or portfolio) approaches the collective guess of all participants, he or she has a greater chance of winning an incentive. Participants improve the accuracy of their next prediction by sharing experiences. As participants have more trial sessions, their collective prediction converges into one common opinion (forecasting the success or failure of new items).

RoCo: A Robotic Desktop Computer
A robotic computer that moves its monitor "head" and "neck," but that has no explicit face, is being designed to interact with users in a natural way for applications such as learning, rapport-building, interactive teaching, and posture improvement. In all these applications, the robot will need to move in subtle ways that express its state and promote appropriate movements in the user, but that don't distract or annoy. Toward this goal, we are giving the system the ability to recognize states of the user and also to have subtle expressions.

Sensor-Enabled Measurement of Stereotypy and Arousal in Individuals with Autism
A small number of studies support the notion of a functional relationship between movement stereotypy and arousal in individuals with ASD, such that changes in autonomic activity either precede or are a consequence of engaging in stereotypical motor movements. Unfortunately, it is difficult to generalize these findings as previous studies fail to report reliability statistics that demonstrate accurate identification of movement stereotypy start and end times, and use autonomic monitors that are obtrusive and thus only suitable for short-term measurement in laboratory settings. The current investigation further explores the relationship between movement stereotypy and autonomic activity in persons with autism by combining state-of-the-art ambulatory heart rate monitors to objectively assess arousal across settings; and wireless, wearable motion sensors and pattern recognition software that can automatically and reliably detect stereotypical motor movements in individuals with autism in real time.

SmileSeeker: Customer and Employee Affect Tagging System
SmileSeeker is a novel, machine-vision system that captures and provides quantified information about nonverbal communication where social interactions naturally happen. For example, in banking services, tellers observe facial expressions, head gestures, and eye gaze of customers, but this tool lets them both observe their own expressions and analyze how these interact with those of the customer to influence their mutual experience. The tool allows either real-time or offline feedback to help people reflect on what these interactions mean and determine how to elicit better experiences, such as true customer delight. The first deployment of this project focuses on eliciting and capturing smiles, and doing so in a way that is respectful of both customer and employee feelings. This project will also explore ways to share this information and link it to outcomes such as banking fee reductions or donations to charity.

Prior Projects:

AboutFace
AboutFace is a user-dependent system that is able to learn patterns and discriminate the different facial movements characterizing confusion and interest. The system uses a piezoelectric sensor to detect eyebrow movements and begins with a training session to calibrate the unique values for each user. After the training session, the system uses these levels to develop an expression profile for the individual user. The system has many potential uses, ranging from computer and video-mediated conversations to interactions with computer agents. This system is an alternative to using camera-based computer vision analysis to detect faces and expressions. Additionally, when communicating with other people, users of this system also have the option of conveying their expressions anonymously by wearing a pair of glasses that conceals their expressions and the sensing device.

Adaptive, Wireless, Signal Detection and Decoding
In this project, we propose a new Bayesian receiver for signal detection in flat-fading channels. First, the detection problem is formulated as an inference problem in a hybrid dynamic system that has both continuous and discrete variables. Then, an expectation propagation algorithm is proposed to address the inference problem. As an extension of belief propagation, expectation propagation efficiently approximates a Bayesian estimation by iteratively propagating information between different nodes in the dynamic system and projecting exact messages into the exponential family. Compared to sequential Monte Carlo filters and smoothers, the new method has much lower computational complexity since it makes analytically deterministic approximations instead of Monte Carlo approximations. Our simulations demonstrate that the new receiver achieves accurate detection without the aid of any training symbols or decision feedbacks. Future work involves joint decoding and channel estimation, where convolutional codes are used to protect signals from noise corruption. Initial results are promising.

Affect in Speech: Assembling a Database
The aim of this project is to build a database of natural speech showing a range of affective variability. It is an extension of our ongoing research focused on building models for automatic detection of affect in speech. At a very basic level, training such systems requires a large corpus of speech containing a range of emotional vocal variation. A traditional approach to this research has been to assemble databases where actors have provided the affective variation on demand. However, this method often results in unnatural sounding speech and/or exaggerated expressions. We have developed a prototype of an interactive system that guides a user through a question and answer session. Without any rehearsals or scripts, the user navigates through touch and spoken language an interface guided by embodied conversational agents which prompt the user to speak about an emotional experience. Some of the issues we are addressing include the design of the text and character behavior (including speech and gesture) so as to obtain a convincing and disclosing interaction with the user.

Affective Carpet
The "Affective Carpet" is a soft, deformable surface made of cloth and foam, which detects continuous pressure with excellent sensitivity and resolution. It is being used as an interface for projects in affective expression, including as a controller to measure a musical performer's direction and intensity in leaning and weight-shifting patterns.

Affective Learning Companion
Affective Learning Companion is a powerful, flexible new research tool for exploring a variety of social-emotional skills in human-machine interaction, and for understanding how machines can work with people to better meet their needs. The platform enables a computational agent to sense and respond, in real time, to a user's non-verbal emotional cues, using video, postural movements, mouse pressure, physiology, and other behaviors communicated by the user to infer, for example, if a user is in a high or low state of interest, or feeling frustrated. We recently developed an animated agent that combines non-verbal mirroring (or not) with multiple kinds of affective and cognitive support during a frustrating learning episode. The system allows us to control factors that have previously been impossible to control, enabling for the first time the study of how these factors interact in helping learners develop the ability to persevere during frustrating learning episodes.

Affective Mirror
The Affective Mirror is an attempt to build a fully automated system that intelligently responds to a person's affective state in real time. Current work is focused on building an agent that realistically mirrors a person's facial expression and posture. The agent detects affective cues through a facial-feature tracker and a posture-recognition system developed in the Affective Computing group; based on what affect a person is displaying, such as interest, boredom, frustration, or confusion, the system responds with matching facial affect and/or posture. This project is designed to be integrated into the Learning Companion Project, as part of an early phase of showing rapport-building behaviors between the computer agent and the human learner.

Affective Social Quest
ASQ investigates ways to teach social-emotion skills to children interactively with toys. One of the first goals is to help autistic children recognize expressions of emotion in social situations. The system uses four "dwarfs" expressing sad, happy, surprise, and angry, and each communicates wirelessly to the system and detects which plush doll was selected by the child. The computer plays short entertaining video clips displaying examples of the four emotions and cues the child to pick a dwarf that closely matches that emotion. Future work includes improving the ability of the system to recognize direct displays of emotion by the child.

Affective Tangibles
People naturally express frustration through the use of their motor skills. The purpose of the Affective Tangibles project is to develop physical objects that can be grasped, squeezed, thrown, or otherwise manipulated via a natural display of affect. Constructed tangibles include a PressureMouse, affective pinwheels that are mapped to skin conductance, and a voodoo doll that can be shaken to express frustration. We have found that people often increase the intensity of muscle movements when experiencing frustrating interactions.

Affective Tigger
The Affective Tigger is a plush toy designed to recognize and react to certain emotinal behaviors of its playmate. For example the toy enters a state of "happy," moving its ears upward and emitting a happy vocalization when it recognizes that the child has postured the toy upright and is bouncing it along the floor. Tigger has five such states, involving recognizing and responding with an emotional behavior. The resulting behavior Tigger demonstrates allows it to serve as an affective mirror for the child's expression. This work involved designing the toy, and evaluating sessions of play with it with dozens of kids. The toy was shown to successfully communicate some aspects of emotion, and to prompt behaviors that are interesting to researchers trying to learn about the development of human emotional skills such as empathy.

AffQuake
AffQuake is an attempt to incorporate signals that relate to a player's affect into ID Software's Quake II in a way that alters game play. Several modifications have been made that cause the player's avatar within Quake to alter its behaviors depending upon one of these signals. In StartleQuake, when a player becomes startled, his or her avatar also becomes startled and jumps back. Quake changes the size of the player's avatar in relation to the user's response as well, representing player excitement by average skin conductance level, and growing the avatar's size when this level is high. A taller avatar means the player can see further; however, it also makes him or her an easier target.

Ambient Displays for Social Support and Diabetes Management
We design and evaluate an ambient blood glucose level visualization and feedback system that uses an Ambient Orb for diabetes self-care and social support. The social support is provided by a friend or family member of an individual with diabetes. This research study was carried out with adult patients at Joslin Diabetes Center.

Automatic Facial Expression Analysis
Recognizing non-verbal cues, which constitute a large percentage of our communication, is a prime facet of building emotionally intelligent systems. Facial expressions and movements such as a smile or a nod are used either to fulfill a semantic function, to communicate emotions, or as conversational cues. We are developing an automatic tool using computer vision and various machine-learning techniques, which can detect the different facial movements and head gestures of people while they are interacting naturally with the computer. Past work on this project determined techniques to track upper facial features (eyes and eyebrows) and detect facial actions corresponding to those features (eyes squintint or widening, eyebrows raised). The ongoing project is expanding its scope to track and detect facial actions corresponding to the lower features. Further, we hope to integrate the facial expression analysis module with other sensors developed by the Affective Computing group to reliably detect and recognize different emotions.

Bayesian Spectral Estimation
This project developed efficient versions of Bayesian techniques for a variety of inference problems, including curve fitting, mixture-density estimation, principal-components analysis (PCA), automatic relevance determination, and spectral analysis. One of the surprising methods that resulted is a new Bayesian spectral analysis tool for nonstationary and unevenly sampled signals, such as electrocardiogram (EKG) signals, where there is a sample with each new (irregularly spaced) R wave. The new method outperforms other methods such as Burg, Music, and Welch, and compares favorably to the multitaper method without requiring any windowing. The ability to use unevenly spaced data helps avoid problems with aliasing. The method runs in real time on either evenly or unevenly sampled data.

BioMod
BioMod is an integrated interface for users of mobile and wearable devices, monitoring various physiological signals such as the electrocardiogram, with the intention of providing useful and comfortable feedback about medically important information. The first version of this system includes new software for monitoring stress and its impact on heart functioning, and the ability to wirelessly communicate this information over a Motorola cell phone. One application under development is the monitoring of stress in patients who desire to stop smoking: the system will alert an "on-call" trained behavior-change assistant when the smoker is exhibiting physiological patterns indicative of stress or likely relapse, offering an opportunity for encouraging intervention at a point of weakness. Challenges in this project include the development of an interface that is easy and efficient to use on the go, is sensitive to user feelings about the nature of the information being communicated, and accurately recognizes the patterns of physiological signals related to the conditions of interest.

Car Phone Stress
We are building a system that can watch for certain signs of stress in drivers, specifically stress related to talking on the car phone, as may be caused by increased mental workload. To gather data for training and testing our system, subjects were asked to 'drive' in a simulator past several curves while keeping their speed close to a predetermined desired constant value. In some cases they were simultaneously asked to listen to random numbers from a speech-synthesis software and to perform simple mathematical tasks over a telephone headset. Several measures drawn from the subjects' driving behavior were examined as possible indicators of the subjects' performance and of their mental workload. When subjects were instructed (by a visible sign) to brake, most braked within 0.7-1.4 seconds after the sign came into view. However, in a significant number of incidents, subjects never braked or braked 1.5-3.5 seconds after the message; almost all of these incidents were when subjects were on the phone. On average, we found that drivers on the phone braked 10% slower than when not on the phone; additionally, the variance in their braking time was four times higher -- suggesting that although delayed driver reactions were infrequent, when delays happened they could be large and potentially dangerous. Furthermore, their infrequency could create a false sense of security. In future experiments, subjects' physiological data will be analyzed jointly with measures of workload, stress and performance.

Cardiac PAF Detection and Prediction
PAF (paroxysmal atrial fibrillation) is a dangerous form of cardiac arrhythmia that poses severe health risks, sometimes leading to heart attacks, the recognized number-one killer in the developed world. The technical challenges for detecting and predicting PAF include accurate sensing, speedy analysis, and a workable classification system. To address these issues, electrocardiogram (ECG) data from the PhysioNet Online Database will be analyzed using new spectrum estimation techniques to develop a program able to predict, as well as recognize, the onset of specific cardiac arrhythmias such as PAF. The system could then be incorporated into wearable/mobile medical devices, allowing for interventions before cardiac episodes occur, and potentially saving many lives.

Causal Learning and Autism
In collaboration with the Early Childhood Cognition Center at MIT BCS, we are developing sensor-enabled toys and infant affect sensors with the goal to understand how children on the autism spectrum use patterns of evidence to learn causal relationships and the extent with which this is state-dependent. We investigate in what respects, if any, causal learning is different in comparison to typically developing children. The results of this research will inform the design of new object-based technologies for language and communication learning.

Conductive Chat
While instant messaging clients are frequently and widely used for interpersonal communication, they lack the richness of face-to-face conversations. Without the benefit of eye contact and other non-verbal "back-channel feedback," text-based chat users frequently resort to typing "emoticons" and extraneous punctuation in an attempt to incorporate contextual affect information in the text communication. Conductive Chat is an instant messenger client that integrates users' changing skin conductivity levels into their typewritten dialogue. Skin conductivity level (also referred to as galvanic skin response) is frequently used as a measure of emotional arousal, and high levels are correlated with cognitive states such as high stress, excitement, and attentiveness. On an expressive level, Conductive Chat communicates information about each user's arousal in a consistent, intuitive manner, without needing explicit controls or explanations. On a communication-theory level, this new communication channel allows for more "media rich" conversations without requiring more work from the users.

Detection and Analysis of Driver Stress
Driving is an ideal test bed for detecting stress in natural situations. Four types of physiological signals (electrocardiogram, electromyogram, respiration, and skin conductivity related to autonomic nervous system activation) were collected in a natural driving situation under various driving conditions. The occurrence of natural stressors was designed into the driving task and validated using driver self-report, real-time, third-party observations, and independently coded video records of road conditions and facial expression. Features reflecting heart-rate variability derived from the adaptive Bayesian spectrum estimation, the rate of skin-conductivity orienting responses, and the spectral characteristics of respiration were extracted from the data. Initial pattern-recognition results show separation for the three types of driving states: rest, city, and highway, and some discrimination within states for cases in which the state precedes or follows a difficult turn-around or toll situation. These results yielded from 89-96 percent accuracy in recognizing stress level. We are currently investigating new, advanced means of modeling the driver data.

EmoteMail
EmoteMail is an email client that is augmented to convey aspects of the state of the writer during the composition of email to the recipient. The client captures facial expressions and typing speed and introduces them as design elements. These contextual cues provide extra information that can help the recipient decode the tone of the email. Moreover, the contextual information is gathered and automatically embedded as the sender composes the email, allowing an additional channel of expression.

Emotion Bottles
The Emotion Bottles are tangibly enticing objects that embody three emotions: angry, happy, and sad. When a bottle is opened, a vocal output is generated as if the emotion that was stored within the bottle is released. The bottles are placed near each other and represent a person in three possible emotional states. Varying degrees of these emotions are "bottled up" inside. The three bottles were chosen to maintain the simplicity of exploring the combination of distinct emotional states (eight possibilities). While not completely representative of the possible emotional state of a person, the bottles explore the interface in accessing emotions, the interaction between conflicting emotions, and the meaning of transition between clear emotional states as a person empathizes with or projects their feelings onto the bottles.

Emotional DJ
The technology in this project changes facial expressions in videos without the system knowing anything in particular about the person's face ahead of time. There are a few reasons to create something like this: first, it provides an artistic tool with which to alter photos or videos; second, it could be set up to let people open-endedly explore their facial communication and expressiveness by playing with a real-time video of their own current face; finally, E-DJ demonstrates an unexpected way in which we can't always trust the video information we love to consume.

Enhanced Sensory Perception
As the population ages, acuity in one or more sensory channels often diminishes or may be totally lost. Augmenting or compensating for loss in the perceptual system by taking advantage of sensory data outside the normal human range and mapping it to meaningful perceptual information has the potential of giving an ordinary person enhanced sensory perception (ESP). Sensory deficiency is not restricted to any particular segment of the population, however. For example, we tend to be myopic about ourselves, and thus can benefit from psychological mirrors in the form of trainers or therapists who can assess and guide our physical and/or mental development. In this spirit, "Reflective Biometrics" is a novel approach to analyzing and interpreting biometric sensory information for self monitoring and examination. It is self-examination via technology as a mirror. Biometric technologies in service of the individual can serve as reflectors that enhance our self-awareness, self-understanding, and health, and they can facilitate our interaction with computers and with each other by augmenting our perceptual system.

EyeJacking: See What I See
While modern communication technologies mean that we can connect to more people, these connections lack the affective subtleties inherent in situated interactions. EyeJacking is an application for the sharing of experiences in which one or more persons “eyejack— a person’s visual field to share what he or she sees. Using a wearable camera/micorphone system, remote interaction partners can share an experience first-hand and play an active role in shaping the experience. We explore the application of EyeJacking as a tool for situated learning for individuals on the autism spectrum, where parents, caregivers, or peers could “eyejack— and tag the world remotely. We also explore the application of EyeJacking to leverage the power of the masses to bootstrap people-sense abilities in robots.

Fostering Affect Awareness and Regulation in Learning
Sometimes learners have to focus while experiencing strong emotions (e.g., family problems). They may also face challenges in perservering when encountering repeated failures in problem solving. The ability to know what one is feeling (e.g., worried, frustrated) and rise above it and handle the situation productively involves meta-affective skills. With such skills, a learner feeling "I can't do this; I want to quit," might instead think, "I am frustrated, but this is OK—it happens to experts. I should look for a different way to solve this." This research develops theory and technology to help learners develop meta-affective skills. Two recent achievements are development of (1) a technology with machine "common-sense" emotion—reasoning for enabling teenage girls to reflect on emotions in stories that they've constructed and improve their affect awareness; and (2) a technology to help students become stronger learners even when they feel like quitting.

Galvactivator
The galvactivator is a glove-like wearable device that senses the wearer's skin conductivity and maps its values to a bright LED display. Increases in skin conductivity across the palm tend to be good indicators of physiological arousal, causing the galvactivator display to glow brightly. The galvactivator has many potentially useful purposes, ranging from self-feedback for stress management, to facilitation of conversation between two people, to new ways of visualizing mass excitement levels in performance situations or visualizing aspects of arousal and attention in learning situations. One of the findings in mass-communication settings was that people tended to "glow" when a new speaker came onstage, and during live demos, laughter, and live audience interaction. They tended to "go dim" during powerpoint presentations. In smaller educational settings, students have commented on how they tend to glow when they are more engaged with learning.

Gene Expression Data Analysis
This research aims to classify gene expression data sets into different categories, such as normal vs. cancer. The main challenge is that thousands of genes are measured in the micro-array data, while only a small subset of genes are believed to be relevant for disease classification. We have developed a novel approach called "predictive automatic relevance determination;" this method brings Bayesian tools to bear on the problem of selecting which genes are relevant, and extends our earlier work on the development of the "expectation propagation" algorithm. In our simulations, the new method outperforms several state-of-the-art methods, including support-vector machines with feature selection and relevance-vector machines.

Guilt Detection
The goal of this project is to produce a guilt detector. We have created an experiment that is designed to produce feelings of guilt of varying levels in different groups while we record EKG and skin conductivity. By examining the differences in physiology across the conditions, we have exlored how one might build a classifier to determine which condition, and thus which level of guilt, an individual is experiencing.

HandWave
HandWave is a small, wireless, networked skin conductance sensor that can be worn or used in many different form factors. Skin conductance is the best known measure of arousal (whether emotional, cognitive, or physical) and this device makes it easy to gather this information from mobile users. Many existing affective computing systems make use of sensors that are inflexible and often physically attached to supporting computers. In contrast, HandWave allows an additional degree of flexibility by providing ad hoc wireless networking capabilities to a wide variety of Bluetooth devices as well as adaptive biosignal amplification. As a consequence, HandWave is useful in games, tutoring systems, experimental data collection, and augmented journaling, among other applications. The Handwave builds on the earlier Galvactivator project.

Human Motion Signatures
Given motion capture samples of Charlie Chaplin's walk, is it possible to synthesize other motions—say, ascending or descending stairs—in his distinctive style? More generally, in analogy with handwritten signatures, do people have characteristic motion signatures that individualize their movements? If so, can these signatures be extracted from example motions? Furthermore, can extracted signatures be used to recognize, say, a particular individual's walk subsequent to observing examples of other movements produced by this individual? We are developing an algorithm that extracts motion signatures and uses them in the animation of graphical characters. For example, given a corpus of walking, stair ascending, and stair descending motion data collected over a group of subjects, plus a sample walking-motion for a new subject, our algorithm can synthesize never-before-seen ascending and descending motions in the distinctive style of this new individual.

In Search of Wonder: Measuring Our Response to the Miraculous
The wonder that occurs while watching a good magic trick or admiring a gorgeous natural vista is a strong emotion that has not been well studied. Educators, media producers, entertainers, scientists and magicians could all benefit from a more robust understanding of wonder. A new model was developed, and an experiment was conducted to investigate how several variables affect how magic tricks are enjoyed. The experiment showed 70 subjects 10 videos of magic while recording their responses and reactions to the tricks. Some individuals were shown the explanations to the magic tricks to gauge their impact on enjoyment. The style of the presentation was varied between two groups to compare the effect of magic presented as a story to magic presented as a puzzle. Presentation style has an effect on magic enthusiasts' enjoyment and a story-oriented presentation is associated with individuals being more generous towards a charity.

INNER-active Journal
The purpose of the INNER-active Journal system is to provide a way for users to reconstruct their emotions around events in their lives, and to see how recall of these events affects their physiology. Expressive writing, a task in which the participant is asked to write about extremely emotional events, is presented as a means towards story construction. Previous use of expressive writing has shown profound benefits for both psychological and physical health. In this system, measures of skin conductivity, instantaneous heart rate, and heart stress entropy are used as indicators of activities occurring in the body. Users have the ability to view these signals after taking part in an expressive writing task.

Interface Tailor
The Interface Tailor is an agent that attempts to adapt a system in response to affective feedback. Frustration is being used as a fitness function to select between a wide variety of different system behaviors. Currently, the Microsoft Office Assistant (or Paperclip) is one example interface that is being made more adaptive. Ultimately the project seeks to provide a generalized framework for making all software more tailor-able.

Learning Companion
"I can't do this" and "I'm not good at this" are common statements made by kids while trying to learn. Usually triggered by affective states of confusion, frustration, and hopelessness, these statements represent some of the greatest problems left unaddressed by educational reform. Education has emphasized conveying a great deal of information and facts, and has not modeled the learning process. When teachers present material to the class, it is usually in a polished form that omits the natural steps of making mistakes (feeling confused), recovering from them (overcoming frustration), deconstructing what went wrong (not becoming dispirited), and finally starting over again (with hope and maybe even enthusiasm). Learning naturally involves failure and a host of associated affective responses. This project aims to build a computerized learning companion that facilitates the child's own efforts at learning. The goal of the companion is to help keep the child's exploration going, by occasionally prompting with questions or feedback, and by watching and responding to the affective state of the child—watching especially for signs of frustration and boredom that may precede quitting, for signs of curiosity or interest that tend to indicate active exploration, and for signs of enjoyment and mastery, which might indicate a successful learning experience. The companion is not a tutor that knows all the answers but rather a player on the side of the student, there to help him or her learn, and in so doing, learn how to learn better.

Moral Sensors
The computer's emerging capacity to communicate an individual's affect raises critical ethical concerns. Additionally, designers of perceptual computer systems face moral decisions about how the information gathered by computers with sensors can be used. As humans, we have ethical considerations that come into play when we observe and report each other's behavior. Computers, as they are currently designed, do not employ such ethical considerations. This project assess the ethical acceptability of affect sensing in three different adversarial contexts, where within each context there are also different kinds of motivations (self-oriented and charity-oriented) for the individuals to perform as best as they can.

Mouse-Behavior Analysis and Adaptive Relational Agents
The goal of this project is to develop tools to sense and adapt to a user's affective state based on his or her mouse behavior. We are developing algorithms to detect frustration level for use in usability studies. We are also exploring how more permanent personality characteristics and changes in mood are reflected in the user’s mouse behavior. Ultimately, we seek to build adaptive relational agents that tailor their interactions with the user based on these sensed affective states.

Mr. Java: Customer Support
Mr. Java is the Media Lab's wired coffee machine, which keeps track of usage patterns and user preferences. The focus of this project is to give Mr. Java a tangible customer-feedback system that collects data on user complaints or compliments. "Thumbs-up" and "thumbs-down" pressure sensors were built and their signals integrated with the state of the machine to gather data from customers regarding their ongoing experiences with the machine. Potentially, the data gathered can be used to learn how to improve the system. The system also portrays an affective, social interface to the user: helpful, polite, and attempting to be responsive to any problems reported.

Online Emotion Recognition
This project is aimed at building a system to recognize emotional expression given four physiological signals. Data was gathered from a graduate student with acting experience as she intentionally tried to experience eight different emotional states daily over a period of several weeks. Several features are extracted from each of her physiological signals. The first classifiers gave a classification result of 88% success when discriminating among 3 emotions (pure chance would be 33.3%), and of 51% when discriminating among 8 emotions (pure chance 12.5%). New, improved classifiers reach an 81% success rate when discriminating among all 8 emotions. Furthermore, an online classifier has now been built using the old method, which gives a success rate only 8% less than its old offline counterpart (i.e. 43%). We expect this percentage to sharply increase when the new methods are adapted to run online.

Personal Heart-Stress Monitor
The saying, "if you can't measure it, you can't manage it" may be appropriate for stress. Many people are unaware of their stress level, and of what is good or bad for it. The issue is complicated by the fact that while too much stress is unhealthy, a certain amount of stress can be healthy as it motivates and energizes. The "right" level varies with temperment, task, and other factors, many of which are unknown. There seems to be no data analyzing how stress levels vary for the average healthy individual, over day-to-day activities. We would like to build a device that helps to gather and present data for improving an individual's understanding of both healthy and unhealthy stress in his or her life. The device itself should be comfortable and should not increase the user's stress. (It is noteworthy that stress monitoring is also important in human-computer interaction for testing new designs.) Currently, we are building a new, wireless, stress-mornitoring system by integrating Fitsense's heart-rate sensors and Motorola's iDen cell phone with our heart-rate-variability estimation algorithm.

Posture Recognition Chair
We have developed a system to recognize posture patterns and associated affective states in real time, in an unobtrusive way, from a set of pressure sensors on a chair. This system discriminates states of children in learning situations, such as when the child is interested, or is starting to take frequent breaks and looking bored. The system uses pattern recognition techniques, while watching natural behaviors, to "learn" what behaviors tend to accompany which states. The system thus detects the surface-level behaviors (postures) and their mappings during a learning situation in an unobtrusive manner so that they don't interfere with the natural learning process. Through the chair, we can reliably detect nine static postures, and four temporal patterns associated with affective states.

Recognizing Affect in Speech
This research project is concerned with building computational models for the automatic recognition of affective expression in speech. We are in the process of completing an investigation of how acoustic parameters extracted from the speech waveform (related to voice quality, intonation, loudness and rhythm) can help disambiguate the affect of the speaker without knowledge of the textual component of the linguistic message. We have carried out a multi-corpus investigation, which includes data from actors and spontaneous speech in English, and evaluated the model's performance. In particular, we have shown that the model exhibits a speaker-dependent performance which reflects human evaluation of these particular data sets, and, held against human recognition benchmarks, the model begins to perform competitively.

Relational Agents
Relational Agents are computational artifacts designed to build and maintain long-term, social-emotional relationships with their users. Central to the notion of relationship is that it is a persistent construct, spanning multiple interactions. Thus, Relational Agents are explicitly designed to remember past history and manage future expectations in their interactions with users. Since face-to-face conversation is the primary context of relationship-building for humans, our work focuses on Relational Agents as a specialized kind of embodied conversational agent (animated humanoid software agents that use speech, gaze, gesture, intonation, and other nonverbal modalities to emulate the experience of human face-to-face conversation). One major achievement was the development of a Relational Agent for health behavior change, specifically in the area of exercise adoption. A study involving 100 subjects interacting with this agent over one month demonstrated that the relational agent was respected more, liked more, and trusted more, and that these ratings were maintained over time (unlike for the non-relational agent, where they were not only significantly lower overall, but also declined over time.) People also expressed significantly greater ratings of perceived caring by the agent, and significantly more desire to keep working with the relational agent after the termination of the study.

Self-Cam
The Self-Cam is a wearable camera apparatus that consists of a chest-mounted camera aimed at the wearer’s face. Self-Cam was designed to be used in conjunction with a belt-mounted computer and real-time mental-state inference software that can be used with visual, auditory, or tactile output as personal feedback for the wearer. As the camera faces inward, many privacy issues are avoided–only those who choose to wear the Self-cam appear in the recorded video. Head movement can be seen and analyzed alongside facial expressions because the system rests on the chest and the light, simple nature of the structure allows it to be worn without any physical discomfort. By wearing the Self-Cam, you can explore who you appear to be from the outside. The Self-Cam acts as an objective point of view that might help you to understand yourself in a different light.

ShyBot
Shybot is a personal mobile robot designed to both embody and elicit reflection on shyness behaviors. Shybot is being designed to detect human presence and familiarity from face detection and proximity sensing in order to categorize people as friends or strangers for interaction. Shybot also can reflect elements of the anxious state of its human companion through LEDs and a spinning propeller. We designed this simple social interaction to open up a new direction for intervention for children living with autism. We hope that from minimal social interaction, a child with autism or social anxiety disorders could reflect on and more deeply attain understanding about personal shyness behaviors, as a first step toward helping to make progress in developing greater capacity for complex social interactions.

The Affective Remixer: Personalized Music Arranging
Affective Remixer is a real-time music-arranging system that reacts to immediate affective cues from a listener. Data was collected on the potential of certain musical dimensions to elicit change in a listener’s affective state using sound files created explicitly for the experiment through composition/production, segmentation, and re-assembly of music along these dimensions. Based on listener data, a probabilistic state transition model was developed to infer the listener’s current affective state. A second model was made that would select music segments and re-arrange ('re-mix') them to induce a target affective state.

The Conductor's Jacket
The Conductor's Jacket is a unique wearable device that measures physiological and gestural signals. Together with the Gesture Construction, a musical software system, it interprets these signals and applies them expressively in a musical context. Sixteen sensors have been incorporated into the Conductor's Jacket in such a way as to not encumber or interfere with the gestures of a working orchestra conductor. The Conductor's Jacket system gathers up to sixteen data channels reliably at rates of 3 kHz per channel, and also provides real-time graphical feedback. Unlike many gesture-sensing systems it not only gathers positional and accelerational data but also senses muscle tension from several locations on each arm. We will demonstrate the Gesture Construction, a musical software system that analyzes and performs music in real-time based on the performer's gestures and breathing signals. A bank of software filters extract several of the features that were found in the conductor study, including beat intensities and the alternation between arms. These features are then used to generate real-time expressive effects by shaping the beats, tempos, articulations, dynamics, and note lengths in a musical score.

Touch-Phone
The Touch-Phone was developed to explore the use of objects to mediate the emotional exchange in interpersonal communication. Through an abstract visualization of screen-based color changes, a standard telephone is modified to communicate how it is being held and squeezed. The telephone receiver includes a touch-sensitive surface which conveys the user's physical response over a computer network. The recipient sees a small colored icon on his computer screen which changes in real time according to the way his conversational partner is interacting with the telephone object.

Wearable Relational Devices for Stress Monitoring
We have created a system for data collection, annotation, and feedback that is part of a longer-term research interest to gather data to understand more about stress and the physiological signals involved in its expression. First, we built a wearable apparatus for gathering data that allows the user to include as many accurate labels (annotations) as possible while going about natural daily activities. Gathering annotations is disruptive and likely to increase stress (thus interfering with the signals being measured). We hypothesized that empathetic ways of interrupting would be less stressful than non-empathetic and found significant effects on many of user's self-reported items such as preference for the more empathetic system, and also on behavioral items, such as the estimated number of times they were interrupted (significantly lower when system was more empathetic.)