Highlighted Projects:

Affect as Index
As members of different groups within the world, we accumulate knowledge that influences how we see, interpret, and, therefore, understand the world. This knowledge can lead to miscommunication and misunderstandings among groups. Many have endeavored to reduce misunderstandings by bringing different groups into contact with one another; however, these meetings do not guarantee that groups will have authentic opportunities to learn from, let alone understand, one another. People must be able to go through the process of analyzing and developing an empathetic eye toward themselves and others—a sort of reflection that we are not often trained to do. This project explores how physiological data can index media content that can guide discussions about diversity and personal experience in order to develop intergroup understanding.

Affective-Cognitive Framework for Machine Learning and Decision Making
Recent findings in affective neuroscience and psychology indicate that human affect and emotional experience play a significant and useful role in human learning and decision-making. Most machine-learning and decision-making models, however, are based on old, purely cognitive models, and are slow, brittle, and awkward to adapt. We aim to redress many of these classic problems by developing new models that integrate affect with cognition. Ultimately, such improvements will allow machines to make smarter and more human-like decisions for better human-machine interaction.

Affective-Cognitive Product Evaluation and Prediction of Customer Decisions
Companies would like more new products to be successful in the marketplace; however, current evaluation methods such as focus groups do not accurately predict customer decisions. We are developing new technology-assisted methods to try to improve the customer evaluation process and better predict customer decisions. The new methods involve multi-modal affective measures (such as facial expression and skin conductance) together with behavioral measures, anticipatory-motivational measures, and self-report cognitive measures. These measures are combined into a novel computational model, the form of which is motivated by findings in affective neuroscience and human behavior. The model is being trained and tested with customer product evaluations and marketplace outcomes from real product launches.

Emotion Communication in Autism
People who have difficulty communicating verbally (such as many people with autism) sometimes send nonverbal messages that do not match what is happening inside them. For example, a child might look calm and receptive to learning, while having a heart rate of over 120 bpm and being on the verge of a meltdown or shutdown. This mismatch can lead to serious problems, including misunderstandings such as "he became aggressive for no reason." We are creating new technologies to address this fundamental communication problem and enable the first long-term, ultra-dense longitudinal data analysis of emotion-related physiological signals. We hope to equip individuals with personalized tools to understand the influences of their physiological state on their own behavior (e.g., "which state helps me best maintain my attention and focus for learning?"). The data from daily life will also advance basic scientific understanding of the role of autonomic nervous system regulation in autism.

Emotional-Social Intelligence Toolkit
Social-emotional communication difficulties lie at the core of autism spectrum disorders, making interpersonal interactions overwhelming, frustrating, and stressful. We are developing the world’s first wearable affective technologies to help the growing number of individuals with autism–approximately 1 in 150 children in the United States–learn about nonverbal communication in a natural, social context. We are also developing technologies that build on the nonverbal communication that individuals on the autism spectrum are already using to express themselves, to help families, educators, and other persons who deal with autism spectrum disorders to better understand these alternative means of nonverbal communication. Our work leverages advances in affect sensing and perception to (1) develop technologies that are sensitive to people's affective-cognitive states; (2) advance autism research; and (3) create new technologies that enhance the social-emotional communications of people diagnosed with autism, as well as those who are not.

Evaluation Tool for Recognition of Social-Emotional Expressions from Facial-Head Movements
To help people improve their reading of faces during natural conversations, we developed a video tool to evaluate this skill. First, we collected over 100 videos of conversations between pairs of both autistic and neurotypical people, each of whom wore a Self-Cam. Next, the videos were manually segmented into chunks of 7-20 seconds according to expressive content, labeled, and sorted by difficulty—all tasks we plan to automate using technologies under development. Next, we built a rating interface including videos of self, peers, familiar adults, strangers, and unknown actors, allowing for performance comparisons across conditions of familiarity and expression. We obtained reliable identification (by coders) of categories of smiling, happy, interested, thinking, and unsure in the segmented videos. The tool was finally used to assess recognition of these five categories for eight neurotypical and five autistic people. Results show some autistics approaching the abilities of the neurotypicals while several score just above random.

EyeJacking: See What I See
While modern communication technologies mean that we can connect to more people, these connections lack the affective subtleties inherent in situated interactions. EyeJacking is an application for the sharing of experiences in which one or more persons “eyejack” a person’s visual field to share what he or she sees. Using a wearable camera/micorphone system, remote interaction partners can share an experience first-hand and play an active role in shaping the experience. We explore the application of EyeJacking as a tool for situated learning for individuals on the autism spectrum, where parents, caregivers, or peers could “eyejack” and tag the world remotely. We also explore the application of EyeJacking to leverage the power of the masses to bootstrap people-sense abilities in robots.

Fostering Affect Awareness and Regulation in Learning
Sometimes learners have to focus while experiencing strong emotions (e.g., family problems). They may also face challenges in perservering when encountering repeated failures in problem solving. The ability to know what one is feeling (e.g., worried, frustrated) and rise above it and handle the situation productively involves meta-affective skills. With such skills, a learner feeling "I can't do this; I want to quit," might instead think, "I am frustrated, but this is OK—it happens to experts. I should look for a different way to solve this." This research develops theory and technology to help learners develop meta-affective skills. Two recent achievements are development of (1) a technology with machine "common-sense" emotion—reasoning for enabling teenage girls to reflect on emotions in stories that they've constructed and improve their affect awareness; and (2) a technology to help students become stronger learners even when they feel like quitting.

Girls Involved in Real-Life Sharing
In this research, a proactive emotional health system, geared toward supporting emotional self-awareness and empathy, was built as a part of a long-term research plan for understanding the role digital technology can play in helping people to reflect on their beliefs, attitudes, and values. The system, G.I.R.L.S. (Girls Involved in Real-Life Sharing), allows users to reflect actively upon the emotions related to their situations through the construction of pictorial narratives. The system employs common-sense reasoning to infer affective content from the users' stories and support emotional reflection. Users of this new system were able to gain new knowledge and understanding about themselves and others through the exploration of authentic and personal experiences. Currently, the project is being turned into an online system for use by school counselors.

iCalm (TM): Wireless Bio-Sensing for iPod and Cell Phone
We are developing a wireless sensor platform that allows easy integration of wearable biosensors with various consumer products, such as an iPod or cell phone. This platform has many applications, including health monitoring for outpatients or eldercare, fitness products, and various types of interactive content (e.g., MP3 music, video) that respond to the wearer's health or mood. Initial applications include: (1) personalized relapse-prevention messages for abstinent drug addicts, triggered by physiological craving signals; (2) mood-triggered music selections; and (3) a personal monitor for understanding the influence of autonomic arousal in autism.

Machine Learning and Pattern Recognition with Multiple Modalities
This project develops new theory and algorithms to enable computers to make rapid and accurate inferences from multiple modes of data, such as determining a person's affective state from multiple sensors—video, mouse behavior, chair pressure patterns, typed selections, physiology, and more. Recent efforts focus on understanding the level of a person's attention, which is useful for things such as determining when to interrupt. Our approach is Bayesian: formulating probabilistic models on the basis of domain knowledge and training data, and then performing inference according to the rules of probability theory. This type of sensor fusion work is especially challenging because of the problems of sensor channel drop-out, different kinds of noise in different channels, dependence between channels, scarce and sometimes inaccurate labels, and patterns to detect that are inherently time-varying. We have constructed a variety of new algorithms for solving these problems and demonstrated their performance gains over other state-of-the-art methods.

Mechatronics and Prompt-Assisted Typing Aids
People on the autism spectrum face a number of challenges, including motor movement issues that can cause limbs to cease activity. Circumstantial evidence suggests that autonomic nervous system influences related to stress and overload may arise from and contribute to these problems. We propose to allow individuals to monitor several physiological parameters to see if there are patterns that recognize or predict the onset of their individual motor problems. We plan to develop new, wearable technology to treat these problems via the use of tiny, vibrotactile devices carefully placed at the joints. We hypothesize that some methods of touch-feedback and vibration at the joints may enable individuals to recover motor functioning during episodes of intermittent loss. We are also exploring the development of personally controlled devices that facilitate finer motor movement for augmenting communication as needed for assisting in typing or pointing.

RoCo: A Robotic Desktop Computer
A robotic computer that moves its monitor "head" and "neck," but that has no explicit face, is being designed to interact with users in a natural way for applications such as learning, rapport-building, interactive teaching, and posture improvement. In all these applications, the robot will need to move in subtle ways that express its state and promote appropriate movements in the user, but that don't distract or annoy. Toward this goal, we are giving the system the ability to recognize states of the user and also to have subtle expressions.

Self-Cam
The Self-Cam is a wearable camera apparatus that consists of a chest-mounted camera aimed at the wearer’s face. Self-Cam was designed to be used in conjunction with a belt-mounted computer and real-time mental-state inference software that can be used with visual, auditory, or tactile output as personal feedback for the wearer. As the camera faces inward, many privacy issues are avoided–only those who choose to wear the Self-cam appear in the recorded video. Head movement can be seen and analyzed alongside facial expressions because the system rests on the chest and the light, simple nature of the structure allows it to be worn without any physical discomfort. By wearing the Self-Cam, you can explore who you appear to be from the outside. The Self-Cam acts as an objective point of view that might help you to understand yourself in a different light.

ShyBot
Shybot is a personal mobile robot designed to both embody and elicit reflection on shyness behaviors. Shybot is being designed to detect human presence and familiarity from face detection and proximity sensing in order to categorize people as friends or strangers for interaction. Shybot also can reflect elements of the anxious state of its human companion through LEDs and a spinning propeller. We designed this simple social interaction to open up a new direction for intervention for children living with autism. We hope that from minimal social interaction, a child with autism or social anxiety disorders could reflect on and more deeply attain understanding about personal shyness behaviors, as a first step toward helping to make progress in developing greater capacity for complex social interactions.

Soothing Soundscapes for Autism
Persons with autism often report extreme hypersensitivity to sound. Researchers believe this hypersensitivity may be related to the acoustic quality of the sound (e.g., its frequency, intensity, and duration), and the context within which it occurs. Our primary aim is to offer persons with autism more control over their acoustic environment, regardless of the context. We are developing new technology for autistic individuals that will allow real-time control over the intensity and frequency characteristics of everyday sounds. This technology will also offer persons with autism new ways to record and document the sounds they find particularly aversive. Psychophysiological sensors will also be incorporated to assess the role of arousal in auditory hypersensitivity.

Prior Projects:

AboutFace
AboutFace is a user-dependent system that is able to learn patterns and discriminate the different facial movements characterizing confusion and interest. The system uses a piezoelectric sensor to detect eyebrow movements and begins with a training session to calibrate the unique values for each user. After the training session, the system uses these levels to develop an expression profile for the individual user. The system has many potential uses, ranging from computer and video-mediated conversations to interactions with computer agents. This system is an alternative to using camera-based computer vision analysis to detect faces and expressions. Additionally, when communicating with other people, users of this system also have the option of conveying their expressions anonymously by wearing a pair of glasses that conceals their expressions and the sensing device.

Adaptive, Wireless, Signal Detection and Decoding
In this project, we propose a new Bayesian receiver for signal detection in flat-fading channels. First, the detection problem is formulated as an inference problem in a hybrid dynamic system that has both continuous and discrete variables. Then, an expectation propagation algorithm is proposed to address the inference problem. As an extension of belief propagation, expectation propagation efficiently approximates a Bayesian estimation by iteratively propagating information between different nodes in the dynamic system and projecting exact messages into the exponential family. Compared to sequential Monte Carlo filters and smoothers, the new method has much lower computational complexity since it makes analytically deterministic approximations instead of Monte Carlo approximations. Our simulations demonstrate that the new receiver achieves accurate detection without the aid of any training symbols or decision feedbacks. Future work involves joint decoding and channel estimation, where convolutional codes are used to protect signals from noise corruption. Initial results are promising.

Affect in Speech: Assembling a Database
The aim of this project is to build a database of natural speech showing a range of affective variability. It is an extension of our ongoing research focused on building models for automatic detection of affect in speech. At a very basic level, training such systems requires a large corpus of speech containing a range of emotional vocal variation. A traditional approach to this research has been to assemble databases where actors have provided the affective variation on demand. However, this method often results in unnatural sounding speech and/or exaggerated expressions. We have developed a prototype of an interactive system that guides a user through a question and answer session. Without any rehearsals or scripts, the user navigates through touch and spoken language an interface guided by embodied conversational agents which prompt the user to speak about an emotional experience. Some of the issues we are addressing include the design of the text and character behavior (including speech and gesture) so as to obtain a convincing and disclosing interaction with the user.

Affective Carpet
The "Affective Carpet" is a soft, deformable surface made of cloth and foam, which detects continuous pressure with excellent sensitivity and resolution. It is being used as an interface for projects in affective expression, including as a controller to measure a musical performer's direction and intensity in leaning and weight-shifting patterns.

Affective Learning Companion
Affective Learning Companion is a powerful, flexible new research tool for exploring a variety of social-emotional skills in human-machine interaction, and for understanding how machines can work with people to better meet their needs. The platform enables a computational agent to sense and respond, in real time, to a user's non-verbal emotional cues, using video, postural movements, mouse pressure, physiology, and other behaviors communicated by the user to infer, for example, if a user is in a high or low state of interest, or feeling frustrated. We recently developed an animated agent that combines non-verbal mirroring (or not) with multiple kinds of affective and cognitive support during a frustrating learning episode. The system allows us to control factors that have previously been impossible to control, enabling for the first time the study of how these factors interact in helping learners develop the ability to persevere during frustrating learning episodes.

Affective Mirror
The Affective Mirror is an attempt to build a fully automated system that intelligently responds to a person's affective state in real time. Current work is focused on building an agent that realistically mirrors a person's facial expression and posture. The agent detects affective cues through a facial-feature tracker and a posture-recognition system developed in the Affective Computing group; based on what affect a person is displaying, such as interest, boredom, frustration, or confusion, the system responds with matching facial affect and/or posture. This project is designed to be integrated into the Learning Companion Project, as part of an early phase of showing rapport-building behaviors between the computer agent and the human learner.

Affective Social Quest
ASQ investigates ways to teach social-emotion skills to children interactively with toys. One of the first goals is to help autistic children recognize expressions of emotion in social situations. The system uses four "dwarfs" expressing sad, happy, surprise, and angry, and each communicates wirelessly to the system and detects which plush doll was selected by the child. The computer plays short entertaining video clips displaying examples of the four emotions and cues the child to pick a dwarf that closely matches that emotion. Future work includes improving the ability of the system to recognize direct displays of emotion by the child.

Affective Tangibles
People naturally express frustration through the use of their motor skills. The purpose of the Affective Tangibles project is to develop physical objects that can be grasped, squeezed, thrown, or otherwise manipulated via a natural display of affect. Constructed tangibles include a PressureMouse, affective pinwheels that are mapped to skin conductance, and a voodoo doll that can be shaken to express frustration. We have found that people often increase the intensity of muscle movements when experiencing frustrating interactions.

Affective Tigger
The Affective Tigger is a plush toy designed to recognize and react to certain emotinal behaviors of its playmate. For example the toy enters a state of "happy," moving its ears upward and emitting a happy vocalization when it recognizes that the child has postured the toy upright and is bouncing it along the floor. Tigger has five such states, involving recognizing and responding with an emotional behavior. The resulting behavior Tigger demonstrates allows it to serve as an affective mirror for the child's expression. This work involved designing the toy, and evaluating sessions of play with it with dozens of kids. The toy was shown to successfully communicate some aspects of emotion, and to prompt behaviors that are interesting to researchers trying to learn about the development of human emotional skills such as empathy.

AffQuake
AffQuake is an attempt to incorporate signals that relate to a player's affect into ID Software's Quake II in a way that alters game play. Several modifications have been made that cause the player's avatar within Quake to alter its behaviors depending upon one of these signals. In StartleQuake, when a player becomes startled, his or her avatar also becomes startled and jumps back. Quake changes the size of the player's avatar in relation to the user's response as well, representing player excitement by average skin conductance level, and growing the avatar's size when this level is high. A taller avatar means the player can see further; however, it also makes him or her an easier target.

Ambient Displays for Social Support and Diabetes Management
We design and evaluate an ambient blood glucose level visualization and feedback system that uses an Ambient Orb for diabetes self-care and social support. The social support is provided by a friend or family member of an individual with diabetes. This research study was carried out with adult patients at Joslin Diabetes Center.

Automatic Facial Expression Analysis
Recognizing non-verbal cues, which constitute a large percentage of our communication, is a prime facet of building emotionally intelligent systems. Facial expressions and movements such as a smile or a nod are used either to fulfill a semantic function, to communicate emotions, or as conversational cues. We are developing an automatic tool using computer vision and various machine-learning techniques, which can detect the different facial movements and head gestures of people while they are interacting naturally with the computer. Past work on this project determined techniques to track upper facial features (eyes and eyebrows) and detect facial actions corresponding to those features (eyes squintint or widening, eyebrows raised). The ongoing project is expanding its scope to track and detect facial actions corresponding to the lower features. Further, we hope to integrate the facial expression analysis module with other sensors developed by the Affective Computing group to reliably detect and recognize different emotions.

Bayesian Spectral Estimation
This project developed efficient versions of Bayesian techniques for a variety of inference problems, including curve fitting, mixture-density estimation, principal-components analysis (PCA), automatic relevance determination, and spectral analysis. One of the surprising methods that resulted is a new Bayesian spectral analysis tool for nonstationary and unevenly sampled signals, such as electrocardiogram (EKG) signals, where there is a sample with each new (irregularly spaced) R wave. The new method outperforms other methods such as Burg, Music, and Welch, and compares favorably to the multitaper method without requiring any windowing. The ability to use unevenly spaced data helps avoid problems with aliasing. The method runs in real time on either evenly or unevenly sampled data.

BioMod
BioMod is an integrated interface for users of mobile and wearable devices, monitoring various physiological signals such as the electrocardiogram, with the intention of providing useful and comfortable feedback about medically important information. The first version of this system includes new software for monitoring stress and its impact on heart functioning, and the ability to wirelessly communicate this information over a Motorola cell phone. One application under development is the monitoring of stress in patients who desire to stop smoking: the system will alert an "on-call" trained behavior-change assistant when the smoker is exhibiting physiological patterns indicative of stress or likely relapse, offering an opportunity for encouraging intervention at a point of weakness. Challenges in this project include the development of an interface that is easy and efficient to use on the go, is sensitive to user feelings about the nature of the information being communicated, and accurately recognizes the patterns of physiological signals related to the conditions of interest.

Car Phone Stress
We are building a system that can watch for certain signs of stress in drivers, specifically stress related to talking on the car phone, as may be caused by increased mental workload. To gather data for training and testing our system, subjects were asked to 'drive' in a simulator past several curves while keeping their speed close to a predetermined desired constant value. In some cases they were simultaneously asked to listen to random numbers from a speech-synthesis software and to perform simple mathematical tasks over a telephone headset. Several measures drawn from the subjects' driving behavior were examined as possible indicators of the subjects' performance and of their mental workload. When subjects were instructed (by a visible sign) to brake, most braked within 0.7-1.4 seconds after the sign came into view. However, in a significant number of incidents, subjects never braked or braked 1.5-3.5 seconds after the message; almost all of these incidents were when subjects were on the phone. On average, we found that drivers on the phone braked 10% slower than when not on the phone; additionally, the variance in their braking time was four times higher -- suggesting that although delayed driver reactions were infrequent, when delays happened they could be large and potentially dangerous. Furthermore, their infrequency could create a false sense of security. In future experiments, subjects' physiological data will be analyzed jointly with measures of workload, stress and performance.

Cardiac PAF Detection and Prediction
PAF (paroxysmal atrial fibrillation) is a dangerous form of cardiac arrhythmia that poses severe health risks, sometimes leading to heart attacks, the recognized number-one killer in the developed world. The technical challenges for detecting and predicting PAF include accurate sensing, speedy analysis, and a workable classification system. To address these issues, electrocardiogram (ECG) data from the PhysioNet Online Database will be analyzed using new spectrum estimation techniques to develop a program able to predict, as well as recognize, the onset of specific cardiac arrhythmias such as PAF. The system could then be incorporated into wearable/mobile medical devices, allowing for interventions before cardiac episodes occur, and potentially saving many lives.

Conductive Chat
While instant messaging clients are frequently and widely used for interpersonal communication, they lack the richness of face-to-face conversations. Without the benefit of eye contact and other non-verbal "back-channel feedback," text-based chat users frequently resort to typing "emoticons" and extraneous punctuation in an attempt to incorporate contextual affect information in the text communication. Conductive Chat is an instant messenger client that integrates users' changing skin conductivity levels into their typewritten dialogue. Skin conductivity level (also referred to as galvanic skin response) is frequently used as a measure of emotional arousal, and high levels are correlated with cognitive states such as high stress, excitement, and attentiveness. On an expressive level, Conductive Chat communicates information about each user's arousal in a consistent, intuitive manner, without needing explicit controls or explanations. On a communication-theory level, this new communication channel allows for more "media rich" conversations without requiring more work from the users.

Detection and Analysis of Driver Stress
Driving is an ideal test bed for detecting stress in natural situations. Four types of physiological signals (electrocardiogram, electromyogram, respiration, and skin conductivity related to autonomic nervous system activation) were collected in a natural driving situation under various driving conditions. The occurrence of natural stressors was designed into the driving task and validated using driver self-report, real-time, third-party observations, and independently coded video records of road conditions and facial expression. Features reflecting heart-rate variability derived from the adaptive Bayesian spectrum estimation, the rate of skin-conductivity orienting responses, and the spectral characteristics of respiration were extracted from the data. Initial pattern-recognition results show separation for the three types of driving states: rest, city, and highway, and some discrimination within states for cases in which the state precedes or follows a difficult turn-around or toll situation. These results yielded from 89-96 percent accuracy in recognizing stress level. We are currently investigating new, advanced means of modeling the driver data.

EmoteMail
EmoteMail is an email client that is augmented to convey aspects of the state of the writer during the composition of email to the recipient. The client captures facial expressions and typing speed and introduces them as design elements. These contextual cues provide extra information that can help the recipient decode the tone of the email. Moreover, the contextual information is gathered and automatically embedded as the sender composes the email, allowing an additional channel of expression.

Emotion Bottles
The Emotion Bottles are tangibly enticing objects that embody three emotions - Angry, Happy, and Sad. When a bottle is opened, a vocal output is generated as if the emotion that was stored within the bottle is released. The bottles are placed near each other and represent a person in three possible emotional states. Varying degrees of these emotions are ‘bottled up’ inside. The three bottles were chosen to maintain the simplicity of exploring the combination of distinct emotional states (8 possibilities). While not completely representative of the possible emotional state of a person, the bottles explore the interface in accessing emotions, the interaction between conflicting emotions, and the meaning of transition between clear emotional states as a person empathizes with or projects their feelings onto the bottles.

Emotional DJ
The technology in this project changes facial expressions in videos without the system knowing anything in particular about the person's face ahead of time. There are a few reasons to create something like this: first, it provides an artistic tool with which to alter photos or videos; second, it could be set up to let people open-endedly explore their facial communication and expressiveness by playing with a real-time video of their own current face; finally, E-DJ demonstrates an unexpected way in which we can't always trust the video information we love to consume.

Enhanced Sensory Perception
As the population ages, acuity in one or more sensory channels often diminishes or may be totally lost. Augmenting or compensating for loss in the perceptual system by taking advantage of sensory data outside the normal human range and mapping it to meaningful perceptual information has the potential of giving an ordinary person enhanced sensory perception (ESP). Sensory deficiency is not restricted to any particular segment of the population, however. For example, we tend to be myopic about ourselves, and thus can benefit from psychological mirrors in the form of trainers or therapists who can assess and guide our physical and/or mental development. In this spirit, "Reflective Biometrics" is a novel approach to analyzing and interpreting biometric sensory information for self monitoring and examination. It is self-examination via technology as a mirror. Biometric technologies in service of the individual can serve as reflectors that enhance our self-awareness, self-understanding, and health, and they can facilitate our interaction with computers and with each other by augmenting our perceptual system.

Galvactivator
The galvactivator is a glove-like wearable device that senses the wearer's skin conductivity and maps its values to a bright LED display. Increases in skin conductivity across the palm tend to be good indicators of physiological arousal, causing the galvactivator display to glow brightly. The galvactivator has many potentially useful purposes, ranging from self-feedback for stress management, to facilitation of conversation between two people, to new ways of visualizing mass excitement levels in performance situations or visualizing aspects of arousal and attention in learning situations. One of the findings in mass-communication settings was that people tended to "glow" when a new speaker came onstage, and during live demos, laughter, and live audience interaction. They tended to "go dim" during powerpoint presentations. In smaller educational settings, students have commented on how they tend to glow when they are more engaged with learning.

Gene Expression Data Analysis
This research aims to classify gene expression data sets into different categories, such as normal vs. cancer. The main challenge is that thousands of genes are measured in the micro-array data, while only a small subset of genes are believed to be relevant for disease classification. We have developed a novel approach called "predictive automatic relevance determination;" this method brings Bayesian tools to bear on the problem of selecting which genes are relevant, and extends our earlier work on the development of the "expectation propagation" algorithm. In our simulations, the new method outperforms several state-of-the-art methods, including support-vector machines with feature selection and relevance-vector machines.

Guilt Detection
The goal of this project is to produce a guilt detector. We have created an experiment that is designed to produce feelings of guilt of varying levels in different groups while we record EKG and skin conductivity. By examining the differences in physiology across the conditions, we have exlored how one might build a classifier to determine which condition, and thus which level of guilt, an individual is experiencing.

HandWave
HandWave is a small, wireless, networked skin conductance sensor that can be worn or used in many different form factors. Skin conductance is the best known measure of arousal (whether emotional, cognitive, or physical) and this device makes it easy to gather this information from mobile users. Many existing affective computing systems make use of sensors that are inflexible and often physically attached to supporting computers. In contrast, HandWave allows an additional degree of flexibility by providing ad hoc wireless networking capabilities to a wide variety of Bluetooth devices as well as adaptive biosignal amplification. As a consequence, HandWave is useful in games, tutoring systems, experimental data collection, and augmented journaling, among other applications.

Human Motion Signatures
Given motion capture samples of Charlie Chaplin's walk, is it possible to synthesize other motions—say, ascending or descending stairs—in his distinctive style? More generally, in analogy with handwritten signatures, do people have characteristic motion signatures that individualize their movements? If so, can these signatures be extracted from example motions? Furthermore, can extracted signatures be used to recognize, say, a particular individual's walk subsequent to observing examples of other movements produced by this individual? We are developing an algorithm that extracts motion signatures and uses them in the animation of graphical characters. For example, given a corpus of walking, stair ascending, and stair descending motion data collected over a group of subjects, plus a sample walking-motion for a new subject, our algorithm can synthesize never-before-seen ascending and descending motions in the distinctive style of this new individual.

In Search of Wonder: Measuring Our Response to the Miraculous
The wonder that occurs while watching a good magic trick or admiring a gorgeous natural vista is a strong emotion that has not been well studied. Educators, media producers, entertainers, scientists and magicians could all benefit from a more robust understanding of wonder. A new model was developed, and an experiment was conducted to investigate how several variables affect how magic tricks are enjoyed. The experiment showed 70 subjects 10 videos of magic while recording their responses and reactions to the tricks. Some individuals were shown the explanations to the magic tricks to gauge their impact on enjoyment. The style of the presentation was varied between two groups to compare the effect of magic presented as a story to magic presented as a puzzle. Presentation style has an effect on magic enthusiasts' enjoyment and a story-oriented presentation is associated with individuals being more generous towards a charity.

INNER-active Journal
The purpose of the INNER-active Journal system is to provide a way for users to reconstruct their emotions around events in their lives, and to see how recall of these events affects their physiology. Expressive writing, a task in which the participant is asked to write about extremely emotional events, is presented as a means towards story construction. Previous use of expressive writing has shown profound benefits for both psychological and physical health. In this system, measures of skin conductivity, instantaneous heart rate, and heart stress entropy are used as indicators of activities occurring in the body. Users have the ability to view these signals after taking part in an expressive writing task.

Interface Tailor
The Interface Tailor is an agent that attempts to adapt a system in response to affective feedback. Frustration is being used as a fitness function to select between a wide variety of different system behaviors. Currently, the Microsoft Office Assistant (or Paperclip) is one example interface that is being made more adaptive. Ultimately the project seeks to provide a generalized framework for making all software more tailor-able.

Learning Companion
"I can't do this" and "I'm not good at this" are common statements made by kids while trying to learn. Usually triggered by affective states of confusion, frustration, and hopelessness, these statements represent some of the greatest problems left unaddressed by educational reform. Education has emphasized conveying a great deal of information and facts, and has not modeled the learning process. When teachers present material to the class, it is usually in a polished form that omits the natural steps of making mistakes (feeling confused), recovering from them (overcoming frustration), deconstructing what went wrong (not becoming dispirited), and finally starting over again (with hope and maybe even enthusiasm). Learning naturally involves failure and a host of associated affective responses. This project aims to build a computerized learning companion that facilitates the child's own efforts at learning. The goal of the companion is to help keep the child's exploration going, by occasionally prompting with questions or feedback, and by watching and responding to the affective state of the child—watching especially for signs of frustration and boredom that may precede quitting, for signs of curiosity or interest that tend to indicate active exploration, and for signs of enjoyment and mastery, which might indicate a successful learning experience. The companion is not a tutor that knows all the answers but rather a player on the side of the student, there to help him or her learn, and in so doing, learn how to learn better.

Moral Sensors
The computer's emerging capacity to communicate an individual's affect raises critical ethical concerns. Additionally, designers of perceptual computer systems face moral decisions about how the information gathered by computers with sensors can be used. As humans, we have ethical considerations that come into play when we observe and report each other's behavior. Computers, as they are currently designed, do not employ such ethical considerations. This project assess the ethical acceptability of affect sensing in three different adversarial contexts, where within each context there are also different kinds of motivations (self-oriented and charity-oriented) for the individuals to perform as best as they can.

Mouse-Behavior Analysis and Adaptive Relational Agents
The goal of this project is to develop tools to sense and adapt to a user's affective state based on his or her mouse behavior. We are developing algorithms to detect frustration level for use in usability studies. We are also exploring how more permanent personality characteristics and changes in mood are reflected in the user’s mouse behavior. Ultimately, we seek to build adaptive relational agents that tailor their interactions with the user based on these sensed affective states.

Mr. Java: Customer Support
Mr. Java is the Media Lab's wired coffee machine, which keeps track of usage patterns and user preferences. The focus of this project is to give Mr. Java a tangible customer-feedback system that collects data on user complaints or compliments. "Thumbs-up" and "thumbs-down" pressure sensors were built and their signals integrated with the state of the machine to gather data from customers regarding their ongoing experiences with the machine. Potentially, the data gathered can be used to learn how to improve the system. The system also portrays an affective, social interface to the user: helpful, polite, and attempting to be responsive to any problems reported.

Online Emotion Recognition
This project is aimed at building a system to recognize emotional expression given four physiological signals. Data was gathered from a graduate student with acting experience as she intentionally tried to experience eight different emotional states daily over a period of several weeks. Several features are extracted from each of her physiological signals. The first classifiers gave a classification result of 88% success when discriminating among 3 emotions (pure chance would be 33.3%), and of 51% when discriminating among 8 emotions (pure chance 12.5%). New, improved classifiers reach an 81% success rate when discriminating among all 8 emotions. Furthermore, an online classifier has now been built using the old method, which gives a success rate only 8% less than its old offline counterpart (i.e. 43%). We expect this percentage to sharply increase when the new methods are adapted to run online.

Personal Heart-Stress Monitor
The saying, "if you can't measure it, you can't manage it" may be appropriate for stress. Many people are unaware of their stress level, and of what is good or bad for it. The issue is complicated by the fact that while too much stress is unhealthy, a certain amount of stress can be healthy as it motivates and energizes. The "right" level varies with temperment, task, and other factors, many of which are unknown. There seems to be no data analyzing how stress levels vary for the average healthy individual, over day-to-day activities. We would like to build a device that helps to gather and present data for improving an individual's understanding of both healthy and unhealthy stress in his or her life. The device itself should be comfortable and should not increase the user's stress. (It is noteworthy that stress monitoring is also important in human-computer interaction for testing new designs.) Currently, we are building a new, wireless, stress-mornitoring system by integrating Fitsense's heart-rate sensors and Motorola's iDen cell phone with our heart-rate-variability estimation algorithm.

Posture Recognition Chair
We have developed a system to recognize posture patterns and associated affective states in real time, in an unobtrusive way, from a set of pressure sensors on a chair. This system discriminates states of children in learning situations, such as when the child is interested, or is starting to take frequent breaks and looking bored. The system uses pattern recognition techniques, while watching natural behaviors, to "learn" what behaviors tend to accompany which states. The system thus detects the surface-level behaviors (postures) and their mappings during a learning situation in an unobtrusive manner so that they don't interfere with the natural learning process. Through the chair, we can reliably detect nine static postures, and four temporal patterns associated with affective states.

Recognizing Affect in Speech
This research project is concerned with building computational models for the automatic recognition of affective expression in speech. We are in the process of completing an investigation of how acoustic parameters extracted from the speech waveform (related to voice quality, intonation, loudness and rhythm) can help disambiguate the affect of the speaker without knowledge of the textual component of the linguistic message. We have carried out a multi-corpus investigation, which includes data from actors and spontaneous speech in English, and evaluated the model's performance. In particular, we have shown that the model exhibits a speaker-dependent performance which reflects human evaluation of these particular data sets, and, held against human recognition benchmarks, the model begins to perform competitively.

Relational Agents
Relational Agents are computational artifacts designed to build and maintain long-term, social-emotional relationships with their users. Central to the notion of relationship is that it is a persistent construct, spanning multiple interactions. Thus, Relational Agents are explicitly designed to remember past history and manage future expectations in their interactions with users. Since face-to-face conversation is the primary context of relationship-building for humans, our work focuses on Relational Agents as a specialized kind of embodied conversational agent (animated humanoid software agents that use speech, gaze, gesture, intonation, and other nonverbal modalities to emulate the experience of human face-to-face conversation). One major achievement was the development of a Relational Agent for health behavior change, specifically in the area of exercise adoption. A study involving 100 subjects interacting with this agent over one month demonstrated that the relational agent was respected more, liked more, and trusted more, and that these ratings were maintained over time (unlike for the non-relational agent, where they were not only significantly lower overall, but also declined over time.) People also expressed significantly greater ratings of perceived caring by the agent, and significantly more desire to keep working with the relational agent after the termination of the study.

Ripley: A Conversational Robot
Ripley is an interactive manipulator robot that uses spoken language and visual perception to interact with humans and its environment. It serves as a platform for investigating sensory-motor foundations of language, mental models for robots, and algorithms for multi-objective planning and active vision. This work has applications in human-robot interaction, design of interactive robots, and other intelligent systems.

TensorFaces: Facial Signatures
The goal of machine vision is automated image understanding and object recognition by a computer. Recent events have redoubled interest in biometrics and the application of computer vision technologies to non-obtrusive identification, surveillance, tracking, etc. Face recognition is a difficult problem for computers. This is due largely to the fact that images are the composite consequence of multiple factors relating to scene structure (i.e., the location and shapes of visible objects), illumination (i.e., the location and types of light sources), and imaging (i.e., viewpoint, viewing direction and camera characteristics). Multiple factors can confuse and mislead an automated recognition system. In addressing this problem, we take advantage of the assets of multilinear algebra, the algebra of higher-order tensors, to obtain a parsimonious representation that separates the various constituent factors. Our new representation of facial images, called TensorFaces, leads to improved recognition algorithms for use in the aforementioned applications.

TensorTexures: Multilinear Image-Based Rendering
An essential goal of computer graphics is photorealistic rendering, the synthesis of images of virtual scenes visually indistinguishable from those of natural scenes. Unlike traditional model-based rendering, whose photorealism is limited by model complexity, an emerging and highly active research area known as {/it image-based rendering} eschews complex geometric models in favor of representing scenes by ensembles of example images. These are used to render novel photoreal images of the scene from arbitrary viewpoints and illuminations, thus decoupling rendering from scene complexity. The challenge is to develop structured representations in high-dimensional image spaces that are rich enough to capture important information for synthesizing new images, including details such as self-occlusion, self-shadowing, interreflections, and subsurface scattering. TensorTextures, a new image-based texture mapping technique, is a rich generative model that, from a sparse set of example images, learns the interaction between viewpoint, illumination, and geometry that determines detailed surface appearance. Mathematically, TensorTextures is a nonlinear model of texture image ensembles that exploits tensor algebra and the N-mode SVD to learn a representation of the bidirectional texture function (BTF) in which the multiple constituent factors, or modes---viewpoints and illuminations---are disentangled and represented explicitly.

The Affective Remixer: Personalized Music Arranging
Affective Remixer is a real-time music-arranging system that reacts to immediate affective cues from a listener. Data was collected on the potential of certain musical dimensions to elicit change in a listener’s affective state using sound files created explicitly for the experiment through composition/production, segmentation, and re-assembly of music along these dimensions. Based on listener data, a probabilistic state transition model was developed to infer the listener’s current affective state. A second model was made that would select music segments and re-arrange (‘re-mix’) them to induce a target affective state.

The Conductor's Jacket
The Conductor's Jacket is a unique wearable device that measures physiological and gestural signals. Together with the Gesture Construction, a musical software system, it interprets these signals and applies them expressively in a musical context. Sixteen sensors have been incorporated into the Conductor's Jacket in such a way as to not encumber or interfere with the gestures of a working orchestra conductor. The Conductor's Jacket system gathers up to sixteen data channels reliably at rates of 3 kHz per channel, and also provides real-time graphical feedback. Unlike many gesture-sensing systems it not only gathers positional and accelerational data but also senses muscle tension from several locations on each arm. We will demonstrate the Gesture Construction, a musical software system that analyzes and performs music in real-time based on the performer's gestures and breathing signals. A bank of software filters extract several of the features that were found in the conductor study, including beat intensities and the alternation between arms. These features are then used to generate real-time expressive effects by shaping the beats, tempos, articulations, dynamics, and note lengths in a musical score.

Touch-Phone
The Touch-Phone was developed to explore the use of objects to mediate the emotional exchange in interpersonal communication. Through an abstract visualization of screen-based color changes, a standard telephone is modified to communicate how it is being held and squeezed. The telephone receiver includes a touch-sensitive surface which conveys the user's physical response over a computer network. The recipient sees a small colored icon on his computer screen which changes in real time according to the way his conversational partner is interacting with the telephone object.

Wearable Relational Devices for Stress Monitoring
We have created a system for data collection, annotation, and feedback that is part of a longer-term research interest to gather data to understand more about stress and the physiological signals involved in its expression. First, we built a wearable apparatus for gathering data that allows the user to include as many accurate labels (annotations) as possible while going about natural daily activities. Gathering annotations is disruptive and likely to increase stress (thus interfering with the signals being measured). We hypothesized that empathetic ways of interrupting would be less stressful than non-empathetic and found significant effects on many of user's self-reported items such as preference for the more empathetic system, and also on behavioral items, such as the estimated number of times they were interrupted (significantly lower when system was more empathetic.)