next up previous
Next: Technology, Interface, and Human Up: Affective Wearables Previous: Prototype of an Affective

Challenges of Ambulatory Affect Sensing

A wearable computer with bio-metric sensors, in constant contact with its user, offers an opportunity to obtain unprecedented amounts of physical and physiological data about a single user. This advantage also presents many challenges. To use the data effectively in learning algorithms, and to interpret the results, the data must be accurately labeled. We are in the early stages of developing ways to record the physical and psychological events of the wearer's life without interrupting their normal activities. Currently, an automatically time stamped notes file allows the user to annotate data from the sensor screen. Adding an audio recording device or video camera would improve the users ability to annotate data after the fact, but would make the annotation cumbersome. We propose to add two sensors to the system to improve annotation: a foot pressure sensor to indicate ambulatory movement and an audio detector that would detect if the wearer was making any vocal sounds (speech, sneezing, etc.) The latter could record and analyze the audio if privacy was not an issue, or if privacy was an issue, it would not record voice, but only use features of the sound signal to prompt the user for annotations, e.g. computer detects speech and prompts the user ``Did you just talk to someone?'' to which the wearer could respond ``Talked to Jane Smith.''

It is important to annotate the physical activities of the user because such physical activities can overwhelm the physiological effect of psychological events. Even under ``ideal'' laboratory conditions, with a resting subject responding to a directed stimulus, scientists have yet to find reliably discernible features of physiological signals for affect detection. In a natural environment where weather, diet, and physical activity are present, the patterns corresponding to particular affective states are at a much greater risk of being obscured. We must identify and account for these confounding variables.

In psychological studies by Lang [LGea93] and Winton, Putnam and Krauss [WPK84], subjects were monitored while looking at a series of photographs which were supposed to elicit an emotional response. These studies showed that heart rate variability was an indication of valence (whether or not the person found the photograph pleasant) and that the ratio of skin conductance to heart rate variability was an indicator of arousal. However, the greatest changes in skin conductance and heart rate in their study were 0.6 micro-Siemens, and 8 beats per minute respectively, in the ten second period after viewing the slides. These results are significant for a resting subject under controlled laboratory conditions, but we have found that such results can be overwhelmed by physical activity in an ambulatory subject.

We conducted experiments in the laboratory to show that the physical activity of ambulatory subjects is an important confounding variable to detecting emotional responses from bio-sensors. Five different subjects were run through the ambulatory bio-sensing task, the results of which are reported in Table 1. The experimental protocol involved subjects wearing the bio-sensors from the FlexComp system while performing a series of ambulatory tasks. The subjects were first shown how to attach the bio-sensors. In this experiment all subjects wore a respiration sensor, BVP, a GSR sensor on the hand and a GSR sensor on the arch of the foot. They were then asked to perform the following tasks: sit in a chair for one minute, resting quietly, stand up and sit down twice, walk around the room for one minute, sit in a chair normally for two minutes, stand up and walk around the room for one minute, sit normally for another minute, jog in place for one minute, then sit for a minute and finally the subjects were asked to cough as if they had a cold twice. An experimenter was in the room with the subjects at all times during the experiment to instruct the subjects to perform the activities. Subjects sometimes had questions during the experiment, and talking was noted in the experimental record. The rest times between the activities sometimes varied, due to questions, so the beginning of the tasks were marked by the experimenter with a specially designed mouse equipped with a sensor which would make a spike in the recording data when depressed.

The data for this experiment, was saved at 16 samples per second using the FlexComp, with a resolution of 0.01 micro-Siemens. The change in the skin conductivity (GSR) was calculated as the difference between the baseline measurement, taken at the time each task was begun, and the first significant local maximum. The heuristic used for determining the significant local maxia, is that the difference between the local maximum and the following local minimum is greater than 0.5 micro-Siemens, although this number should be scaled with regard to the overall variance of the signal. The heart rate was calculated from the peak-to-peak intervals and stored at 16 samples per second. To calculate the change in heart rate the average of the heart rate for 10 seconds before the task was subtracted from the average for the heart rate ten seconds following the task. An example of data from the experiments is shown in Figure 2.

Table: A wearer's activities can cause large changes in physiological signals. These changes need to be understood so that they can be taken into account by the system trying to recognize affect.
Activity Increase in HR (bpm)
  S1 S2 S3 S4 S5
stand 16.0 15.8 19.7 22.0 15.2
walk 26.4 18.9 27.7 24.6 19.9
jog 68.2 60.8 74.0 80.9 87.4
cough 22.0 22.2 18.8 53.7 14.66
Activity GSR change for Hand (micro-Siemens)
  S1 S2 S3 S4 S5
stand 0.5 10.6 0.2 1.7 N/A
walk 2.0 14.3 0.8 2.8 N/A
jog 5.9 16.1 3.5 2.2 N/A
cough 5.4 11.0 2.1 2.7 N/A
Activity GSR change for Foot Arch (micro-Siemens)
  S1 S2 S3 S4 S5
stand 0.5 12.6 0.3 4.1 3.7
walk 1.5 10.4 1.7 6.9 3.4
jog 11.5 11.0 5.0 5.0 6.4
cough 6.9 9.9 2.4 5.9 3.2


Figure: An example of data collected from the ambulatory experiment. Data is scaled and offset for show the relationships between the signals.

It is therefore necessary to have an accurate recording of the physical activity of the wearer and to understand how this affects the user's physiology, and expression of affect. From this information we can attempt to decouple the psychological effects from the physical. For example, although the heart rate of the wearer changes drastically with inhalation and exhalation, the heart rate and respiration signals are highly correlated as shown in Figure 3. By recording respiration, we can state which heart rate changes are effected by this source and which changes must be attributed to other sources.

Figure: The heart rate and respiration signals from the ambulatory experiment, showing that although respiration has a large effect on heart rate, these effects are well correlated with the respiration signal and can be predicted.

The selection of which signals to measure is another challenge for an affective wearable. There is evidence to believe that physiological signals including accelerated heart rate [Lev92], raised blood pressure, increased skin conductivity, and constriction of the peripheral blood vessels [Hel78] are associated with stress in humans and heart rate variability [WPK84], pupillary dilation and right and left hemispheric activation in the brain [Dav93] are associated with like/dislike.

We are experimenting to determine which physiological signals can be best monitored under ambulatory conditions to produce the most salient affective features with sensors that are comfortable to wear. Such a system would be unobtrusive, lightweight, easy to use and does not require the user to make exceptions to his daily routine. Our goal is to create a sensor system which is comfortable and robust which can extract complex features in both the time and frequency domain and record annotations and context information.

The GSR sensor is one of the most robust wearable sensors because it is a bulk measurement and not very sensitive to exact placement of the sensor. Traditionally, this measurement is taken across the palm of the hand, however, this is one of the least ideal places for sensor placement. Ordinary activities such as washing your hands changes the baseline reading and repetitive motion of the fingers in activities like typing, creates a noise signal that confounds the signal. To alleviate this problem, we have explored the option of placing the electrodes on the toes and measuring skin conductivity across the sole of the foot.

In this experiment, another five subjects were seated and startled with five white noise bursts while wearing skin conductivity sensors on three locations. The sensors were placed on the two middle sections of the first and second finger on the dominant hand, on the two middle sections of the first and second toes on the same foot, and on the either end of the arch of the same foot. An electret microphone was used to record the startling tone burst as a reference. All skin conductivity data was sampled and saved at 16 samples per second with 0.01 micro-Siemens resolution. An example of the data recorded during these experiments is shown in Figure 4.

Figure: An example of data collected from the multiple startle experiment. This experiment was designed to test the correlation of the hand and the foot responses under extreme conditions.

Figure: The shoe provides a convenient location for sensor placement. Here, a skin conductivity sensor is placed in the arch of the shoe and a pressure sensitive resistor is placed on the heel. Sensors look unobtrusive when worn.(Photograph courtesy of Fernando Padilla)

Table: The similarity between skin conductivity readings measured from the palm of the hand and the arch of the foot offers the possibility of placing sensors in the wearer's shoe instead of on the hand.
Activity Correlation Coefficients for GSR
  S1 S2 S3 S4 S5
hand-toes .93 .90 .97 .53 .86
hand-arch .86 .86 .99 .95 .81
arch-toes .89 .96 .98 .57 .86


The startle experiment was designed to test the correlation of the hand and foot responses under extreme audio stimuli. The correlation coefficient was used to measure similarity between the signals taken from the three locations. The results of these correlations are reported in Table 2. In the natural sciences, the correlation coefficient is used in the Pearson correlation to test significance of traits, based on the number of data points. If we consider all samples of the signal data points for this test, all the results are extremely significant with p>0.01, however if we consider only some subset of the points e.g., those that represent the magnitude of the startle, then the results are only significant for three of the five subjects, with the most significant result corresponding to the subject who was least susceptible to startle. Therefore, the correlation coefficients themselves are reported.

We also found strong correlations between the hand and foot responses in the Ambulatory study with a correlation coefficients of .90, .85, .83 and .88 for the first four subjects respectively (the fifth subject detached the hand GSR during the experiment). From these results we concluded that the skin conductivity responses of the hand and the foot were highly correlated. Therefore, either placement of the sensor could be used to measure the GSR response.

The blood volume pressure sensor uses photoplethysmography, a technique where an LED is used to ``look'' at the amount of blood flowing in the vessel. From this reading both the heart beat and constriction of the blood vessel can be determined. This sensor is sensitive to correct placement and motion artifacts; however, in a single-subject experiment we conducted driving through Boston at rush hour, where the wearer moved the hand with the sensors routinely to shift gears, adjust the radio, and turn the steering wheel, we found that although the signal was disturbed during motion, the signal re-stabilized after movements. Nonetheless, the wires felt like they were in the way. We are interested in moving them off the hands to sense what people are feeling during certain consumer tasks such as diapering an infant. This is a particularly challenging task as it is important not to drag the wires through the dirty diaper, or let the infant grab them and pull them, or put them in his mouth. In other words, we are investigating options for moving the sensors off the hands.

Respiration is the easiest sensor to wear and is the most immune from motion artifacts. It uses a constrained Hall effect sensor to measure the expansion and contraction of the chest cavity. The confounding variables of this measurement mostly audio (e.g. talking, sneezing, coughing and sighing) all of which we hope to compensate for with the audio detector.

We have been investigating the optimal placement for the EMG sensor to detect stress. The greatest challenge of EMG placement is to find a muscle to which the EMG electrode pad will stick well and which is not involved in muscle motion. To solve the problem of affixing the sensors we are currently considering the use of new electrodes and glue designed by the Boston University Neuro-muscular Research Center which have been demonstrated to stay affixed through rigorous physical activity. To solve the problem of motion we may need to create sensors which detect motion and use this to correct the straight EMG reading.

Figure: PIC chips on iRx boards can be used to sample data from sensors and transmit the digital signal using infrared. Small self-contained sensor systems such as this allow affective information to be wirelessly transmitted to a larger computer for analysis. (Photograph courtesy of Frank Dabek).

In the future we hope to create a wireless interface for all of these sensors. We have currently designed working prototypes of skin conductance and respiration sensors which transmit data via an IR signal using an iRx PIC series chip. This data is collected from multiple receivers in a room (See Figure 6).

next up previous
Next: Technology, Interface, and Human Up: Affective Wearables Previous: Prototype of an Affective
Jennifer Healey -