Communicating Emotions: Multi-modal emotion recognition and the experience of emotional intensity
Keywords
No Thumbnail Available
Authors
Issue Date
2011-07-13
Language
en
Document type
Journal Title
Journal ISSN
Volume Title
Publisher
Title
ISSN
Volume
Issue
Startpage
Endpage
DOI
Abstract
In the past years various studies have focussed on automated emotion recognition using various
modalities (video, audio, physiology) separately. The work we present comprises an investigation
of the usefulness of each of these separate modalities and various modality combinations, for
automated emotion recognition. Video, audio, and physiology signals were recorded from persons
telling about an emotional event from their life, for five separate emotions: happy, relaxed, sad,
angry and neutral. Features were extracted for each modality separately. The data from the
three modalities was merged using feature-level fusion. Classification was done for each modality
separately and for every modality combination, using a Support-Vector Machine (SVM) and a
MultiLayer Perceptron (MLP). When looking at the classification performances for each modality
separately, physiology showed promising results, especially for the SVM classifier. For both
classifiers, there was an overall improvement in classification performance when modalities were
merged instead of used separately.
For automated emotion recognition, physiology appeared to be a quite influential modality.
In order to investigate whether this could also be the case in human-human interaction, we
conducted an experiment on the effect of a physiological signal on the judgment of the intensity of
someone's emotional state. Participants were shown videos of persons telling about an emotional
event (angry or neutral) from their life. Each video was accompanied by a heartbeat sound
(slow or fast). Participants were told that the heartbeats were those of the persons in the videos
at the moment they were telling the story. The participants were given the task to judge the
emotional intensity of the persons the videos. Results showed that both faster heart rates and
angry facial expressions caused higher emotional intensity ratings when compared to slower heart
rates and neutral facial expressions. These results show the potential for a whole new way of
communicating emotions.
Description
Citation
Supervisor
Faculty
Faculteit der Sociale Wetenschappen