Synesthesia-inspired cross-modal learning of common representation using GANs
Keywords
No Thumbnail Available
Authors
Issue Date
2020-07-10
Language
en
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Synesthesia is a phenomenon in which the stimulation of one sensory
modality simultaneously leads to the sensation in one another. A well-known
type of synesthesia is grapheme-color synesthesia, i.e., letters and digits are
consistently associated with speci c colors. Understanding the way crossmodal
perception in synesthesia works has broadened the research in the eld
of arti cial intelligence (AI) and its applications in dealing with multimodal
data. Here, we describe a novel application of the cross-modal generative
adversarial networks (CM-GANs) approach in order to learn the cross-modal
common representation enforced by the shared semantic classes between the
visual letter grapheme modality and the color modality, as in grapheme-color
synesthesia. In order to evaluate the e ectiveness of the model, we perform
two cross-modal retrieval tasks: bi-modal retrieval (i.e., retrieving the correct
matching color instances using letters as queries) and all-modal retrieval
(i.e., retrieving the correct matching letter and color instances using letters as
queries). The experimental results, obtained from the cross-modal retrieval
tasks, are shown to be relatively high, indicating that the shared semantics
between two modalities have a cross-modal e ect in common representation
learning. Regarding multimodal representation learning, we investigate the effectiveness
of the CM-GANs network and discuss the approaches to overcome
its shortcomings. As for grapheme-color synesthesia, we assess the applicability
of the model in mathematically modeling the cross-modal perceptual
association experience.
Description
Citation
Supervisor
Faculty
Faculteit der Sociale Wetenschappen