A Deep Learning Model of Perception based on Color-Grapheme Synesthesia
Keywords
Loading...
Authors
Issue Date
2020-07-10
Language
en
Document type
Journal Title
Journal ISSN
Volume Title
Publisher
Title
ISSN
Volume
Issue
Startpage
Endpage
DOI
Abstract
Synesthesia is a well-known perceptual effect of cross-modal interaction within the brain. It is defined as a perceptual phenomenon in which stimulation of one sensory or cognitive pathway leads to involuntary experiences in a second sensory or cognitive pathway (Cytowic et al., 2009). Synesthesia is a rare condition, occurring in 2%-4% of the population (Sinner et al., 2006); of which 66.4% is color-grapheme synesthesia (Safran et al., 2015). This thesis is inspired by the theory of local cross-activation, the findings presented in fMRI studies conducted by Hubbard & Ramanchandran into color grapheme synesthesia and by Witthoft et al.’ supported hypothesis stating that synesthesia can be a learned form of mental imagery. Using these inspirations this thesis studies cross-modal perception through the prism of deep learning modeling. Here, we describe and compare two neural network architectures (deep Convolutional Neural Network based on auto-coloring principles as presented by Agrawal, M., & Sawhney, K. (2016) and deep Generative Adversarial Neural Networks as proposed by Bock, J. R., (2018)) under the framework of color-grapheme synesthesia. To evaluate the performance of the networks both network implementations will be compared to each other based on their performance and color discrepancies when comparing average HSV-color distance metrics of network generated images to test images. Results show that the proposed models are capable of reproducing the color-letter associations as seen in perception produces by synesthetic brains. In so doing this work contributes a duo of powerful machine learning tools to the state of the art, similar to local cross activation models based in color-grapheme synesthesia. This work potentially opens the door for future research into the integration of similar computational approaches to for example VR synesthetic experiences, in an attempt to make synesthesia more tangible and approachable for the remaining 96%-98% of researchers and public not capable of experiencing synesthesia.
Description
Citation
Supervisor
Faculty
Faculteit der Sociale Wetenschappen