Reconstructing naturalistic movies from fMRI brain responses: Comparing motion-energy features with convolutional neural network representations
Keywords
Loading...
Authors
Issue Date
2019-08-01
Language
en
Document type
Journal Title
Journal ISSN
Volume Title
Publisher
Title
ISSN
Volume
Issue
Startpage
Endpage
DOI
Abstract
In 2011 Nishimoto et al. successfully managed to reconstruct naturalistic movies from
the brain activity of the visual cortex. In this study we repeat their experiment with
a bigger dataset containing 24 hours of densely sampled fMRI data of a single subject
watching a television series, overcoming the problem of having to adjust for a low
amount of data in multiple brains. Originally, a motion-energy model was used to
create features for the encoding model, which is a model of how motion is processed
in the early visual cortex. We compared the performance of motion-energy features to
the performance of an encoding model that uses features from a trained convolutional
neural network in order to cover more higher-order areas in the visual cortex. These
two types of features were also combined to create an encoding model selective to
both lower- and higher-order information. We showed that this combination performs
signi cantly better than the performance of the features separately. However, it must
also be concluded that the current method of reconstruction is not su cient to create
scenes that have the complexity that a regular television series displays, leaving room
for future research in methods to create more detailed reconstructions.
Description
Citation
Supervisor
Faculty
Faculteit der Sociale Wetenschappen