Reconstructing naturalistic movies from fMRI brain responses: Comparing motion-energy features with convolutional neural network representations

Abstract

In 2011 Nishimoto et al. successfully managed to reconstruct naturalistic movies from the brain activity of the visual cortex. In this study we repeat their experiment with a bigger dataset containing 24 hours of densely sampled fMRI data of a single subject watching a television series, overcoming the problem of having to adjust for a low amount of data in multiple brains. Originally, a motion-energy model was used to create features for the encoding model, which is a model of how motion is processed in the early visual cortex. We compared the performance of motion-energy features to the performance of an encoding model that uses features from a trained convolutional neural network in order to cover more higher-order areas in the visual cortex. These two types of features were also combined to create an encoding model selective to both lower- and higher-order information. We showed that this combination performs signi cantly better than the performance of the features separately. However, it must also be concluded that the current method of reconstruction is not su cient to create scenes that have the complexity that a regular television series displays, leaving room for future research in methods to create more detailed reconstructions.