Analytical Properties of Model Performance Evaluation Using Predictive Power in Neuroscience
Keywords
No Thumbnail Available
Authors
Issue Date
2017-10-24
Language
en
Document type
Journal Title
Journal ISSN
Volume Title
Publisher
Title
ISSN
Volume
Issue
Startpage
Endpage
DOI
Abstract
The presence of external and internal noise is a ubiquitous challenge when modelling complex neurobiological processes. In particular, to assess the quality of a model, it is essential to evaluate its predictions in the context of noise, or more generally, uncon-trolled variance. A recently defined indicator, predictive power, provides model quality estimates corrected for the uncontrolled variance. We provide an analytic derivation of predictive power and explore its convergence properties and model dependence. We find that predictive power and its variance exhibit fast convergence as a function of the number of trials. Reliable results are achieved with both linear, semilinear (e.g. gen-eralized linear) and nonlinear models for Gaussian noise, although behaviour for other distributions is less consistent. Predictive power further exhibits a dependence on model dimension, which can be compensated by the combination of cross-validation and in-sample estimates. In summary, predictive power exhibits fast and reliable convergence for different models and noise-characteristics and thus provides a useful tool for the assessment of model quality across many disciplines in neuroscience and computational biology.
Description
Citation
Supervisor
Faculty
Faculteit der Sociale Wetenschappen