Analytical Properties of Model Performance Evaluation Using Predictive Power in Neuroscience

Abstract

The presence of external and internal noise is a ubiquitous challenge when modelling complex neurobiological processes. In particular, to assess the quality of a model, it is essential to evaluate its predictions in the context of noise, or more generally, uncon-trolled variance. A recently defined indicator, predictive power, provides model quality estimates corrected for the uncontrolled variance. We provide an analytic derivation of predictive power and explore its convergence properties and model dependence. We find that predictive power and its variance exhibit fast convergence as a function of the number of trials. Reliable results are achieved with both linear, semilinear (e.g. gen-eralized linear) and nonlinear models for Gaussian noise, although behaviour for other distributions is less consistent. Predictive power further exhibits a dependence on model dimension, which can be compensated by the combination of cross-validation and in-sample estimates. In summary, predictive power exhibits fast and reliable convergence for different models and noise-characteristics and thus provides a useful tool for the assessment of model quality across many disciplines in neuroscience and computational biology.