On the Reliability of the Uncertainty Quanti ed by a Convolutional Bayesian Neural Network
Keywords
Loading...
Authors
Issue Date
2021-06-18
Language
en
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Deep learning (DL) is achieving a lot of breakthroughs in di erent elds, such as object
detection, segmentation, and recognition [8, 15, 20]. However, DL still fails to reason about
its decisions and hence is not widely used in safety critical applications yet [1]. Integrating
Bayesian statistics with DL, e.g. Bayesian Neural Networks (BNN), provides versatility and
the reasoning ability through decision con dence for DL. BNNs o er to accompany each
decision taken with an uncertainty quanti cation metric that can be decomposed into two
kinds of uncertainty: aleatoric and epistemic. Aleatoric uncertainty being the uncertainty
caused due to inherent noise in the data, and epistemic uncertainty being the uncertainty
due to improper model or lack of knowledge, i.e. lack of training data. A considerable
amount of research underlies what these uncertainties capture and how they can be interpreted,
however, not much on their reliability. Therefore, the research question is: How
reliable is the uncertainty quanti ed by a Bayesian Lenet[17], a convolutional bayesian neu-
ral network (CBNN) trained using Variational Inference?
To answer the research question, seven hypotheses are proposed capturing the expected
behaviour of the prediction accuracy, aleatoric uncertainty, and epistemic uncertainty, all
quanti ed by the Bayesian Lenet using two di erent parameter sampling techniques, under
small and large data shifts. In particular, small data shifts are approached by adding noise
following Gaussian and Poisson distributions, or masking out parts of the inputs. Large data
shifts are approached by passing data from a di erent dataset, e.g., training the network on
recognizing digits and then passing alphabets or pictures of objects to it. Accordingly, the
network's uncertainty is quanti ed, and visualized using Layer Wise Propagation approach.
As a result of this study, a pipeline towards qualitatively understanding and visualizing the
uncertainty quanti ed by the Bayesian Lenet model is constructed. This pipeline can be
further used to explore the reliability of di erent uncertainty quanti cation methods and
network architectures.
The hypotheses were qualitatively evaluated on two datasets, namely, MNIST and CIFAR10.
Overall, results show that the uncertainty quanti cation method used in this work
is able to detect small and large data shifts. However, out of the seven hypotheses, only one
hypothesis holds, four partially hold, and two do not hold. It is observed that the reliability
of the uncertainty quanti cation methods used is dependent on many factors, such as, the
model's achieved training accuracy, the model parameters' sampling technique, dataset, and
noise type.
Description
Citation
Supervisor
Faculty
Faculteit der Sociale Wetenschappen