Demystifying Node Pruning through the Perspective of Subspaces
Keywords
Loading...
Authors
Issue Date
2024-08-01
Language
en
Document type
Journal Title
Journal ISSN
Volume Title
Publisher
Title
ISSN
Volume
Issue
Startpage
Endpage
DOI
Abstract
A significant increase in the commercial use of deep neural network models increases the
need for efficient AI. Node pruning is the art of removing computational nodes such as
neurons, filters, attention heads, or even entire layers, such that the network performance
is kept at a maximum after retraining. This can significantly reduce the inference time of
a deep network and thus enhance its efficiency. Previous works have exploited the ability
to recover performance by reorganizing network parameters while pruning via linear
least squares approximation of the original neural responses. In this work, we relate
this method to projecting node activity to an orthogonal subspace, such that pruning
within the subspace translates to effective node pruning. We find that this reconstruction
barely affects network performance compared to simply pruning units after retraining.
However, we use these findings to develop novel layer-wise importance scoring methods
and the relation of such layer-wise rankings across layers within a network to formulate
a measure of global unit importance. Through application to VGG-16 trained on the
ILSVRC dataset, we show that this global unit importance outperforms the baseline
methods tested herein.
Description
Citation
Supervisor
Faculty
Faculteit der Sociale Wetenschappen