University of Twente Student Theses


Towards understanding and modelling sparse training algorithms at extreme sparsity regime

Engers, Vincent P.G. van (2021) Towards understanding and modelling sparse training algorithms at extreme sparsity regime.

[img] PDF
Abstract:Deep neural networks have proven useful in practical applications. However, many redundant connections unnecessarily inflate network size and computational complexity. Inspired by pruning in biological brains, sparse training methods such as Sparse Evolutionary Training (SET) and Accuracy based Sparse Evolutionary Training (AccSET) prove more efficient than fully connected counterparts by dynamically adding and removing connections. Contrasting to the great amount of research on deep neural networks, little research exists on the effect of varying hyperparameters and structure on the performance and behaviour of sparse neural networks. This paper investigates the influence of varying sparsity levels on the behaviour and performance of Sparse Evolutionary Training and provides new insights related to sparse training over a large sparsity horizon. This paper categorizes different levels of sparsity and defines the extreme sparse contour. It further delivers a systematic analysis of extremely sparse neural networks and a mathematical formulation of the relation between sparsity and accuracy which can estimate with great accuracy the expected accuracy of a model at a given sparsity level. The experiments in this research show that certain sparse neural networks can be trained at extreme sparsity levels. With these results, this paper contributes to the understanding of sparse training in artificial neural networks and underlines the importance of sparse neural networks to the implementation of machine learning in limited computing devices.
Item Type:Essay (Bachelor)
Faculty:EEMCS: Electrical Engineering, Mathematics and Computer Science
Subject:54 computer science
Programme:Business & IT BSc (56066)
Link to this item:
Export this item as:BibTeX
HTML Citation
Reference Manager


Repository Staff Only: item control page