University of Twente Student Theses


Regrowing Strategies for Dynamic Sparse Training

Scholte, Harmen (2022) Regrowing Strategies for Dynamic Sparse Training.

[img] PDF
Abstract:Deep learning yields currently the most, practical results in the field of machine learning. There is however one big problem. Solving real-world, practical problems requires very large neural networks. Networks consisting of millions or even billions of parameters, are being used. The size of these large networks leads to impractical training times and high computation costs. People try to solve these problems by inducing sparsity into the networks, either by pruning (dense-to-sparse training), or by static or dynamic sparse-to-sparse training. Sparse Evolutionary Training (SET) introduced dynamic sparse-to-sparse training, and Adaptive Performance-based Connectivity of SET (AccSET) tries to improve on SET by focusing on the amount of connections that should be regrown. In this research, the regrow step is analyzed further. This research develops strategies regarding (1) which connections should be regrown, (2) on what performance metric should we base the connectivity of SET, and (3) does alternating between static and dynamic sparse training improve the performance. These strategies are tested on a multi-layer perceptron model that classifies images from the CIFAR-10 dataset.
Item Type:Essay (Bachelor)
Faculty:EEMCS: Electrical Engineering, Mathematics and Computer Science
Subject:54 computer science
Programme:Business & IT BSc (56066)
Link to this item:
Export this item as:BibTeX
HTML Citation
Reference Manager


Repository Staff Only: item control page