University of Twente Student Theses

Login
As of Friday, 8 August 2025, the current Student Theses repository is no longer available for thesis uploads. A new Student Theses repository will be available starting Friday, 15 August 2025.

Regrowing Strategies for Dynamic Sparse Training

Scholte, Harmen (2022) Regrowing Strategies for Dynamic Sparse Training.

[img] PDF
710kB
Abstract:Deep learning yields currently the most, practical results in the field of machine learning. There is however one big problem. Solving real-world, practical problems requires very large neural networks. Networks consisting of millions or even billions of parameters, are being used. The size of these large networks leads to impractical training times and high computation costs. People try to solve these problems by inducing sparsity into the networks, either by pruning (dense-to-sparse training), or by static or dynamic sparse-to-sparse training. Sparse Evolutionary Training (SET) introduced dynamic sparse-to-sparse training, and Adaptive Performance-based Connectivity of SET (AccSET) tries to improve on SET by focusing on the amount of connections that should be regrown. In this research, the regrow step is analyzed further. This research develops strategies regarding (1) which connections should be regrown, (2) on what performance metric should we base the connectivity of SET, and (3) does alternating between static and dynamic sparse training improve the performance. These strategies are tested on a multi-layer perceptron model that classifies images from the CIFAR-10 dataset.
Item Type:Essay (Bachelor)
Faculty:EEMCS: Electrical Engineering, Mathematics and Computer Science
Subject:54 computer science
Programme:Business & IT BSc (56066)
Link to this item:https://purl.utwente.nl/essays/91812
Export this item as:BibTeX
EndNote
HTML Citation
Reference Manager

 

Repository Staff Only: item control page