University of Twente Student Theses
As of Friday, 8 August 2025, the current Student Theses repository is no longer available for thesis uploads. A new Student Theses repository will be available starting Friday, 15 August 2025.
Regrowing Strategies for Dynamic Sparse Training
Scholte, Harmen (2022) Regrowing Strategies for Dynamic Sparse Training.
PDF
710kB |
Abstract: | Deep learning yields currently the most, practical results in the field of machine learning. There is however one big problem. Solving real-world, practical problems requires very large neural networks. Networks consisting of millions or even billions of parameters, are being used. The size of these large networks leads to impractical training times and high computation costs. People try to solve these problems by inducing sparsity into the networks, either by pruning (dense-to-sparse training), or by static or dynamic sparse-to-sparse training. Sparse Evolutionary Training (SET) introduced dynamic sparse-to-sparse training, and Adaptive Performance-based Connectivity of SET (AccSET) tries to improve on SET by focusing on the amount of connections that should be regrown. In this research, the regrow step is analyzed further. This research develops strategies regarding (1) which connections should be regrown, (2) on what performance metric should we base the connectivity of SET, and (3) does alternating between static and dynamic sparse training improve the performance. These strategies are tested on a multi-layer perceptron model that classifies images from the CIFAR-10 dataset. |
Item Type: | Essay (Bachelor) |
Faculty: | EEMCS: Electrical Engineering, Mathematics and Computer Science |
Subject: | 54 computer science |
Programme: | Business & IT BSc (56066) |
Link to this item: | https://purl.utwente.nl/essays/91812 |
Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Repository Staff Only: item control page