University of Twente Student Theses
Survival of the Most Connected : An Understanding of How Sparse Neural Networks Grow
Matab, D.S.K.D. (2024) Survival of the Most Connected : An Understanding of How Sparse Neural Networks Grow.
This is the latest version of this item.
|
PDF
1MB |
| Abstract: | Artificial neural networks excel at complex tasks; however, scaling them increases parameters and computational power disproportionately, often without corresponding performance gains. Sparse neural networks, which have significantly fewer connections, address this issue. Dynamic Sparse Training facilitates the evolution of sparse networks through an iterative process of pruning and regrowing connections. While pruning strategies are well-studied, regrow strategies are less understood. We present an analysis of state-of-the-art regrow strategies, examining their effects on performance and network structure. Our study reveals that gradientbased regrow methods quickly create highly connected neurons, random regrow methods promote uniform connectivity, and momentum-based methods offer a balance between speed and stability. These findings highlight the trade-offs in speed and stability among different regrow strategies. By integrating performance metrics and network properties from initialization to post-training, our results provide valuable insights for designing efficient sparse neural network training procedures. This research enhances the sparse-to-sparse training paradigm and contributes to the understanding of sparse network dynamics, including connectivity patterns and neuron activation. Our findings benefit the fields of neural network optimization, network science, and machine learning. |
| Item Type: | Essay (Master) |
| Faculty: | EEMCS: Electrical Engineering, Mathematics and Computer Science |
| Subject: | 54 computer science |
| Programme: | Computer Science MSc (60300) |
| Link to this item: | https://purl.utwente.nl/essays/100461 |
| Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Show download statistics for this publication
Show download statistics for this publicationRepository Staff Only: item control page