University of Twente Student Theses

Login

Exploring the effect of merging techniques on the performance of merged sparse neural networks in a highly distributed setting

Steerneman, E.H. (2022) Exploring the effect of merging techniques on the performance of merged sparse neural networks in a highly distributed setting.

[img] PDF
5MB
Abstract:Sparse neural networks can significantly reduce the number of parameters of dense neural networks, thereby memory usage and computational costs. To keep the memory usage and computational costs of sparse neural networks low, these should be sparse throughout the training process. This can be accomplished by algorithms such as SET, which train sparse neural networks from scratch. This evolution prevents the use of conventional parallelization training techniques in decentralized settings. This research evaluates techniques that may enable the training of sparse neural networks in a parallel decentralized setting. Related work suggests that merging sparse neural networks should boost performance due to the bias-variance tradeoff. Evaluation of these techniques shows that merging sparse neural networks based on the magnitude of their parameters gives the best results. Under the circumstances of this research, sparse neural networks have been successfully merged both with and without incurring loss in performance. These results, combined with the SET algorithm, strengthen the idea that parameter magnitude is an essential factor in sparse neural networks. Using these techniques, conventional parallelization techniques can once again be applied. This research provides a basis for merging sparse neural networks with different structures.
Item Type:Essay (Master)
Faculty:EEMCS: Electrical Engineering, Mathematics and Computer Science
Subject:54 computer science
Programme:Computer Science MSc (60300)
Link to this item:https://purl.utwente.nl/essays/91995
Export this item as:BibTeX
EndNote
HTML Citation
Reference Manager

 

Repository Staff Only: item control page