Design and Impact of Activation Functions for Sparse Neural Networks

Currently, the optimizer, activation function, and regularizer of Dense Neural Networks (DNN) are most commonly utilized in the training process of Sparse Neural Networks (SNN), but it has been demonstrated that the default activation functions used for DNN can disadvantage SNN. For SNN, structural design and training should be revisited. % Our goal is to fully comprehend the SNN's nature, assess its activation function, and build a more suitable activation function for the SNN.