University of Twente Student Theses
Fisher Information aware dynamic compression of Language Transformer networks using SVD
Hofstee, S.B.H.C. (2024) Fisher Information aware dynamic compression of Language Transformer networks using SVD.
|
PDF
419kB |
| Abstract: | We propose a novel approach to Singular Value Decomposition (SVD) for low-rank compression, addressing limitations in previous methods such as Fisher Weighted SVD (FWSVD) and True Fisher Weighted SVD (TFWSVD), which apply uniform compression ratios across layers without considering intra- and inter-layer information characteristics. Unlike FWSVD and TFWSVD, our approach dynamically determines layer-wise compression ratios based on these characteristics, enhancing task performance efficiency. Specifically, we explore three novel methods in which ranks are dynamically determined for low-rank compression based on the inter- or intra-layer Fisher Information (FI): (1) dynamically determining the rank for low-rank compression based on inter-layer Fisher Information (FI), (2) maintaining a fixed percentage of intra-layer FI and (3) optimizing to maximize total (or layer-wise) FI given a fixed overall compression ratio. These methods are evaluated on a transformer-based language model and benchmarked against the state-of-the-art. One of the proposed methods, relying on specifying a fixed percentage of FI to keep per layer, has been shown to outperform the current state of the art on average in excess of 5%, and very significantly for inference and similarity tasks. The work furthermore provides valuable insights for future work to further explore the dynamic compression of layers in transformer networks using FI, in particular by displaying the effectiveness of dynamic compression using intra-layer FI. |
| Item Type: | Essay (Master) |
| Faculty: | EEMCS: Electrical Engineering, Mathematics and Computer Science |
| Subject: | 54 computer science |
| Programme: | Computer Science MSc (60300) |
| Link to this item: | https://purl.utwente.nl/essays/101837 |
| Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Show download statistics for this publication
Show download statistics for this publicationRepository Staff Only: item control page