{"title":"Dynamic Error-Bounded Hierarchical Matrices in Neural Network Compression","authors":"John Mango, Ronald Katende","doi":"arxiv-2409.07028","DOIUrl":null,"url":null,"abstract":"This paper presents an innovative framework that integrates hierarchical\nmatrix (H-matrix) compression techniques into the structure and training of\nPhysics-Informed Neural Networks (PINNs). By leveraging the low-rank properties\nof matrix sub-blocks, the proposed dynamic, error-bounded H-matrix compression\nmethod significantly reduces computational complexity and storage requirements\nwithout compromising accuracy. This approach is rigorously compared to\ntraditional compression techniques, such as Singular Value Decomposition (SVD),\npruning, and quantization, demonstrating superior performance, particularly in\nmaintaining the Neural Tangent Kernel (NTK) properties critical for the\nstability and convergence of neural networks. The findings reveal that H-matrix\ncompression not only enhances training efficiency but also ensures the\nscalability and robustness of PINNs for complex, large-scale applications in\nphysics-based modeling. This work offers a substantial contribution to the\noptimization of deep learning models, paving the way for more efficient and\npractical implementations of PINNs in real-world scenarios.","PeriodicalId":501162,"journal":{"name":"arXiv - MATH - Numerical Analysis","volume":"10 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-11","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - MATH - Numerical Analysis","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.07028","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
This paper presents an innovative framework that integrates hierarchical
matrix (H-matrix) compression techniques into the structure and training of
Physics-Informed Neural Networks (PINNs). By leveraging the low-rank properties
of matrix sub-blocks, the proposed dynamic, error-bounded H-matrix compression
method significantly reduces computational complexity and storage requirements
without compromising accuracy. This approach is rigorously compared to
traditional compression techniques, such as Singular Value Decomposition (SVD),
pruning, and quantization, demonstrating superior performance, particularly in
maintaining the Neural Tangent Kernel (NTK) properties critical for the
stability and convergence of neural networks. The findings reveal that H-matrix
compression not only enhances training efficiency but also ensures the
scalability and robustness of PINNs for complex, large-scale applications in
physics-based modeling. This work offers a substantial contribution to the
optimization of deep learning models, paving the way for more efficient and
practical implementations of PINNs in real-world scenarios.