Neural NetworksPub Date : 2025-04-01Epub Date: 2024-12-31DOI: 10.1016/j.neunet.2024.107098
Zhongyuan Lu, Jin Liu, Miaozhong Xu
{"title":"Identity Model Transformation for boosting performance and efficiency in object detection network.","authors":"Zhongyuan Lu, Jin Liu, Miaozhong Xu","doi":"10.1016/j.neunet.2024.107098","DOIUrl":"10.1016/j.neunet.2024.107098","url":null,"abstract":"<p><p>Modifying the structure of an existing network is a common method to further improve the performance of the network. However, modifying some layers in network often results in pre-trained weight mismatch, and fine-tune process is time-consuming and resource-inefficient. To address this issue, we propose a novel technique called Identity Model Transformation (IMT), which keep the output before and after transformation in an equal form by rigorous algebraic transformations. This approach ensures the preservation of the original model's performance when modifying layers. Additionally, IMT significantly reduces the total training time required to achieve optimal results while further enhancing network performance. IMT has established a bridge for rapid transformation between model architectures, enabling a model to quickly perform analytic continuation and derive a family of tree-like models with better performance. This model family possesses a greater potential for optimization improvements compared to a single model. Extensive experiments across various object detection tasks validated the effectiveness and efficiency of our proposed IMT solution, which saved 94.76% time in fine-tuning the basic model YOLOv4-Rot on DOTA 1.5 dataset, and by using the IMT method, we saw stable performance improvements of 9.89%, 6.94%, 2.36%, and 4.86% on the four datasets: AI-TOD, DOTA1.5, coco2017, and MRSAText, respectively.</p>","PeriodicalId":49763,"journal":{"name":"Neural Networks","volume":"184 ","pages":"107098"},"PeriodicalIF":6.0,"publicationDate":"2025-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142957832","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Neural NetworksPub Date : 2025-04-01Epub Date: 2025-01-03DOI: 10.1016/j.neunet.2024.107113
Varun Kumar, Somdatta Goswami, Katiana Kontolati, Michael D Shields, George Em Karniadakis
{"title":"Synergistic learning with multi-task DeepONet for efficient PDE problem solving.","authors":"Varun Kumar, Somdatta Goswami, Katiana Kontolati, Michael D Shields, George Em Karniadakis","doi":"10.1016/j.neunet.2024.107113","DOIUrl":"10.1016/j.neunet.2024.107113","url":null,"abstract":"<p><p>Multi-task learning (MTL) is an inductive transfer mechanism designed to leverage useful information from multiple tasks to improve generalization performance compared to single-task learning. It has been extensively explored in traditional machine learning to address issues such as data sparsity and overfitting in neural networks. In this work, we apply MTL to problems in science and engineering governed by partial differential equations (PDEs). However, implementing MTL in this context is complex, as it requires task-specific modifications to accommodate various scenarios representing different physical processes. To this end, we present a multi-task deep operator network (MT-DeepONet) to learn solutions across various functional forms of source terms in a PDE and multiple geometries in a single concurrent training session. We introduce modifications in the branch network of the vanilla DeepONet to account for various functional forms of a parameterized coefficient in a PDE. Additionally, we handle parameterized geometries by introducing a binary mask in the branch network and incorporating it into the loss term to improve convergence and generalization to new geometry tasks. Our approach is demonstrated on three benchmark problems: (1) learning different functional forms of the source term in the Fisher equation; (2) learning multiple geometries in a 2D Darcy Flow problem and showcasing better transfer learning capabilities to new geometries; and (3) learning 3D parameterized geometries for a heat transfer problem and demonstrate the ability to predict on new but similar geometries. Our MT-DeepONet framework offers a novel approach to solving PDE problems in engineering and science under a unified umbrella based on synergistic learning that reduces the overall training cost for neural operators.</p>","PeriodicalId":49763,"journal":{"name":"Neural Networks","volume":"184 ","pages":"107113"},"PeriodicalIF":6.0,"publicationDate":"2025-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142967318","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Neural NetworksPub Date : 2025-04-01Epub Date: 2024-12-31DOI: 10.1016/j.neunet.2024.107071
Azadeh Faroughi, Parham Moradi, Mahdi Jalili
{"title":"Enhancing Recommender Systems through Imputation and Social-Aware Graph Convolutional Neural Network.","authors":"Azadeh Faroughi, Parham Moradi, Mahdi Jalili","doi":"10.1016/j.neunet.2024.107071","DOIUrl":"10.1016/j.neunet.2024.107071","url":null,"abstract":"<p><p>Recommendation systems are vital tools for helping users discover content that suits their interests. Collaborative filtering methods are one of the techniques employed for analyzing interactions between users and items, which are typically stored in a sparse matrix. This inherent sparsity poses a challenge because it necessitates accurately and effectively filling in these gaps to provide users with meaningful and personalized recommendations. Our solution addresses sparsity in recommendations by incorporating diverse data sources, including trust statements and an imputation graph. The trust graph captures user relationships and trust levels, working in conjunction with an imputation graph, which is constructed by estimating the missing rates of each user based on the user-item matrix using the average rates of the most similar users. Combined with the user-item rating graph, an attention mechanism fine tunes the influence of these graphs, resulting in more personalized and effective recommendations. Our method consistently outperforms state-of-the-art recommenders in real-world dataset evaluations, underscoring its potential to strengthen recommendation systems and mitigate sparsity challenges.</p>","PeriodicalId":49763,"journal":{"name":"Neural Networks","volume":"184 ","pages":"107071"},"PeriodicalIF":6.0,"publicationDate":"2025-04-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"142967247","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Neural NetworksPub Date : 2025-03-31DOI: 10.1016/j.neunet.2025.107419
Junwu Wang , Jiansong Deng , Dong Liu
{"title":"Deep prior embedding method for Electrical Impedance Tomography","authors":"Junwu Wang , Jiansong Deng , Dong Liu","doi":"10.1016/j.neunet.2025.107419","DOIUrl":"10.1016/j.neunet.2025.107419","url":null,"abstract":"<div><div>This paper presents a novel deep learning-based approach for Electrical Impedance Tomography (EIT) reconstruction that effectively integrates image priors to enhance reconstruction quality. Traditional neural network methods often rely on random initialization, which may not fully exploit available prior information. Our method addresses this by using image priors to guide the initialization of the neural network, allowing for a more informed starting point and better utilization of prior knowledge throughout the reconstruction process. We explore three different strategies for embedding prior information: non-prior embedding, implicit prior embedding, and full prior embedding. Through simulations and experimental studies, we demonstrate that the incorporation of accurate image priors significantly improves the fidelity of the reconstructed conductivity distribution. The method is robust across varying levels of noise in the measurement data, and the quality of the reconstruction is notably higher when the prior closely resembles the true distribution. This work highlights the importance of leveraging prior information in EIT and provides a framework that could be extended to other inverse problems where prior knowledge is available.</div></div>","PeriodicalId":49763,"journal":{"name":"Neural Networks","volume":"188 ","pages":"Article 107419"},"PeriodicalIF":6.0,"publicationDate":"2025-03-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143759542","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Neural NetworksPub Date : 2025-03-29DOI: 10.1016/j.neunet.2025.107427
Heng-yang Lu , Xin Guo , Wenyu Jiang , Chenyou Fan , Yuntao Du , Zhenhao Shao , Wei Fang , Xiaojun Wu
{"title":"MuSIA: Exploiting multi-source information fusion with abnormal activations for out-of-distribution detection","authors":"Heng-yang Lu , Xin Guo , Wenyu Jiang , Chenyou Fan , Yuntao Du , Zhenhao Shao , Wei Fang , Xiaojun Wu","doi":"10.1016/j.neunet.2025.107427","DOIUrl":"10.1016/j.neunet.2025.107427","url":null,"abstract":"<div><div>In the open world, out-of-distribution (OOD) detection is crucial to ensure the reliability and robustness of deep learning models. Traditional OOD detection methods are often limited to using single-source information coupled with the abnormal activations of OOD data, resulting in poor detection performance for OOD samples. To this end, we propose MuSIA (Multi-Source Information Fusion with Abnormal Activations) to obtain effective information from multiple information sources and capture abnormal activations to improve the performance of OOD detection. To verify the effectiveness of MuSIA, we conducted experiments with six OOD datasets on six pre-trained models (ViT, RepVGG, DeiT, etc.). Experimental results show that compared with the SOTA method, MuSIA reduces FPR95 (<span><math><mi>↓</mi></math></span>) by an average of 7.78%. Further ablation studies deeply explore the role of each component in MuSIA, especially the synergy of capturing abnormal activation and multi-source information fusion.</div></div>","PeriodicalId":49763,"journal":{"name":"Neural Networks","volume":"188 ","pages":"Article 107427"},"PeriodicalIF":6.0,"publicationDate":"2025-03-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143759538","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Neural NetworksPub Date : 2025-03-29DOI: 10.1016/j.neunet.2025.107421
Xiaojun Yang , Bin Li , Weihao Zhao , Sha Xu , Jingjing Xue , Feiping Nie
{"title":"Fast self-supervised discrete graph clustering with ensemble local cluster constraints","authors":"Xiaojun Yang , Bin Li , Weihao Zhao , Sha Xu , Jingjing Xue , Feiping Nie","doi":"10.1016/j.neunet.2025.107421","DOIUrl":"10.1016/j.neunet.2025.107421","url":null,"abstract":"<div><div>Spectral clustering (SC) is a graph-based clustering algorithm that has been widely used in the field of data mining and image processing. However, most graph-based clustering methods ignore the utilization of additional prior information. This information can help clustering models further reduce the difference between their clustering results and ground-truth, but is difficult to obtain in unsupervised settings. Moreover, traditional graph-based clustering algorithms require additional hyperparameters and full graph construction to obtain good performance, increasing the tuning pressure and time cost. To address these issues, a simple fast self-supervised discrete graph clustering (FSDGC) is proposed. Specifically, the proposed method has the following features: (1) a novel self-supervised information, based on ensemble local cluster constraints, is used to constrain the sample indicator matrix; (2) the anchor graph technique is introduced for mining the structure between samples and anchors to handle large scale datasets. Meanwhile, a fast coordinate ascent (CA) optimization method, based on self-supervised constraints, is proposed to obtain discrete indicator matrices. Experimental clustering results demonstrate that FSDGC has efficient clustering performance.</div></div>","PeriodicalId":49763,"journal":{"name":"Neural Networks","volume":"188 ","pages":"Article 107421"},"PeriodicalIF":6.0,"publicationDate":"2025-03-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143746268","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Neural NetworksPub Date : 2025-03-29DOI: 10.1016/j.neunet.2025.107423
Zhengzhang Hou , Zhanshan Li , Jingyao Li
{"title":"Bidirectional Semantic Consistency Guided Contrastive Embedding for Generative Zero-Shot Learning","authors":"Zhengzhang Hou , Zhanshan Li , Jingyao Li","doi":"10.1016/j.neunet.2025.107423","DOIUrl":"10.1016/j.neunet.2025.107423","url":null,"abstract":"<div><div>Generative zero-shot learning methods synthesize features for unseen classes by learning from image features and class semantic vectors, effectively addressing bias in transferring knowledge from seen to unseen classes. However, existing methods directly employ global image features without incorporating semantic information, failing to ensure that synthesized features for unseen classes maintain semantic consistency. This results in a lack of discriminative power for these synthesized features. To address these limitations, we propose a Bidirectional Semantic Consistency Guided (BSCG) generation model. The BSCG model utilizes a Bidirectional Semantic Guidance Framework (BSGF) that combines Attribute-to-Visual Guidance (AVG) and Visual-to-Attribute Guidance (VAG) to enhance interaction and mutual learning between visual features and attribute semantics. Additionally, we propose a Contrastive Consistency Space (CCS) to optimize feature quality further by improving intra-class compactness and inter-class separability. This approach ensures robust knowledge transfer and enhances the model’s generalization ability. Extensive experiments on three benchmark datasets show that the BSCG model significantly outperforms existing state-of-the-art approaches in both conventional and generalized zero-shot learning settings. The codes are available at: <span><span>https://github.com/ithicker/BSCG</span><svg><path></path></svg></span>.</div></div>","PeriodicalId":49763,"journal":{"name":"Neural Networks","volume":"188 ","pages":"Article 107423"},"PeriodicalIF":6.0,"publicationDate":"2025-03-29","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143746937","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Neural NetworksPub Date : 2025-03-28DOI: 10.1016/j.neunet.2025.107426
Kun Fang , Qinghua Tao , Yingwen Wu , Tao Li , Xiaolin Huang , Jie Yang
{"title":"Multi-head ensemble of smoothed classifiers for certified robustness","authors":"Kun Fang , Qinghua Tao , Yingwen Wu , Tao Li , Xiaolin Huang , Jie Yang","doi":"10.1016/j.neunet.2025.107426","DOIUrl":"10.1016/j.neunet.2025.107426","url":null,"abstract":"<div><div>Randomized Smoothing (RS) is a promising technique for certified robustness, and recently in RS the ensemble of multiple Deep Neural Networks (DNNs) has shown state-of-the-art performances due to its variance reduction effect over Gaussian noises. However, such an ensemble brings heavy computation burdens in both training and certification, and yet under-exploits individual DNNs and their mutual effects, as the communication between these classifiers is commonly ignored in optimization. In this work, we consider a novel <em>ensemble</em>-based training way for a <em>single</em> DNN with multiple augmented heads, named as SmOothed Multi-head Ensemble (SOME). In SOME, similar to the pursuit of variance reduction via ensemble, an ensemble of multiple heads imposed with a cosine constraint inside a single DNN is employed with much cheaper training and certification computation overloads in RS. In such network structure, an associated training strategy is designed by introducing a circular communication flow among those augmented heads. That is, each head teaches its neighbor with the self-paced learning strategy using smoothed losses, which are specifically designed in relation to certified robustness. The deployed multi-head structure and the circular-teaching scheme in SOME jointly contribute to the diversities among multiple heads and benefit their ensemble, leading to a competitively stronger certifiably-robust RS-based defense than ensembling multiple DNNs (effectiveness) at the cost of much less computational expenses (efficiency), verified by extensive experiments and discussions.</div></div>","PeriodicalId":49763,"journal":{"name":"Neural Networks","volume":"188 ","pages":"Article 107426"},"PeriodicalIF":6.0,"publicationDate":"2025-03-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143807967","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Neural NetworksPub Date : 2025-03-27DOI: 10.1016/j.neunet.2025.107424
Qianyao Qiang , Bin Zhang , Chen Jason Zhang , Feiping Nie
{"title":"Adaptive bigraph-based multi-view unsupervised dimensionality reduction","authors":"Qianyao Qiang , Bin Zhang , Chen Jason Zhang , Feiping Nie","doi":"10.1016/j.neunet.2025.107424","DOIUrl":"10.1016/j.neunet.2025.107424","url":null,"abstract":"<div><div>As a crucial machine learning technology, graph-based multi-view unsupervised dimensionality reduction aims to learn compact low-dimensional representations for unlabeled multi-view data using graph structures. However, it faces several challenges, including the integration of multiple heterogeneous views, the absence of label guidance, the rigidity of predefined similarity graphs, and high computational intensity. To address these issues, we propose a novel method called adaptive Bigraph-based Multi-view Unsupervised Dimensionality Reduction (BMUDR). BMUDR dynamically learns view-specific anchor sets and adaptively constructs a bigraph shared by multiple views, facilitating the discovery of low-dimensional representations through sample-anchor relationships. The generation of anchors and the construction of anchor similarity matrices are integrated into the dimensionality reduction process. Diverse contributions of different views are automatically weighed to leverage their complementary and consistent properties. In addition, an optimization algorithm is designed to enhance computational efficiency and scalability, and it provides impressive performance in low-dimensional representation learning, as demonstrated by extensive experiments on various benchmark datasets.</div></div>","PeriodicalId":49763,"journal":{"name":"Neural Networks","volume":"188 ","pages":"Article 107424"},"PeriodicalIF":6.0,"publicationDate":"2025-03-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143725256","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Neural NetworksPub Date : 2025-03-27DOI: 10.1016/j.neunet.2025.107422
Hongyi Nie , Shiqi Fan , Yang Liu , Quanming Yao , Zhen Wang
{"title":"Using samples with label noise for robust continual learning","authors":"Hongyi Nie , Shiqi Fan , Yang Liu , Quanming Yao , Zhen Wang","doi":"10.1016/j.neunet.2025.107422","DOIUrl":"10.1016/j.neunet.2025.107422","url":null,"abstract":"<div><div>Recent studies have shown that effectively leveraging samples with label noise can enhance model robustness by uncovering more reliable feature patterns. While existing methods, such as label correction methods and loss correction techniques, have demonstrated success in utilizing noisy labels, they assume that noisy and clean samples (samples with correct annotations) share the same label space.However, this assumption does not hold in continual machine learning, where new categories and tasks emerge over time, leading to label shift problems that are specific to this setting. As a result, existing methods may struggle to accurately estimate the ground truth labels for noisy samples in such dynamic environments, potentially exacerbating label noise and further degrading performance. To address this critical gap, we propose a <strong>S</strong>hift-<strong>A</strong>daptive <strong>N</strong>oise <strong>U</strong>tilization (<strong>SANU</strong>) method, designed to transform samples with label noise into usable samples for continual learning. SANU introduces a novel source detection mechanism that identifies the appropriate label space for noisy samples, leveraging a meta-knowledge representation module to improve the generalization of the detection process. By re-annotating noisy samples through label guessing and label generation strategies, SANU adapts to label shifts, turning noisy data into useful inputs for training. Experimental results across three continual learning datasets demonstrate that SANU effectively mitigates the label shift problem, significantly enhancing model performance by utilizing re-annotated samples with label noise.</div></div>","PeriodicalId":49763,"journal":{"name":"Neural Networks","volume":"188 ","pages":"Article 107422"},"PeriodicalIF":6.0,"publicationDate":"2025-03-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143759540","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":1,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}