Journal of Machine Learning for Modeling and Computing最新文献

{"title":"INDEX, VOLUME 4, 2023","authors":"","doi":"10.1615/jmachlearnmodelcomput.v4.i4.60","DOIUrl":"https://doi.org/10.1615/jmachlearnmodelcomput.v4.i4.60","url":null,"abstract":"","PeriodicalId":320604,"journal":{"name":"Journal of Machine Learning for Modeling and Computing","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135609123","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"HOW IMPORTANT ARE ACTIVATION FUNCTIONS IN REGRESSION AND CLASSIFICATION? A SURVEY, PERFORMANCE COMPARISON, AND FUTURE DIRECTIONS","authors":"Ameya Jagtap, George Em Karniadakis","doi":"10.1615/jmachlearnmodelcomput.2023047367","DOIUrl":"https://doi.org/10.1615/jmachlearnmodelcomput.2023047367","url":null,"abstract":"Inspired by biological neurons, the activation functions play an essential part in the learning process of any artificial neural network (ANN) commonly used in many real-world problems. Various activation functions have been proposed in the literature for classification as well as regression tasks. In this work, we survey the activation functions that have been employed in the past as well as the current state-of-the-art. In particular, we present various developments in activation functions over the years and the advantages as well as disadvantages or limitations of these activation functions. We also discuss classical (fixed) activation functions, including rectifier units, and adaptive activation functions. In addition to discussing the taxonomy of activation functions based on characterization, a taxonomy of activation functions based on applications is presented. To this end, the systematic comparison of various fixed and adaptive activation functions is performed for classification datasets such as MNIST, CIFAR-10, and CIFAR-100. In recent years, a physics-informed machine learning framework has emerged for solving problems related to scientific computations. For this purpose, we also discuss various requirements for activation functions that have been used in the physics-informed machine learning framework. Furthermore, various comparisons are made among different fixed and adaptive activation functions using various machine learning libraries such as TensorFlow, PyTorch, and JAX. Our findings show that activation functions such as rectified linear unit (ReLU) and its variants, which are currently the state-of-the-art for many classification problems, do not work well in physics-informed machine learning frameworks due to the stringent requirement of the existence of derivatives, whereas other activation functions such as hyperbolic tangent, swish, and sine give better performance, with superior results achieved with adaptive activation functions, especially for multiscale problems.","PeriodicalId":320604,"journal":{"name":"Journal of Machine Learning for Modeling and Computing","volume":"86 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136092803","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Physics-informed PointNet: On how many irregular geometries can it solve an inverse problem simultaneously? Application to linear elasticity","authors":"Ali Kashefi, Leonidas Guibas, Tapan Mukerji","doi":"10.1615/jmachlearnmodelcomput.2023050011","DOIUrl":"https://doi.org/10.1615/jmachlearnmodelcomput.2023050011","url":null,"abstract":"Regular physics-informed neural networks (PINNs) predict the solution of partial differential equations using sparse labeled data but only over a single domain. On the other hand, fully supervised learning models are first trained usually over a few thousand domains with known solutions (i.e., labeled data) and then predict the solution over a few hundred unseen domains. Physics-informed PointNet (PIPN) is primarily designed to fill this gap between PINNs (as weakly supervised learning models) and fully supervised learning models. In this article, we demonstrate for the first time that PIPN predicts the solution of desired partial differential equations over a few hundred domains simultaneously, while it only uses sparse labeled data. This framework benefits fast geometric designs in the industry when only sparse labeled data are available. Particularly, we show that PIPN predicts the solution of a plane stress problem over more than 500 domains with different geometries, simultaneously. Moreover, we pioneer implementing the concept of remarkable batch size (i.e., the number of geometries fed into PIPN at each sub-epoch) into PIPN. We systematically try batch sizes of 7, 14, 19, 38, 76, and 133. Additionally, we systematically investigate for the first time the effect of the PIPN size, symmetric function in the PIPN architecture, and static and dynamic weights for the component of the sparse labeled data in the PIPN loss function.","PeriodicalId":320604,"journal":{"name":"Journal of Machine Learning for Modeling and Computing","volume":"13 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135551564","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Error Estimates of Residual Minimization using NNs for Linear PDEs","authors":"Yeonjong Shin, Zhongqiang Zhang, George Em Karniadakis","doi":"10.1615/jmachlearnmodelcomput.2023050411","DOIUrl":"https://doi.org/10.1615/jmachlearnmodelcomput.2023050411","url":null,"abstract":"We propose an abstract framework for analyzing the convergence of least-squares methods based on residual minimization when feasible solutions are neural networks. With the norm relations and compactness arguments, we derive error estimates for both continuous and discrete formulations of residual minimization in strong and weak forms. The formulations cover recently developed physicsinformed neural networks based on strong and variational formulations.","PeriodicalId":320604,"journal":{"name":"Journal of Machine Learning for Modeling and Computing","volume":"285 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135844869","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"ChatGPT FOR PROGRAMMING NUMERICAL METHODS","authors":"Ali Kashefi, Tapan Mukerji","doi":"10.1615/jmachlearnmodelcomput.2023048492","DOIUrl":"https://doi.org/10.1615/jmachlearnmodelcomput.2023048492","url":null,"abstract":"ChatGPT is a large language model recently released by the OpenAI company. In this technical report, we explore for the first time the capability of ChatGPT for programming numerical algorithms. Specifically, we examine the capability of GhatGPT for generating codes for numerical algorithms in different programming languages, for debugging and improving written codes by users, for completing missed parts of numerical codes, rewriting available codes in other programming languages, and for parallelizing serial codes. Additionally, we assess if ChatGPT can recognize if given codes are written by humans or machines. To reach this goal, we consider a variety of mathematical problems such as the Poisson equation, the diffusion equation, the incompressible Navier-Stokes equations, compressible inviscid flow, eigenvalue problems, solving linear systems of equations, storing sparse matrices, etc. Furthermore, we exemplify scientific machine learning such as physics-informed neural networks and convolutional neural networks with applications to computational physics. Through these examples, we investigate the successes, failures, and challenges of ChatGPT. Examples of failures are producing singular matrices, operations on arrays with incompatible sizes, programming interruption for relatively long codes, etc. Our outcomes suggest that ChatGPT can successfully program numerical algorithms in different programming languages, but certain limitations and challenges exist that require further improvement of this machine learning model.","PeriodicalId":320604,"journal":{"name":"Journal of Machine Learning for Modeling and Computing","volume":"2 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135843258","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"FLOW MAP LEARNING FOR UNKNOWN DYNAMICAL SYSTEMS: OVERVIEW, IMPLEMENTATION, AND BENCHMARKS","authors":"Victor Churchill, Dongbin Xiu","doi":"10.1615/jmachlearnmodelcomput.2023049717","DOIUrl":"https://doi.org/10.1615/jmachlearnmodelcomput.2023049717","url":null,"abstract":"Flow map learning (FML), in conjunction with deep neural networks (DNNs), has shown promise for data driven modeling of unknown dynamical systems. A remarkable feature of FML is that it is capable of producing accurate predictive models for partially observed systems, even when their exact mathematical models do not exist. In this paper, we present an overview of the FML framework, along with the important computational details for its successful implementation. We also present a set of well defined benchmark problems for learning unknown dynamical systems. All the numerical details of these problems are presented, along with their FML results, to ensure that the problems are accessible for cross-examination and the results are reproducible.","PeriodicalId":320604,"journal":{"name":"Journal of Machine Learning for Modeling and Computing","volume":"195 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135828709","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Deep Learning of Subgrid-Scale Dynamics in Scalar Turbulence and Generalization to other Transport Regimes","authors":"Ali Akhavan-Safaei, Mohsen Zayernouri","doi":"10.1615/jmachlearnmodelcomput.2023048824","DOIUrl":"https://doi.org/10.1615/jmachlearnmodelcomput.2023048824","url":null,"abstract":"We present a deep learning surrogate model in the form of fully-connected feed-forward neural networks to predict the SGS scalar flux in the context of large-eddy simulation of turbulent transport. The deep neural network model is trained and validated using filtered DNS dataset at $Pe_lambda=240, , Sc=1$ that includes the filtered scalar and velocity gradients as input features. We provide a sampling strategy in time and space to construct a sufficiently inclusive dataset for successful model optimization. Moreover, using transfer learning notion, we adjust this trained model to perform accurately for turbulent scalar transport regimes with higher $Pe_lambda$ and $Sc$ numbers with a fairly low amount of data and computations. In the inference mode on the full-size 3-D Large-Eddy Simulation (LES) grids over time, we carry out textit{a priori} tests for the developed deep learning models, and compare them to the results obtained from conventional models of SGS flux that are defined based on gradients of filtered fields. We show that the deep learning model outperforms traditional SGS models that either suffer from their fundamental restrictions in following the physics or under/over-prediction of SGS quantities. For the transfer learned models, these comparisons are made between the base (pre-trained) model and the re-trained one on the new transport regime, and we discuss the importance of transfer learning in reduction of notably large prediction errors of the pre-trained model. At the end, as an example, we test our base model in an LES sense, and realize that unlike the conventional gradient-based SGS models, it successfully upholds the temporal accuracy of resolve","PeriodicalId":320604,"journal":{"name":"Journal of Machine Learning for Modeling and Computing","volume":"65 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135562668","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"PHYSICS-INFORMED NEURAL NETWORKS BASED ON SEQUENTIAL TRAINING FOR CO2 UTILIZATION AND STORAGE IN SUBSURFACE RESERVOIR","authors":"Kiarash Mansour Pour, Denis Voskov","doi":"10.1615/jmachlearnmodelcomput.2023048866","DOIUrl":"https://doi.org/10.1615/jmachlearnmodelcomput.2023048866","url":null,"abstract":"CO₂ utilization and storage (CCUS) simulation in subsurface reservoirs with complex heterogeneous structures necessitates a model that can capture multiphase compositional flow and transport. The governing equations are highly nonlinear due to the complex thermodynamic behavior, which involves the appearance and disappearance of multiple phases. Accurate simulation of these processes necessitates the use of stable numerical methods. While machine learning (ML) approaches have been used to solve a variety of nonlinear computational problems, a new approach based on physics-informed neural networks (PINNs) has been proposed for solving partial differential equations (PDEs). Unlike typical ML algorithms that require a large dataset for training, PINNs can train the network with unlabeled data. The applicability of this method has been explored for multiphase flow and transport in porous media. However, for nonlinear hyperbolic transport equations, the solution degrades significantly. This work proposes sequential training PINNs to simulate two-phase transport in porous media. The main concept is to retrain the neural network to solve the PDE over successive time segments rather than train for the entire time domain simultaneously. We observe that sequential training can capture the solution more accurately concerning the standard training for conventional two-phase problems. Furthermore, we extend the sequential training approach for compositional problems in which nonlinearity is more significant due to the complex phase transition. Our approach was tested on miscible and immiscible test cases and showed higher accuracy than the standard training method.","PeriodicalId":320604,"journal":{"name":"Journal of Machine Learning for Modeling and Computing","volume":"32 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135445259","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"MyCrunchGPT: A chatGPT assisted framework for scientific machine learning","authors":"Varun Kumar, Leonard Gleyzer, Adar Kahana, Khemraj Shukla, George Em Karniadakis","doi":"10.1615/jmachlearnmodelcomput.2023049518","DOIUrl":"https://doi.org/10.1615/jmachlearnmodelcomput.2023049518","url":null,"abstract":"Scientific machine learning (SciML) has advanced recently across many different areas in computational science and engineering. The objective is to integrate data and physics seamlessly without the need of employing elaborate and computationally taxing data assimilation schemes. However, preprocessing, problem formulation, code generation, postprocessing, and analysis are still time- consuming and may prevent SciML from wide applicability in industrial applications and in digital twin frameworks. Here, we integrate the various stages of SciML under the umbrella of ChatGPT, to formulate MyCrunchGPT, which plays the role of a conductor orchestrating the entire workflow of SciML based on simple prompts by the user. Specifically, we present two examples that demonstrate the potential use of MyCrunchGPT in optimizing airfoils in aerodynamics, and in obtaining flow fields in various geometries in interactive mode, with emphasis on the validation stage. To demonstrate the flow of the MyCrunchGPT, and create an infrastructure that can facilitate a broader vision, we built a web app based guided user interface, that includes options for a comprehensive summary report. The overall objective is to extend MyCrunchGPT to handle diverse problems in computational mechanics, design, optimization and controls, and general scientific computing tasks involved in SciML, hence using it as a research assistant tool but also as an educational tool. While here the examples focus on fluid mechanics, future versions will target solid mechanics and materials science, geophysics, systems biology, and bioinformatics.","PeriodicalId":320604,"journal":{"name":"Journal of Machine Learning for Modeling and Computing","volume":"26 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"135844238","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

{"title":"Locally Adaptive and Differentiable Regression","authors":"Mingxuan Han, Varun Shankar, Chenglong Ye, Jeff Phillips","doi":"10.1615/jmachlearnmodelcomput.2023049746","DOIUrl":"https://doi.org/10.1615/jmachlearnmodelcomput.2023049746","url":null,"abstract":"Overparameterized models like deep nets and random forests have become very popular in machine learning. However, the natural goals of continuity and differentiability, common in regression models, are now often ignored in modern overparametrized, locally adaptive models. We propose a general framework to construct a global continuous and differentiable model based on a weighted average of locally learned models in corresponding local regions. This model is competitive in dealing with data with different densities or scales of function values in different local regions. We demonstrate that when we mix kernel ridge and polynomial regression terms in the local models, and stitch them together continuously, we achieve faster statistical convergence in theory and improved performance in various practical settings.","PeriodicalId":320604,"journal":{"name":"Journal of Machine Learning for Modeling and Computing","volume":"36 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"136008231","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}