Multimodal learning of heat capacity based on transformers and crystallography pretraining

Thermal properties of materials are essential to many applications of thermal electronic devices. Density functional theory (DFT) has shown capability in obtaining an accurate calculation. However, the expensive computational cost limits the application of the DFT method for high-throughput screening of materials. Recently, machine learning models, especially graph neural networks (GNNs), have demonstrated high accuracy in many material properties’ prediction, such as bandgap and formation energy, but fail to accurately predict heat capacity(CV) due to the limitation in capturing crystallographic features. In our study, we have implemented the material informatics transformer (MatInFormer) framework, which has been pretrained on lattice reconstruction tasks. This approach has shown proficiency in capturing essential crystallographic features. By concatenating these features with human-designed descriptors, we achieved a mean absolute error of 4.893 and 4.505 J/(mol K) in our predictions. Our findings underscore the efficacy of the MatInFormer framework in leveraging crystallography, augmented with additional information processing capabilities.