Transformer-based auto-encoder with combined multi-head-attention for industrial soft-sensor modeling

Soft-sensor modeling is common in industrial production, but high data dimensionality, a lack of labeled features, and inadequate methods complicate extracting nonlinear feature representations. This paper proposes a Transformer-based auto-encoder with a combined multi-head-attention approach (TAE-CMHA) for soft-sensor modeling, which offers advantages for nonlinear feature representation. It introduces a combined multi-head-attention mechanism (CMHA) that improves feature-extraction accuracy and robustness. The Transformer's global feature extraction capabilities are leveraged in the auto-encoder for better nonlinear feature extraction. Additionally, label information optimizes the auto-encoder's reconstruction loss function which improves feature acquisition for predicting target outputs. Compared to supervised methods, the unsupervised auto-encoder uses abundant unlabeled industrial data to improve generalizability. Experiments were conducted on the industrial steam flow and debutanizer column datasets. The results show that the mean squared error (MSE) of the proposed method reaches a minimum of 0.00297 and the coefficient of determination (R²) is 0.881 in debutanizer column datasets, which shows the advantages of the model.