Online monitoring method for chip pin with minor deformation defects based on depth-histogram modalities and target-oriented multimodal self-attention mechanism

In the process of chip SMT (surface mounting technology), the quality of the chip pins determines the success rate of the mounting process. However, existing target detection algorithms present poor performance when dealing with deformations in the pins, which is insufficient to meet the industrial demands for accuracy and speed of online monitoring. To solve this problem, a real-time detection method based on DH (Depth-Histogram) Modalities and TMSM (Target-oriented Multimodal Self-attention Mechanism) is proposed. There are three parts in this method, including feature extraction, feature fusion, and decision module. Firstly, a lightweight network for feature extraction and fusion is employed to extract geometric information from the depth images. Subsequently, the Decision Module is used to determine whether there are defects in the pins. Within this framework, the HIEF (Histogram-Integrated Embedding Function) is utilized to extract a one-dimensional vector with height information from the histogram, which is then aligned with the flattened depth image to form DH Modalities. To validate the effectiveness of the proposed algorithm, two datasets are constructed. Experimental results demonstrate that the proposed method has a good performance to meet the speed and accuracy requirements of online monitoring.