Automated multimodal process knowledge graph construction for intelligent process planning with Cross-Modal Transformers

Intelligent process planning is pivotal in modern manufacturing systems, enabling efficient, precise, and flexible production by optimizing resource allocation, enhancing machining accuracy, and shortening production cycles. Knowledge graphs integrate multi-source heterogeneous data to support this process, yet traditional single-modal approaches hinder the exploration of complex relationships in multimodal data, falling short of the needs for complex part planning. This paper examines machining features, the foundational units of process planning, and introduces an automatic construction method for a Multimodal Process Knowledge Graph (MPKG) tailored to intelligent process planning, powered by Cross-Modal Transformers. We developed the MF36 dataset, encompassing 36 machining features with 3D models, engineering views, and descriptive texts. A cross-modal framework integrating LERT-CRF and PA-ViT models automates the extraction and fusion of multimodal process knowledge, with PA-ViT’s pooling attention mechanism markedly boosting machining feature recognition accuracy. Experiments demonstrate superior performance over baselines, achieving F1 scores of 0.895 in entity extraction and 0.877 in image recognition. A case study validates the method’s reliability for precise process recommendations, providing fresh insights into advancing intelligent process planning.