Synthesis application of electronic tongue and electronic eye combined with a transformer-graph fusion network for hawthorn origin traceability

Hawthorn is a renowned medicinal food herb that is widely recognized for its cardio protective efficacy and ability to lower blood pressure. Hawthorns from various regions display similar morphological characteristics, which poses significant challenges for their differentiation. This article presents a novel method for rapid traceability of hawthorn origin via a synthesis application of electronic tongue (ET) and electronic eye (EE) combined with a Transformer-graph fusion network. First, the ET and EE are utilized separately to collect the gustatory fingerprints and visual image of hawthorn samples from different production areas. A time series Transformer encoder (TSTE) and an improved Swin Transformer encoder (ISTE) are designed to capture both global temporal dependencies and local features from ET and EE signals, respectively. A graph fusion network (GFN) is subsequently proposed to fuse the features information of ET and EE, and thereby leveraging the complementarity of both modalities for more comprehensive characteristic representations. Subsequently, the fused information ET and EE is input into the decision network for recognition and classification. The experimental results show that adopting a fusion network strategy based on ET and EE senses achieves better recognition accuracy than does using independent sensing devices. The accuracy, precision, recall rate, and F1 score of the test set reached 99.20%, 99.21%, 99.20%, and 0.991, respectively. This research provides a novel method for quickly identifying the origin of hawthorn, offering broad application prospects for its wide use with other types of traditional Chinese herbal medicines.