Airline baggage classification/recognition and measurement based on computer vision

The current airline baggage handling is mainly by manual, which exist serious problems such as crucial handling, baggage loss, low efficiency, high human labor cost, and so on. To solve these problems, an automatic baggage handling process is more and more needed within current airport operation. To this end, high-accuracy classification and high-precision measurement of airline baggage are essential. In this paper, three works are reported: a baggage classification recognition method based on Convolutional Neural Network (CNN) model, a baggage measurement algorithm using a combination of two-dimensional(2D) image and three-dimensional(3D) point cloud, and their realizations in an embedded platform. Firstly, gray feature of image of an airline baggage was fused with height and gradient features of point cloud of the same baggage to construct a baggage information sample. Two thousand fused baggage information samples were fed into two CNNs (vgg16 and mobilenetv3) for training. The best one was selected as the final predictor. Secondly, three-dimensional size, centroid point position and deflection angle of a baggage were measured in 3D point cloud with help of edge information extracted from the 2D image of the same baggage by Scharr operator. Finally, the proposed recognition method and measurement algorithm were transplanted into an embedded platform for efficiency purpose. Experimental results show that average classification accuracy of the proposed 2D image and 3D point cloud fused baggage information CNN model increased 10% at the best shot compared to former reported models. The proposed 2D-3D combined measurement algorithm also obtained comparable precision versus three former jobs. Most importantly, total processing time of the proposed classification and measurement program takes 86 milliseconds, which is one fifth to one tenth of the best result of former works. Plus, a lightweight version in an embedded platform took 54 milliseconds, 200 times faster than PC terminal's 13 seconds including time of data transmission. Considering a distance of dozens of kilometers in airport remote baggage handling system, the proposed embedded platform version of classification and measurement program is promising in the future's automatic scenarios, such as baggage self-service check-in, baggage tracking, automatic baggage palletization, and so on.