Intelligent Medicine Identification System Using a Combination of Image Recognition and Optical Character Recognition

Abstract

This research aims to develop an automatic verification system with deep learning techniques to verify prescription dispensing accuracy. The proposed method will be able to help pharmacies to reduce errors that lead to patients receiving the wrong medicine to patients. The system consists of two models: image classification and text classification. The image classification model uses raw medicine blister pack images, then removes the background to interpret the features based on the pattern recognition for Histograms of Oriented Gradients (HOG) of the model. It is composed of Convolution Neural Network (CNN), Linear Regression, and Logistic Regression. The text classification model uses text extraction to obtain imprints appearing on the blister package then matches the words to a bag of word. The dataset collected two-hundred types of medicine blister packs images inside plastic zip bags as a dataset. It includes 300 high-quality images of front-side medicine blister packages for each type of package in light-controlled conditions with a black background, which are used for training the model. The automatic verification system uses the majority vote based on the confidence of the two models. Experimental results, indicate that the image classification model of CNN with HOG feature extraction has the highest accuracy at 95.83 percent. In-text classification results show that the method using Character Region Awareness For Text detection (CRAFT), Keras-OCR, and text correction gave the highest accuracy at 92 percent. Overall accuracy was 94.23 percent.