Deep Learning with Hyperspectral and Normal Camera Images for Automated Recognition of Orally-administered Drugs

Abstract

Patient compliance during drug trials and adherence to treatment regimens after a medical diagnosis are known pervasive problems in the practice of medicine. Any practical solution to this problem will require an easy method to identify and to verify the administration of orally-ingested drugs. Deep learning algorithms were applied to images of drugs in pill form. These images were taken using both a smart phone camera and using a hyperspectral imager based on a low-cost CMOS camera. As a proof-of-concept demonstration, 1, 7SS images were taken using a normal CMOS camera of four common pill types. The images of acetaminophen, acetylsalicylic acid and ibuprofen were taken using various backgrounds, image angles, and lighting conditions. The results show over 90% accuracy when the convolutional neural network is trained and tested using only normal camera images. The results improved to 100% when trained and tested using4 baseline “datacubes” taken with a low-cost hyperspectral camera solution; however, due to matrix dimensional differences, a ID CNN was used in this case, while a 2D CNN was used with the normal camera images. Each hyperspectral cube included information from effectively 31 wavebands. With more hyperspectral images to expand the drug training set, this approach would be promising for daily use to quickly identify similar pills in the clinical or home environment as well as in smart phone apps to remotely monitor patient compliance to a drug-based treatment regimen.