Gap Analysis of the Accuracy of Doctors versus Machine Learning Models for Pneumonia Detection from X-Rays

Abstract

Machine learning (ML) can help in analyzing xray images to assist human doctors. ML algorithms are not perfect and when a ML algorithm makes a diagnostic error, it is often unclear why - were the features genuinely confusing or was it a badly trained algorithm. In this work, we first compare the accuracy of four well-known ML algorithms (KNN, Decision Tree, Logistic Regression and Convolutional Neural network) to detect pneumonia from chest X-rays of pediatric patients. We show that an algorithm based on Convolutional Neural Networks (CNN) gave the best accuracy of 90.7%. We then present a small test set of the X-rays which were wrongly diagnosed by the CNN algorithm to a panel of 14 doctors to investigate why the algorithm may have failed. We analyze the gap between ML algorithms and real doctors. The panel of doctors was able to diagnose 37% of the images correctly, while it was confused on the remaining 63% of images. This shows that better ML algorithms and training methods can improve the accuracy up to 94%. For the truly confusing images, the doctors identified the following additional features that could be included to help in the diagnosis: Oxygen saturation level (SPO2), Age, Respiratory Rate, and Body Temperature.