The transfer learning gap: quantifying transfer learning in a medical image case

Abstract

Transfer learning is a widely used technique in medical imaging and other research fields where a scarcity of available data limit the training of machine learning algorithms. Despite its widespread use and extensive supporting body of research, the specific mechanisms behind transfer learning are not completely understood. In this work, we quantify the effectiveness of transfer learning in medical image classification scenarios for different numbers of training set images. We trained ResNet50, a popular deep learning model used in medical image classification, using two scenarios: 1) applying transfer learning to a pre-trained network and 2) training the same model from scratch (i.e., starting with randomly selected weights). We analyzed the performance of the model under both scenarios as the number of training set images increased from 5,000 to 160,000 medical images. We introduced and evaluated a metric, the transfer learning gap (TLG), to quantify the differences between the two scenarios. The TLG measured the difference in the area under the loss curves (AULCs) when transfer learning was applied and when the model was trained from scratch. Our experiments show that as the training set size increases, the TLG trends to zero, suggesting that the advantage of using transfer learning decreases. The trend in the AULC suggests a training set size where the two scenarios would have equal losses. At this point, the model reaches the same performance regardless of if transfer learning or training from scratch was used. This study is important because it provides a novel metric to understand and quantify the effect of transfer learning.