COLLECT: A Counterfactual Oral Lesion Library for Trustworthy AI Evaluation

Abstract

Intraoral mucosal lesions are difficult to diagnose because they vary widely in appearance and cause. Deep-learning models perform well for oral lesion classification, but limited interpretability restricts clinical use. Counterfactual explanations, which modify image features to test model sensitivity, offer a practical way to improve transparency, yet no counterfactual dataset exists for intraoral images. This study presents COLLECT, the Counterfactual Oral Lesion Library for Explainable Concept Testing, a curated counterfactual reference dataset and evaluation framework for intraoral lesion images, along with a workflow for assessing model interpretability. COLLECT includes 600 manually edited counterfactual images across 4 lesion types: aphthous ulcer, geographic tongue, hairy tongue, and oral squamous cell carcinoma. Images were modified along clinically important concepts such as size, color, and opacity. We assessed the interpretability of common convolutional neural networks using true class probability, predictive entropy, testing with concept activation vectors (TCAV), and attribution scores. Counterfactual edits reduced true class probability and increased entropy, with the strongest effects observed for opacity and color changes. TCAV results showed that early network layers were less responsive to concept changes, while later layers depended more on baseline features. Attribution scores confirmed that opacity changes lowered model confidence most, followed by color and size. Some models showed limited response to lesion removal, indicating reliance on background anatomy rather than lesion-specific cues. COLLECT supports the systematic evaluation of computer vision models under clinically grounded counterfactual conditions and provides a resource for improving transparency in the automated diagnosis of oral lesions.