An agentic, no-code artificial intelligence workflow for developing a thyroid nodule ultrasound malignancy classifier.

Abstract

Background: Convolutional neural networks can classify thyroid nodules on ultrasound, yet published models are seldom available for independent testing, require machine learning expertise to develop and deploy, and are validated mostly on papillary thyroid carcinoma.

Objective: To test whether an autonomous ("agentic"), no-code artificial intelligence (AI) agent can develop a calibrated thyroid nodule malignancy classifier and validate it internally and on an external cohort spanning multiple cancer histologies.

Methods: This was a retrospective, computational diagnostic study. A no-code agent (Hugging Face ML-Intern) autonomously audited the data, selected and trained the model, and calibrated probabilities, using the open-source TN5000 dataset. The locked ResNet-18 model was externally validated on 232 nodules from the University of Colorado and benchmarked against a previously published classifier evaluated on the same source.

Results: On the internal test set, the agentic model achieved an area under the receiver-operating-characteristic curve (AUROC) of 0.94 (95% CI, 0.92-0.95), sensitivity 0.90, and specificity 0.81. On external validation, the model achieved an AUROC of 0.90 (95% CI, 0.85-0.93); sensitivity was 0.92, specificity 0.68, positive predictive value 0.52, and negative predictive value 0.96, exceeding the performance of the benchmark model (AUROC 0.83; paired DeLong P = .03). Missed cancers clustered among follicular pattern tumors.

Conclusion: An agentic, no-code AI workflow produced a calibrated, externally validated thyroid nodule classifier, supporting accessible, reproducible, and independently testable medical AI development. Prospective validation and local recalibration are required before clinical use.