Conversational LLM-Based Decision Support for Defect Classification in AFM Images

Abstract

Atomic force microscopy (AFM) has emerged as a powerful tool for nanoscale imaging and quantitative characterization of organic (e.g., live cells, proteins, DNA, and lipid bilayers) and inorganic (e.g., silicon wafers and polymers) specimens. However, image artifacts in AFM height and peak force error images directly affect the precision of nanomechanical measurements. Experimentalists face considerable challenges in obtaining high-quality AFM images due to the requirement of specialized expertise and constant manual monitoring. Another challenge is the lack of high-quality AFM datasets to train machine learning models for automated defect detection. In this work, we propose a two-step AI framework that combines a vision-based deep learning (DL) model for classifying AFM image defects with a large language model (LLM)-based conversational assistant that provides real-time corrective guidance in natural language, making it particularly valuable for non-AFM experts aiming to obtain high-quality images. We curated an annotated AFM defect dataset spanning organic and inorganic samples to train the defect detection model. Our defect classification model achieves 91.43% overall accuracy, with a recall of 93% for tip contamination and 60% not-tracking defects. We further develop an intuitive user interface that enables seamless interaction with the DL model and integrates an LLM-based guidance feature to support users in understanding defects and improving future experiments. We then evaluate the performance of multiple state-of-the-art LLMs on AFM-related queries, offering users flexibility in LLM selection based on their specific needs. LLM evaluations and the benchmark questions are available at: https://github.com/idealab-isu/AFM-LLM-Defect-Guidance.