ADEPT: An advanced data exploration and processing tool for clinical data insights

Abstract

Background and objective

The rapid growth of clinical data creates challenges in analysis and interpretation for medical professionals. To address these issues, we developed the Advanced Data Exploration and Processing Tool (ADEPT), integrating data preprocessing, modeling, visualization, and statistical reporting to resolve common problems like inaccurate terminology, outliers, and missing values.

Methods

ADEPT incorporates advanced preprocessing, including standardizing numerical values, detecting outliers via Isolation Forest and DBSCAN, and filling missing data with KNN and MissForest. Tokenized text features are processed through keyword-based classification and K-means clustering. Five machine learning models—Gradient Boosting Machine, Random Forest, Extreme Gradient Boosting, Logistic Regression, and Support Vector Machine—are combined with a dynamic voting mechanism. Performance was assessed using precision, sensitivity, and specificity.

Results

ADEPT demonstrated substantial performance improvements, with the Area Under the Curve (AUC) increasing by over 14 %. Key results include enhanced precision, sensitivity, and specificity, validating the tool's ability to extract valuable insights from complex datasets.

Conclusions

ADEPT offers a comprehensive solution for automated clinical data analysis, combining rigorous preprocessing with advanced modeling. Its dynamic voting mechanism and integrated tools enhance accuracy and interpretability, addressing critical challenges in clinical data management and decision-making.