Thumb: A forensic automation framework leveraging MLLMs and OCR on Android device

Abstract

The forensic of Android devices is challenging due to automated thumbnail generation by applications and the operating system, complicating attribution to specific user actions. This paper presents the design, implementation, and evaluation of a forensic framework, Thumb, which performs real-time experiments on physical Android devices. Thumb integrates multimodal large language models (MLLM) and Optical Character Recognition (OCR) to capture on-screen information and simulate user interactions, while extracting data from internal storage to monitor changes in cached and thumbnail files. A proof-of-concept implementation demonstrates the framework's accuracy across various applications, highlighting its potential to simplify Android forensic analysis. However, current MLLM limitations and the framework's structure pose challenges in complex scenarios and detailed data analysis.