Real-time tracking railway intruders using multiple-agent cooperated large language models with edge stream processing engine

Abstract

Tracking intruders is crucial for ensuring safe railway operations globally, particularly in high-speed railway systems. Traditional methods either rely on post-processing on cloud platforms or suffer from limited analytical capabilities on edge devices. Although large language models (LLMs) have shown great potential to support general intelligence, challenges remain for edge devices in accurately and timely tracking of intruders along railway lines. This study proposes a novel method that combines a multi-agent cooperative framework (MetaGPT) with an edge stream processing engine (GeoEkuiper). Unlike most methods, this study adopts an agent-cooperative spatial data analysis approach employing a debate-and-vote strategy. Specifically, GeoEkuiper is responsible for processing high-speed and large volume of geospatial data streams regarding location history and object characteristics, while the modified MetaGPT framework facilitates information sharing and decision-making between agents that use LLMs. By enabling each edge device to engage in a debate about the presence of detected targets within their monitoring areas, the system utilizes a simple voting agent to determine which devices are most likely to observe the target. Considering the resource limitations of edge devices, we fine-tuned small yet powerful LLMs to direct GeoEkuiper to iteratively compute spatial affinity relationships using Structured Query Language (SQL) statements, which facilitate human-edge interaction. Based on tests conducted on resource-constrained edge devices such as Raspberry Pi devices interconnected in an unstable networking environment, we found that this approach significantly enhances the accuracy and responsiveness of intruder tracking in real-time scenarios, providing a robust solution for railway security applications.