LLM and AI Agents for Autonomous Systems: A Survey of Applications, Datasets, and Security Challenges

Abstract

The rapid integration of Large Language Models (LLMs) into autonomous systems marks a significant transition from modular, rule-based approaches to reasoning-driven, agent-based, and multimodal intelligence. LLM reasoning enables adaptive decision-making, context-aware planning, and human-aligned interaction, while AI agents extend these capabilities into structured autonomy pipelines that coordinate perception, reasoning, and control. These advancements are particularly critical in safety-sensitive domains such as autonomous driving (AD) and unmanned aerial vehicles (UAVs). This survey provides a comprehensive review of LLM reasoning and AI agents across scenario generation, decision-making, multimodal perception, cooperative V2X interactions, and UAV swarm autonomy. We examine the role of simulation platforms and datasets, including CARLA, Apollo ADS, AirSim, nuScenes, DriveLM, and emerging synthetic environments, in supporting reproducible evaluation and benchmarking. In addition, we analyze pressing security and robustness challenges, including adversarial prompt injection, data poisoning, multimodal perturbations, privacy leakage, and vulnerabilities in cooperative agent communication. Finally, we propose future research directions including adversarially robust pipelines, hybrid symbolic LLM planning, secure multimodal fusion, privacy-preserving human alignment, distributed trust mechanisms for swarm autonomy, and optimized Drone-LLM deployment across on-drone, edge, and cloud environments. By unifying applications, datasets, benchmarks, reasoning, agents, and security, this survey establishes a roadmap for developing robust, trustworthy, and secure LLM-enabled autonomous systems.