SOLAR: Illuminating LLM performance in API discovery and service ranking for edge AI and IoT

Abstract

The growing complexity of web service and API discovery calls for robust methods to evaluate how well Large Language Models (LLMs) retrieve, rank, and assess APIs. However, current LLMs often produce inconsistent results, highlighting the need for structured, multi-dimensional evaluation. This paper introduces SOLAR (Systematic Observability of LLM API Retrieval), a framework that assesses LLM performance across three key dimensions: functional capability, implementation feasibility, and service sustainability. We evaluate four leading LLMs—GPT-4 Turbo (OpenAI), Claude 3.5 Sonnet (Anthropic), LLaMA 3.2 (Meta), and Gemini 2.0 Flash (Google)—on their ability to identify, prioritize, and evaluate APIs across varying query complexities. Results show GPT-4 Turbo and Claude 3.5 Sonnet achieve high functional alignment (FCA ≥ 0.75 for simple queries) and strong ranking consistency (Spearman’s ρ ≈ 0.95). However, all models struggle with implementation feasibility and long-term sustainability, with feasibility scores declining as complexity increases and sustainability scores remaining low (SSI ≈ 0.40), limiting deployment potential. Despite retrieving overlapping APIs, models often rank them inconsistently, raising concerns for AI-driven service selection. SOLAR identifies strong correlations between functional accuracy and ranking stability but weaker links to real-world feasibility and longevity. These findings are particularly relevant for Edge AI environments, where real-time processing, distributed intelligence, and reliable API integration are critical. SOLAR offers a comprehensive lens for evaluating LLM effectiveness in service discovery, providing actionable insights to advance robust, intelligent API integration across IoT and AI-driven systems. Our work aims to inform both future model development and deployment practices in high-stakes computing environments.