Tool or Toy: Are SCA tools ready for challenging scenarios?

Abstract

The widespread adoption of open-source software (OSS) has introduced new security challenges to the software supply chain. While existing studies confirm the basic capabilities of Software Composition Analysis (SCA) tools, such as vulnerability detection and dependency resolution. They often focus on single ecosystems or detection aspects. This limited scope overlooks real-world complexities, including multi-language ecosystems, source and binary dependencies, and adversarial threats. Without a comprehensive evaluation, SCA tools may perform well in controlled settings but struggle in more complex scenarios. To address this gap, this study proposes a evaluation framework centered on the core functionalities of SCA tools: dependency detection, vulnerability identification, and license inspection. It covers three key dimensions including multi-language ecosystems compatibility, build forms, and attack defense. Using standardized datasets and quantitative metrics, such as precision, recall, F1-score and standard deviation, we evaluate four representative SCA tools, including both open-source and commercial options. Results reveal significant limitations in binary dependencies, language coverage, and license consistency. SCA tools also face challenges in balancing precision, coverage and robustness. The study highlights systemic shortcomings in current SCA tools, revealing that many perform like limited-use toys under real-world conditions. It offers data-driven recommendations to guide the evolution of these tools into practical, reliable solutions for supply chain security governance.