LightAlloy-Bench: Benchmarking reasoning abilities of large language models in light alloys

Abstract

Light alloys play a critical role in lightweight engineering and sustainable development owing to their high specific strength and resource efficiency. However, their intelligent design remains challenging because of complex structure–process–property relationships and the scarcity of structured, high-quality data. Large language models (LLMs) have recently emerged as a promising paradigm for AI-driven materials research, yet their reasoning capabilities in the context of light alloys have not been systematically assessed. Here, we introduce LightAlloy-Bench, a hierarchical benchmark tailored for light alloy research, comprising a foundational layer with 2300 questions covering domain knowledge and a reasoning layer with 10,096 questions designed to evaluate multi-step reasoning. Using this benchmark, we evaluate 13 general-purpose and 3 materials-domain LLMs. Among all models, GPT-4o achieves the highest reasoning accuracy without any task-specific optimization. Building on this baseline, we further investigate the effects of Chain-of-Thought (CoT) prompting and reinforcement learning (RL)-based optimization. Notably, Qwen3-14B attains the strongest reasoning performance under CoT prompting, outperforming larger-scale LLMs. This result demonstrates that model scale alone does not determine reasoning capability and highlights the critical role of reasoning-oriented optimization strategies. Overall, LightAlloy-Bench establishes a quantitative foundation for advancing LLM-driven reasoning in intelligent light alloy design and offers a generalizable framework for developing reasoning-focused benchmarks across other alloy systems.