Towards Optimal Speculative Decoding: A Comprehensive Survey on Optimizations and Future Directions

Abstract

Speculative decoding (SD) has emerged as a compelling approach to breaking the sequential bottleneck of autoregressive large language model (LLM) inference. By coupling lightweight drafting with parallel verification, SD reduces expensive target-model decoding while preserving generation fidelity. Yet its realized speedup often falls short of its theoretical potential, as inefficiencies persist throughout the speculative decoding pipeline. This survey presents a unified analysis of the factors that ultimately bound the acceleration of SD. We examine how draft effectiveness, verification efficiency, and system execution jointly determine end-to-end performance, and critically synthesize the trade-offs underlying existing optimization directions. This perspective clarifies where speculative gains are created, where they are lost, and why isolated improvements do not necessarily translate into practical speedup. We further identify the unresolved challenges that continue to limit SD and outline promising directions toward more efficient and scalable LLM inference.