Techniques for the construction of small and fast lexical analyzers

Abstract

The paper discusses two major issues in the construction of table-driven lexical analyzers. It first examines an encoding of FSM state actions which allows the system to be truly table-driven with little or no program modification required to change the FSM being modeled. This encoding makes use of the knowledge that these actions are typically drawn from a reasonably small set. The second issue involves the storage of the "next-state" or transition table used by almost all general purpose scanning systems. A fortuitous encoding of FSM states can result in large savings in space with little cost in time. These techniques can be combined with the standard automata-theoretic approach to yield efficient analyzers.Previous results have shown that the speed of compilation is heavily influenced by the speed of the lexical analyzer. Therefore, these techniques could be used to improve the speed of new or already existing compilers.