A program is scanned to break up the source character stream into tokens. Despite being
different in form, many tokens have the same meaning. For example, all the three tokens
It is for this reason that the description of the lexical structure of programs deals with such terms as "numeric literal" and "word literal" rather that "number" and "word".
Besides, a token like "character string literal" represents a sequence of characters rather that a single syntax entity.
Thus, when describing the syntax, we assume the token stream to have been "normalized", each
token having been reduced to its
In addition we assume each character string literal to have been broken up into the string of separate tokens, a token representing a single character.
The above enables us to describe the syntax in terms of syntax "entities" rather than "representations of syntax entities".
Here is the correspondence between the source tokens and the normalized tokens:
The