Lexical tokens

A program is a sequence of lexical tokens, optionally separated by whitespace. A token is either an atom, a number, or any of the various keywords and punctuation symbols that appear in typewriter font in the grammar expressions that follow.

An atom is

• A string consisting of alphanumeric characters and the characters, and the characters ``$'' and ``_'', beginning with an alphabetic character, or
• A string containing any character except the space character, delimited by an initial backslash (``$\backslash$'') and a final space character. The delimiters do not count as part of the atom.

As an example ``foo_123'' is an atom. It is exactly equivalent to ``$\backslash$foo_123 '' (note the terminating space character). Using backslash and space as delimiters allows any character (including backslash, but excluding space) to be included in an atom.

A number is a string of digits. Whitespace is any string of space characters, tab characters and newline characters.