Syntax of Tokens as Character Strings


token --> name
| natural-number
| unsigned-float
| variable
| string
| punctuation-char
| space
| comment
| full-stop

name --> quoted-name
| word
| symbol
| solo-char
| [ ?layout-char... ]
| { ?layout-char... }

quoted-name --> ' ?quoted-item... '

quoted-item --> char {other than ' or \}
| ''
| \ escape-sequence {unless character escapes have been switched off}

word --> small-letter ?alpha...

symbol --> symbol-char... {except in the case of a full-stop or where the first 2 chars are /* }

natural-number --> digit...
| base ' alphanumeric... {where each alphanumeric must be less than base; count a as 10, b as 11, etc.}
| zero ' quoted-item {This yields the ASCII equivalent of quoted-item}

base --> digit... {must be in the range 0..36}

zero --> 0
unsigned-float --> simple-float
| simple-float E exponent

simple-float -->
digit... decimal-point digit...

decimal-point --> .

E --> E | e

exponent --> digit...
| - digit...
| + digit...

variable --> underline ?alpha...

variable --> capital-letter ?alpha..

string --> " ?string-item... "

string-item --> char {other than " or \}
| ""
| \ escape-sequence {unless character escapes have been switched off}

escape-sequence -->b {backspace, character code 8}
| t {horizontal tab, character code 9}
| n {newline, character code 10}
| v {vertical tab, character code 11}
| f {form feed, character code 12}
| r {carriage return, character code 13}
| e {escape, character code 27}
| d {delete, character code 127}
| ^? {delete, character code 127}
| a {alarm, character code 7}
| xCD {character code hex CD, 2 digits}
| Oct {character code octal Oct, up to 3 digits}
| ^letter {the control character letter mod 32}
| c?layout-char... {ignored}
| layout-char {ignored}
| char {other than the above, represents itself}

space --> layout-char...

comment --> /* ?char... */ {where ?char... must not contain */ }
| % rest-of-line

rest-of-line --> newline
| ?not-end-of-line... newline

not-end-of-line --> {any character except newline}

newline --> {ASCII code 10}

full-stop --> . layout-char


char --> layout-char
| alpha
| symbol-char
| solo-char
| punctuation-char
| quote-char

layout-char --> {ASCII codes 1..32 and 127 -- includes space, tab, newline, and del}

alpha --> alphanumeric | underline

alphanumeric --> letter | digit

letter --> capital-letter | small-letter

capital-letter --> A | B | C | D | E | F | G | H | I | J | K | L | M | N | O | P | Q | R | S | T | U | V | W | X | Y | Z

small-letter --> a | b | c | d | e | f | g | h | i | j | k | l | m | n | o | p | q | r | s | t | u | v | w | x | y | z

digit --> 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9

symbol-char --> + | - | * | / | \ | ^ | < | > | = | ` | ~ | : | . | ? | @ | # | $ | &

solo-char --> ; | !

punctuation-char --> ( | ) | [ | ] | { | } | , | | | %

quote-char --> ' | "

underline --> _