Lexical analysis
Introduced
Vocabulary
| English | Meaning |
|---|---|
| tokens/ˈtəʊkənz/ | Lexical units such as identifiers, keywords, literals and operators produced by a lexer. |
Characters in, tokens out
- Each token class is a regex; the lexer's DFA runs over the source with greedy longest match and emits (type, lexeme, position). Whitespace and comments are discarded; positions are kept for errors.
{exercise:uni-cs1104-tokens}
Practice
The lexer emits —
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Type for the parser, lexeme for messages, position for errors.
Why longest match
- The regexes
if,ident,numoverlap on prefixes: greedy longest match resolvesifxas one IDENT, notif+x— tie-broken by priority. Get this wrong and every variable starting withifbreaks. - The course line
count = a1 + b2;lexes to IDENT(count) ASSIGN IDENT(a1) PLUS IDENT(b2) SEMI at columns 1, 7, 9, 12, 14, 16.
{exercise:uni-cs1104-longest}
Practice
The input ifx lexes as —
Difficulty 1/5 · 100 coins + a small bonus on first completion today
Greedy longest match: the ident regex permits the longer lexeme.
Error policy is specification
12abclexes greedily as NUM(12) then IDENT(abc): legal in some languages, an error in others. Ours reports it — the error policy is part of the lexer's specification, not an afterthought.- Discard positions and every later diagnostic degrades: positions flow from the lexer through the parser into semantic errors.
{exercise:uni-cs1104-policy}
Practice
The course line count = a1 + b2; lexes to — tokens
Difficulty 1/5 · 100 coins + a small bonus on first completion today
IDENT ASSIGN IDENT PLUS IDENT SEMI - six.