Lexical analysis
Introduit
Vocabulaire
| English | Sens |
|---|---|
| tokens/ˈtəʊkənz/ | tokens |
English
Characters in, tokens out
- Each token class is a regex; the lexer's DFA runs over the source with greedy longest match and emits (type, lexeme, position). Whitespace and comments are discarded; positions are kept for errors.
{exercise:uni-cs1104-tokens}
Pratique
The lexer emits —
Difficulté 1/5 · 100 pièces + un petit bonus lors de la première réussite aujourd'hui
Type for the parser, lexeme for messages, position for errors.
English
Why longest match
- The regexes
if,ident,numoverlap on prefixes: greedy longest match resolvesifxas one IDENT, notif+x— tie-broken by priority. Get this wrong and every variable starting withifbreaks. - The course line
count = a1 + b2;lexes to IDENT(count) ASSIGN IDENT(a1) PLUS IDENT(b2) SEMI at columns 1, 7, 9, 12, 14, 16.
{exercise:uni-cs1104-longest}
Pratique
The input ifx lexes as —
Difficulté 1/5 · 100 pièces + un petit bonus lors de la première réussite aujourd'hui
Greedy longest match: the ident regex permits the longer lexeme.
English
Error policy is specification
12abclexes greedily as NUM(12) then IDENT(abc): legal in some languages, an error in others. Ours reports it — the error policy is part of the lexer's specification, not an afterthought.- Discard positions and every later diagnostic degrades: positions flow from the lexer through the parser into semantic errors.
{exercise:uni-cs1104-policy}
Pratique
The course line count = a1 + b2; lexes to — tokens
Difficulté 1/5 · 100 pièces + un petit bonus lors de la première réussite aujourd'hui
IDENT ASSIGN IDENT PLUS IDENT SEMI - six.