Lexical analysis
Introducido
Vocabulario
| English | Significado |
|---|---|
| tokens/ˈtəʊkənz/ | Lexical units such as identifiers, keywords, literals and operators produced by a lexer. |
English
Characters in, tokens out
- Each token class is a regex; the lexer's DFA runs over the source with greedy longest match and emits (type, lexeme, position). Whitespace and comments are discarded; positions are kept for errors.
{exercise:uni-cs1104-tokens}
Práctica
The lexer emits —
Dificultad 1/5 · 100 monedas + un pequeño bono en el primer intento de hoy
Type for the parser, lexeme for messages, position for errors.
English
Why longest match
- The regexes
if,ident,numoverlap on prefixes: greedy longest match resolvesifxas one IDENT, notif+x— tie-broken by priority. Get this wrong and every variable starting withifbreaks. - The course line
count = a1 + b2;lexes to IDENT(count) ASSIGN IDENT(a1) PLUS IDENT(b2) SEMI at columns 1, 7, 9, 12, 14, 16.
{exercise:uni-cs1104-longest}
Práctica
The input ifx lexes as —
Dificultad 1/5 · 100 monedas + un pequeño bono en el primer intento de hoy
Greedy longest match: the ident regex permits the longer lexeme.
English
Error policy is specification
12abclexes greedily as NUM(12) then IDENT(abc): legal in some languages, an error in others. Ours reports it — the error policy is part of the lexer's specification, not an afterthought.- Discard positions and every later diagnostic degrades: positions flow from the lexer through the parser into semantic errors.
{exercise:uni-cs1104-policy}
Práctica
The course line count = a1 + b2; lexes to — tokens
Dificultad 1/5 · 100 monedas + un pequeño bono en el primer intento de hoy
IDENT ASSIGN IDENT PLUS IDENT SEMI - six.