Lexical analysis
Introduzido
Vocabulário
| English | Significado |
|---|---|
| tokens/ˈtəʊkənz/ | tokens |
English
Characters in, tokens out
- Each token class is a regex; the lexer's DFA runs over the source with greedy longest match and emits (type, lexeme, position). Whitespace and comments are discarded; positions are kept for errors.
{exercise:uni-cs1104-tokens}
Prática
The lexer emits —
Dificuldade 1/5 · Moedas 100 + pequeno bônus na primeira conclusão hoje
Type for the parser, lexeme for messages, position for errors.
English
Why longest match
- The regexes
if,ident,numoverlap on prefixes: greedy longest match resolvesifxas one IDENT, notif+x— tie-broken by priority. Get this wrong and every variable starting withifbreaks. - The course line
count = a1 + b2;lexes to IDENT(count) ASSIGN IDENT(a1) PLUS IDENT(b2) SEMI at columns 1, 7, 9, 12, 14, 16.
{exercise:uni-cs1104-longest}
Prática
The input ifx lexes as —
Dificuldade 1/5 · Moedas 100 + pequeno bônus na primeira conclusão hoje
Greedy longest match: the ident regex permits the longer lexeme.
English
Error policy is specification
12abclexes greedily as NUM(12) then IDENT(abc): legal in some languages, an error in others. Ours reports it — the error policy is part of the lexer's specification, not an afterthought.- Discard positions and every later diagnostic degrades: positions flow from the lexer through the parser into semantic errors.
{exercise:uni-cs1104-policy}
Prática
The course line count = a1 + b2; lexes to — tokens
Dificuldade 1/5 · Moedas 100 + pequeno bônus na primeira conclusão hoje
IDENT ASSIGN IDENT PLUS IDENT SEMI - six.