Lexical analysis
도입됨
어휘
| English | 의미 |
|---|---|
| tokens/ˈtəʊkənz/ | tokens |
English
Characters in, tokens out
- Each token class is a regex; the lexer's DFA runs over the source with greedy longest match and emits (type, lexeme, position). Whitespace and comments are discarded; positions are kept for errors.
{exercise:uni-cs1104-tokens}
연습
The lexer emits —
난이도 1/5 · 100코인 + 오늘 첫 완료 시 작은 보너스 추가
Type for the parser, lexeme for messages, position for errors.
English
Why longest match
- The regexes
if,ident,numoverlap on prefixes: greedy longest match resolvesifxas one IDENT, notif+x— tie-broken by priority. Get this wrong and every variable starting withifbreaks. - The course line
count = a1 + b2;lexes to IDENT(count) ASSIGN IDENT(a1) PLUS IDENT(b2) SEMI at columns 1, 7, 9, 12, 14, 16.
{exercise:uni-cs1104-longest}
연습
The input ifx lexes as —
난이도 1/5 · 100코인 + 오늘 첫 완료 시 작은 보너스 추가
Greedy longest match: the ident regex permits the longer lexeme.
English
Error policy is specification
12abclexes greedily as NUM(12) then IDENT(abc): legal in some languages, an error in others. Ours reports it — the error policy is part of the lexer's specification, not an afterthought.- Discard positions and every later diagnostic degrades: positions flow from the lexer through the parser into semantic errors.
{exercise:uni-cs1104-policy}
연습
The course line count = a1 + b2; lexes to — tokens
난이도 1/5 · 100코인 + 오늘 첫 완료 시 작은 보너스 추가
IDENT ASSIGN IDENT PLUS IDENT SEMI - six.