Lexical analysis

مفردات

English	المعنى
tokens/ˈtəʊkənz/	Lexical units such as identifiers, keywords, literals and operators produced by a lexer.

English

Characters in, tokens out

Each token class is a regex; the lexer's DFA runs over the source with greedy longest match and emits (type, lexeme, position). Whitespace and comments are discarded; positions are kept for errors.

{exercise:uni-cs1104-tokens}

تدريب

The lexer emits —

الصعوبة 1/5 · 100 عملة + مكافأة صغيرة عند الإنجاز الأول اليوم

English

The regexes if, ident, num overlap on prefixes: greedy longest match resolves ifx as one IDENT, not if + x — tie-broken by priority. Get this wrong and every variable starting with if breaks.
The course line count = a1 + b2; lexes to IDENT(count) ASSIGN IDENT(a1) PLUS IDENT(b2) SEMI at columns 1, 7, 9, 12, 14, 16.

{exercise:uni-cs1104-longest}

تدريب

The input ifx lexes as —

الصعوبة 1/5 · 100 عملة + مكافأة صغيرة عند الإنجاز الأول اليوم

English

12abc lexes greedily as NUM(12) then IDENT(abc): legal in some languages, an error in others. Ours reports it — the error policy is part of the lexer's specification, not an afterthought.
Discard positions and every later diagnostic degrades: positions flow from the lexer through the parser into semantic errors.

{exercise:uni-cs1104-policy}

تدريب

The course line count = a1 + b2; lexes to — tokens

الصعوبة 1/5 · 100 عملة + مكافأة صغيرة عند الإنجاز الأول اليوم