Character codes — ASCII and Unicode

The email that arrived as gibberish

• Through the 1990s, a message typed in Warsaw and read in Tokyo often arrived as a wall of nonsense. Both computers stored eight bits per character. They simply disagreed about what the top 128 patterns meant.
• Poland's code page put Polish letters there. Japan's put its own characters there. Nothing was corrupted in transit: the bytes arrived intact and were read against a different table.
• The fix was to stop having tables per country and give every character in every script one number, for ever. That is Unicode 统一码, and it is why the emoji you send arrives as the same picture.
• This lesson is how text becomes numbers: character sets 字符集, ASCII, Unicode, and the encodings that store them.

A character set gives each character a number

• A computer stores no letters, only numbers. A character set is the set of characters a computer can represent, each with its own binary code.
• The number for one character is its code point 码点. A is 65, a is 97, 0 the digit is 48.
• The code point is not the character's meaning; it is an agreed label. Text is readable only because both machines use the same character set.

The letter is the picture; the byte is the number

ASCII

• ASCII ASCII码, the American Standard Code for Information Interchange, uses 7 bits, giving $2^7 = 128$ code points: the basic Latin letters, digits, punctuation, and 32 control codes such as carriage return.
• Extended ASCII uses 8 bits, giving 256 code points. The lower 128 are identical to ASCII; the upper 128 vary by region, which is exactly what broke that email.
• You are never asked to memorise a code, but you are asked for the counts: 7 bits, 128; 8 bits, 256.

Unicode

• Unicode is a universal character set: one code point for almost every character in every script alive or dead, plus symbols and emoji, over 149,000 of them.
• The code point is separate from how it is stored. An encoding 编码 turns a code point into bytes.
• UTF-8 UTF-8编码 uses 1 to 4 bytes per character and is ASCII-compatible: the first 128 code points are one byte, identical to ASCII. UTF-16 uses 2 or 4 bytes; UTF-32 uses a fixed 4.

Worked example: compare ASCII and Unicode

Worked example: the size of a text file

Reading a character in binary and hex

• Because a code point is just a number, it converts like any other. A is 65, which is 0100 0001 in binary and 41 in hexadecimal.
• Two useful patterns: the digits 0 to 9 run from 48, and lower case is 32 more than upper case, so a (97) is A (65) plus 32. That difference is a single bit.
• A question that gives you A = 65 and asks for E wants $65 + 4 = 69$, not a memorised table.

Marks that slip away

You've got it