One emoji, length five
There is one thing on screen: a family emoji. Ask Python how long it is.
>>> len("👨👩👧")
5Ask JavaScript and it says 8. Count the bytes and there are 18. And a person looking at the screen would say 1.
None of them is wrong. They are counting four different things.
Its code points, in order
- 👨
U+1F468 U+200Djoiner (invisible)- 👩
U+1F469 U+200Djoiner (invisible)- 👧
U+1F467
You see 1 character. It is made of 5 code points. The pieces are separate entries in the Unicode table. The screen draws them as one.
- On screen
- 1
- Python len()
- 5
- JavaScript .length
- 8
- Bytes in UTF-8
- 18
Layer 1: code points
Computers store text as numbers. Unicode is the agreed table that assigns a number to every character in every writing system, plus symbols and emoji. Each entry is a code point, written like U+0041 for the letter A.
There are more than a million possible code points and around 150,000 are in use.
The surprise is that what you see as one character is often several code points.
Example
Taking the family apart
The family emoji is five code points in a row:
- Man
U+1F468 - Zero-width joiner
U+200D - Woman
U+1F469 - Zero-width joiner
U+200D - Girl
U+1F467
The zero-width joiner is invisible. It is an instruction to the screen: "draw the things on either side of me as one picture". Three people and two invisible joiners: five.
Other things built the same way:
- Skin tones. A thumbs-up followed by a skin tone code point.
- Flags. Two "regional letter" code points. The flag of India is the letters I and N.
- Accents. "é" can be one code point, or a plain "e" followed by a separate accent that attaches to it. They look identical.
Layer 2: how code points are stored
A code point is a number, and it still has to be written as bytes. There are several ways, called encodings.
UTF-8 uses 1 to 4 bytes per code point. Plain English letters take 1 byte, which is why it is compatible with older text. Most accented letters take 2, most Asian scripts 3, emoji 4. It is the standard for files and the web.
UTF-16 uses units of 2 bytes. Common characters take one unit. Code points above U+FFFF, which includes almost all emoji, take two units, called a surrogate pair.
That explains every number:
| What is counted | Family emoji | Used by |
|---|---|---|
| What a person sees (grapheme clusters) | 1 | Swift, and screen cursors |
| Code points | 5 | Python len() |
| UTF-16 units | 8 | JavaScript .length, Java length() |
| UTF-8 bytes | 18 | File sizes, network limits, Go len() |
Where it goes wrong
Cutting text to a length. "Truncate to 10 characters" can slice through the middle of an emoji or an accented letter and leave a broken symbol.
Reversing a string. Reverse the code points of the family emoji and the joiners end up in the wrong places. Reverse "é" in its two-piece form and the accent lands on a different letter.
Comparing text that looks the same.
>>> "é" == "é" # é as one piece, é as two
FalseTwo strings that are indistinguishable on screen are not equal, so a search fails or a username can be registered twice. The fix is to normalize both to the same form before comparing:
import unicodedata
unicodedata.normalize("NFC", "é") == "é" # TrueLength limits. A database column of 20 "characters", a text message of 160, a form field: each means something different, and an emoji can use up several.
What to do
Decide what you are counting
Bytes for storage and network limits. What the user sees for anything shown on screen.
Use a library for user-visible characters
Splitting text into what people see as characters follows long rules. Languages provide it:
Intl.Segmenterin JavaScript, for example.Normalize before comparing or storing
Especially for usernames, search and anything used as a key.
Use UTF-8 everywhere you can choose
And always say which encoding a file or message uses.
Key takeaways
- Unicode gives every character a number called a code point.
- One visible character can be several code points: emoji with joiners, flags, letters with accents.
- Code points are stored with an encoding. UTF-8 uses 1 to 4 bytes each, UTF-16 uses 1 or 2 units.
- "Length" means code points in Python, UTF-16 units in JavaScript and Java, and bytes on disk.
- Cutting, reversing and comparing text by code point can break what the user sees.