Pointer Walk

Unicode and String Length

One emoji on screen, length five. Code points, joiners, and why length is not what you see.

Languages and RuntimesBeginner8 min read

One emoji, length five

There is one thing on screen: a family emoji. Ask Python how long it is.

python
>>> len("👨‍👩‍👧")
5

Ask JavaScript and it says 8. Count the bytes and there are 18. And a person looking at the screen would say 1.

None of them is wrong. They are counting four different things.

What is the length of this?Interactive
Text
👨‍👩‍👧

Its code points, in order

  1. 👨U+1F468
  2. U+200Djoiner (invisible)
  3. 👩U+1F469
  4. U+200Djoiner (invisible)
  5. 👧U+1F467

You see 1 character. It is made of 5 code points. The pieces are separate entries in the Unicode table. The screen draws them as one.

On screen
1
Python len()
5
JavaScript .length
8
Bytes in UTF-8
18

Layer 1: code points

Computers store text as numbers. Unicode is the agreed table that assigns a number to every character in every writing system, plus symbols and emoji. Each entry is a code point, written like U+0041 for the letter A.

There are more than a million possible code points and around 150,000 are in use.

The surprise is that what you see as one character is often several code points.

Example

Taking the family apart

The family emoji is five code points in a row:

  1. Man U+1F468
  2. Zero-width joiner U+200D
  3. Woman U+1F469
  4. Zero-width joiner U+200D
  5. Girl U+1F467

The zero-width joiner is invisible. It is an instruction to the screen: "draw the things on either side of me as one picture". Three people and two invisible joiners: five.

Other things built the same way:

  • Skin tones. A thumbs-up followed by a skin tone code point.
  • Flags. Two "regional letter" code points. The flag of India is the letters I and N.
  • Accents. "é" can be one code point, or a plain "e" followed by a separate accent that attaches to it. They look identical.

Layer 2: how code points are stored

A code point is a number, and it still has to be written as bytes. There are several ways, called encodings.

UTF-8 uses 1 to 4 bytes per code point. Plain English letters take 1 byte, which is why it is compatible with older text. Most accented letters take 2, most Asian scripts 3, emoji 4. It is the standard for files and the web.

UTF-16 uses units of 2 bytes. Common characters take one unit. Code points above U+FFFF, which includes almost all emoji, take two units, called a surrogate pair.

That explains every number:

What is countedFamily emojiUsed by
What a person sees (grapheme clusters)1Swift, and screen cursors
Code points5Python len()
UTF-16 units8JavaScript .length, Java length()
UTF-8 bytes18File sizes, network limits, Go len()

Where it goes wrong

Cutting text to a length. "Truncate to 10 characters" can slice through the middle of an emoji or an accented letter and leave a broken symbol.

Reversing a string. Reverse the code points of the family emoji and the joiners end up in the wrong places. Reverse "é" in its two-piece form and the accent lands on a different letter.

Comparing text that looks the same.

python
>>> "é" == "é"     # é as one piece, é as two
False

Two strings that are indistinguishable on screen are not equal, so a search fails or a username can be registered twice. The fix is to normalize both to the same form before comparing:

python
import unicodedata
unicodedata.normalize("NFC", "é") == "é"     # True

Length limits. A database column of 20 "characters", a text message of 160, a form field: each means something different, and an emoji can use up several.

What to do

  1. Decide what you are counting

    Bytes for storage and network limits. What the user sees for anything shown on screen.

  2. Use a library for user-visible characters

    Splitting text into what people see as characters follows long rules. Languages provide it: Intl.Segmenter in JavaScript, for example.

  3. Normalize before comparing or storing

    Especially for usernames, search and anything used as a key.

  4. Use UTF-8 everywhere you can choose

    And always say which encoding a file or message uses.

Key takeaways

  • Unicode gives every character a number called a code point.
  • One visible character can be several code points: emoji with joiners, flags, letters with accents.
  • Code points are stored with an encoding. UTF-8 uses 1 to 4 bytes each, UTF-16 uses 1 or 2 units.
  • "Length" means code points in Python, UTF-16 units in JavaScript and Java, and bytes on disk.
  • Cutting, reversing and comparing text by code point can break what the user sees.

Your progress

Check your progress

0%

  1. Read the page50% of this page

    When you have gone through the page and its animations, tick it off.

  2. Take the quiz50% of this page

    7 questions, from easy to hard. Take it as often as you like: the questions change, and only your best score counts.

Saved on this device only. See all your progress