# Unicode and Character Encoding

Unicode assigns characters standardized code points, while encodings such as UTF-8 specify how those values are represented as bytes.

- Type: framework
- Domain: Engineering & architecture
- Level: Intermediate
- Last reviewed: 2026-10-19
- Original: [Soroush’s Product Wiki](<https://wiki.soroush.uk/topic/unicode-and-character-encoding/>)

## What it means

Unicode provides a common character repertoire and associated properties. A character encoding maps text to a byte representation; UTF-8 is a variable-length Unicode encoding, not a separate language or font. Bytes, code units, code points, and user-perceived characters are different counting units, especially for combining marks and emoji.

## Why it matters

Encoding mismatches can corrupt names, searches, and imported records. Confusing byte length with visible character length can truncate text or reject legitimate input, particularly when a product expands beyond a single writing system.

## When to use it

Apply these distinctions anywhere text is stored, transmitted, imported, searched, or length-limited. Prefer a consistent encoding through the full pipeline and declare it correctly. Choosing UTF-8 does not by itself provide correct fonts, sorting, directionality, or translation.

## How it works

1. Identify the actual encoding at every input, storage, and output boundary.
2. Use a consistent Unicode-capable pipeline and matching declarations.
3. Decode and encode explicitly when exchanging legacy formats.
4. Choose the correct counting and comparison behavior for each product rule.
5. Test round trips with accented names, mixed scripts, and multi-code-point characters.

## A practical example

A profile form allows forty visible characters, but its backend truncates after forty bytes. Several Persian names are shortened or corrupted. The team separates storage byte limits from the user-facing length rule and tests that complete names survive save and reload.

## Where it breaks

Unicode does not eliminate visually similar characters or language-dependent comparison rules. Normalization and grapheme segmentation need appropriate libraries and explicit product choices. An incorrect encoding label can still produce unreadable text even when the original bytes are intact.

## Common mistakes

- Assuming one byte equals one character.
- Changing an encoding declaration without converting the underlying bytes.
- Expecting Unicode support to solve font coverage and text direction automatically.

## Sources

1. [FAQ - UTF-8, UTF-16, UTF-32 & BOM](<https://www.unicode.org/faq/utf_bom.html>) — Unicode Consortium · Standards guidance
2. [Character encodings for beginners](<https://www.w3.org/International/questions/qa-what-is-encoding>) — W3C · Technical guidance

## Related topics

- [Internationalization \(i18n\)](<https://wiki.soroush.uk/topic/internationalization/>)
- [Localization \(l10n\)](<https://wiki.soroush.uk/topic/localization/>)
- [Form Validation](<https://wiki.soroush.uk/topic/form-validation/>)
