How to Fix Mojibake and Unicode Encoding Errors

Mojibake (文字化け) is a Japanese term meaning "character transformation" — it describes the garbled text that appears when a string is interpreted using the wrong character encoding. Instead of "résumé" you see "résumé". Instead of a smart apostrophe you see "’". The characters are not corrupted at the byte level — the bytes are fine. The problem is that the software reading them is using the wrong rulebook to translate bytes into characters.

The byte-to-character translation problem

Every character encoding is a mapping between numbers (bytes) and characters. The letter "e" is byte 0x65 in virtually every encoding. But an accented "é" is represented differently depending on the encoding: in UTF-8 it is the two-byte sequence 0xC3 0xA9, while in Latin-1 it is the single byte 0xE9. When a program reads the UTF-8 bytes 0xC3 0xA9 using Latin-1 rules, it produces two separate characters — Ã and © — instead of the single character é. That substitution is mojibake.

• é becomes é — UTF-8 bytes read as Latin-1 (the most common pattern)
• ü becomes ü — same mismatch for German u-umlaut
• ’ (right single quote) becomes ’ — UTF-8 smart quote misread as Latin-1
• – (en dash) becomes – — common in text pasted from Word or PDF
• Unicode replacement character — a valid character was forced into a context that cannot represent it

Encoding errors are a system boundary problem. They occur when text crosses a boundary — between a file and an application, between a database and an API, between a web server and a browser — and the sending side and receiving side disagree about which encoding the text uses. In a world where UTF-8 is the universal standard, these errors should be rare. But they are common because legacy systems, Windows tools, and certain databases still use older encodings by default.

The most common sources

• CSV files from Excel — Excel saves CSV as Windows-1252 on Windows by default, not UTF-8. When opened on Linux or in Python, the accented characters garble.
• MySQL databases with latin1 charset — older MySQL installations default to `latin1` for column character sets. Storing UTF-8 data in them causes mojibake on read.
• Email headers and bodies — email clients that do not declare a charset, or that misread a charset declaration, produce mojibake in From, Subject, and body fields.
• PDF text extraction — PDF files embed fonts with custom glyph mappings that do not always correspond to standard Unicode codepoints, producing garbled output when copied.
• HTTP responses without Content-Type charset — a server that sends `Content-Type: text/html` without `; charset=utf-8` lets the browser guess, and it often guesses wrong.

UTF-8 vs Windows-1252 — the most frequent mismatch

Windows-1252 (also called CP1252) is a superset of Latin-1 developed by Microsoft for Western European languages. It uses a single byte per character and covers 256 code points — enough for English, French, German, Spanish, and Portuguese characters. UTF-8 uses one to four bytes per character and covers all 1.1 million Unicode code points. When UTF-8 text is read as Windows-1252, the multi-byte sequences produce the characteristic two- or three-character mojibake patterns. The reverse — Windows-1252 text read as UTF-8 — produces replacement characters (□ or ?) because the single high bytes are not valid UTF-8 sequences.

Text without encoding metadata is not text — it is a sequence of bytes waiting to be misinterpreted.

Detecting an encoding problem is usually straightforward — the visual appearance of mojibake is distinctive enough to identify the mismatch pattern. But for programmatic detection, or for text that looks correct but contains invisible problems, you need specific techniques.

Visual pattern recognition

The most reliable diagnostic is the mojibake pattern itself. UTF-8 text read as Latin-1 produces a characteristic "Ã" prefix followed by a second character — for example, é for é, à for à, ü for ü. Smart typographic punctuation from Word or rich text editors produces sequences starting with †— for example, ’ for ' and “ for ". If you see these sequences, the fix is to re-encode the bytes from Latin-1 back to UTF-8.

Programmatic encoding detection

For files and data streams where you cannot see the content visually, use the `chardet` library (Python) or the `jschardet` package (Node.js) to auto-detect encoding from byte patterns. These libraries analyse the frequency and distribution of byte values to identify the most likely encoding. They are not 100% accurate — short strings and strings with few non-ASCII characters are harder to classify confidently — but they handle the common cases well. Use the Unicode Character Lookup to inspect individual codepoints and their expected byte representations when debugging specific characters.

The fastest way to fix mojibake text is the Unicode and Encoding Repair Tool. It runs entirely in your browser — no server upload, no registration — and handles the most common UTF-8/Latin-1 mismatches, smart quote garbling, and invisible character removal automatically.

Different encoding mismatches produce different visual patterns. Recognising the pattern tells you both the cause and the correct fix — which encoding pair to apply, or which repair operation to run.

The double-encoding problem

A particularly damaging variant is double mojibake — where text has been mis-encoded twice. This produces sequences like © instead of ©, or Æ¢ instead of ¢. It happens when already-mojibaked text is saved and then read with the wrong encoding a second time. The repair process requires two rounds of re-encoding in reverse order. The Unicode and Encoding Repair Tool detects and handles the most common double-encoding patterns automatically.

The long-term fix for mojibake is to enforce UTF-8 at every boundary in your system — file storage, database, API, and web layer. These are the specific places where encoding declarations must be explicit.

Databases — declare UTF-8 everywhere

In MySQL and MariaDB, set the default character set to `utf8mb4` (not `utf8` — MySQL's `utf8` only covers 3-byte sequences and excludes emoji and rare Unicode characters). Set it at three levels: the server default in `my.cnf`, the database charset, and individual column charsets. In PostgreSQL, the default is already UTF-8 for new installations. In SQLite, text is always UTF-8 by default.

-- MySQL: set all levels to utf8mb4
ALTER DATABASE mydb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE users CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

Files — always save as UTF-8

When saving text files, CSV exports, or configuration files, always choose UTF-8 as the encoding. In VS Code, the encoding is shown in the status bar — click it to change. In Excel, export CSV using "CSV UTF-8 (Comma delimited)" rather than the default "CSV (Comma delimited)". For Python scripts that write files, always pass `encoding='utf-8'` to the `open()` call explicitly rather than relying on the system default. Clean text after pasting from external sources using the Email Text Cleaner which strips invisible characters and normalises whitespace automatically.

APIs and HTTP — declare charset in Content-Type

Every HTTP response that contains text must include a charset declaration: `Content-Type: application/json; charset=utf-8` or `Content-Type: text/html; charset=utf-8`. Without it, the client guesses — and older HTTP clients default to Latin-1 for `text/html` responses. For HTML pages, also add `<meta charset="UTF-8">` as the first element inside `<head>`. For JSON APIs, `Content-Type: application/json` implies UTF-8 per the RFC, but being explicit eliminates ambiguity and prevents issues with non-conforming clients.

Encoding checklist by context

• Python files: add `# -*- coding: utf-8 -*-` header for Python 2; in Python 3 this is unnecessary but harmless
• MySQL connection strings: include `charset=utf8mb4` in the DSN — e.g. `mysql+pymysql://user:pass@host/db?charset=utf8mb4`
• Node.js file reads: pass `encoding: "utf8"` to `fs.readFile()` or use `Buffer.from(data).toString("utf8")`
• XML and HTML documents: always include `&lt;?xml version="1.0" encoding="UTF-8"?&gt;` and `&lt;meta charset="UTF-8"&gt;`
• Email sending libraries: set `Content-Type: text/plain; charset=utf-8` and `Content-Transfer-Encoding: 8bit` or `quoted-printable`

Beyond the classic UTF-8/Latin-1 mismatch, two other Unicode issues produce subtle bugs that are harder to detect: normalisation mismatches and invisible characters. Both can cause string comparisons to fail, database lookups to miss, and search results to be incomplete — even when the text looks identical on screen.

Unicode normalisation forms

Unicode allows the same visible character to be represented in multiple ways. The letter "é" can be encoded as a single precomposed code point (U+00E9) or as the combination of "e" (U+0065) followed by a combining acute accent (U+0301). Both look identical on screen but are different byte sequences that fail string equality comparisons. This is why searching a database for "résumé" sometimes misses results — the stored text uses a different normalisation form. NFC (Canonical Decomposition, followed by Canonical Composition) is the recommended normal form for most applications. The Unicode and Encoding Repair Tool normalises text to NFC as part of its repair process.

Invisible and zero-width characters

Zero-width characters are Unicode code points that take up space in a string but render as nothing visible. They are frequently inserted by word processors, chat applications, and copy-paste operations from PDFs or web pages. Common culprits include the zero-width space (U+200B), zero-width non-breaking space (U+FEFF, also the byte order mark), and various bidirectional text control characters (U+200E, U+200F, U+202A–U+202E). These characters break regex patterns, confuse tokenisers, and cause identical-looking strings to compare as unequal. Use the Unicode Character Lookup to identify suspicious codepoints in a string.