chardetng

crates.io docs.rs Apache 2 / MIT dual-licensed

A character encoding detector for legacy Web content.

Licensing

Please see the file named COPYRIGHT.

Documentation

Generated API documentation is available online.

Purpose

The purpose of this detector is user retention for Firefox by ensuring that the long tail of the legacy Web is not more convenient to use in Chrome than in Firefox. (Chrome deployed ced, which left Firefox less convenient to use until the deployment of this detector.)

About the Name

chardet was the name of Mozilla's old encoding detector. I named this one chardetng, because this the next generation of encoding detector in Firefox. There is no code reuse from the old chardet.

Optimization Goals

This crate aims to be more accurate than ICU, more complete than chardet, more explainable and modifiable than compact_enc_det (aka. ced), and, in an application that already depends on encoding_rs for other reasons, smaller in added binary footprint than compact_enc_det.

Principle of Operation

In general chardetng prefers to do negative matching (rule out possibilities from the set of plausible encodings) than to do positive matching. Since negative matching is insufficient, there is positive matching, too.

Notes About Encodings

UTF-8
Detected only if explicitly permitted by the argument to the `guess` method. It's harmful for Web browsers to detect UTF-8 without requiring user action, such as choosing a menu item, because Web developers would start relying on the detection.
UTF-16[BE|LE]
Not detected: Detecting these belongs on the BOM layer.
x-user-defined
Not detected: This encoding is for XHR. `` in HTML is not unlabeled and means windows-1252.
Replacement
Not detected.
GB18030
Detected as GBK.
GBK
Big5
EUC-KR
Shift_JIS
windows-1250
windows-1251
windows-1252
windows-1253
windows-1254
windows-1255
windows-1256
windows-1257
windows-1258
windows-874
ISO-8859-2
ISO-8859-7
Detected: Historical locale-specific fallbacks.
EUC-JP
ISO-2022-JP
KOI8-U
ISO-8859-5
IBM866
Detected: Detected by multiple browsers past and present.
KOI8-R
Detected as KOI8-U. (Always guessing the U variant is less likely to corrupt non-box drawing characters.)
ISO-8859-8-I
Detected as windows-1255.
ISO-8859-4
Detected: Detected by IE and Chrome; in menu in IE and Firefox.
ISO-8859-6
Detected: Detected by IE and Chrome.
ISO-8859-8
Detected: Available in menu in IE and Firefox.
ISO-8859-13
Detected: Detected by Chrome. This encoding is so similar to windows-1257 that menu items for windows-1257 can be considered to accommodate this one in IE and Firefox. Due to the mechanics of this detector, if this wasn't included as a separate item, the windows-1257 detection wouldn't catch the cases that use curly quotes and are invalid as windows-1257.
x-mac-cyrillic
Not detected: Not detected by IE and Chrome. (Was previously detected by Firefox.)
ISO-8859-3
ISO-8859-10
ISO-8859-14
ISO-8859-15
ISO-8859-16
macintosh
Not detected: These encodings have never been a locale-specific fallback in a major browser or a menu item in IE.

Of the detected encodings, ISO-8859-5, ISO-8859-6, and ISO-8859-4 are the ones that something else is the most likely to be misdetected as, and, I believe, the three least-used encodings of the ones detected, so these three are the most likely ones to either be removed or downplayed.

Known Problems

Roadmap

Release Notes

0.1.1

0.1.0