Natural language detection for Rust with focus on simplicity and performance.
Add to you Cargo.toml:
```
[dependencies]
whatlang = "0.12.0" ```
Example:
```rust extern crate whatlang;
use whatlang::{detect, Lang, Script};
fn main() { let text = "Ĉu vi ne volas eklerni Esperanton? Bonvolu! Estas unu de la plej bonaj aferoj!";
let info = detect(text).unwrap();
assert_eq!(info.lang(), Lang::Epo);
assert_eq!(info.script(), Script::Latin);
assert_eq!(info.confidence(), 1.0);
assert!(info.is_reliable());
} ```
For more details (e.g. how to blacklist some languages) please check the documentation.
| Feature | Description |
|------------|---------------------------------------------------------------------------------------|
| enum-map | Lang and Script implement Enum trait from enum-map |
The algorithm is based on the trigram language models, which is a particular case of n-grams. To understand the idea, please check the original whitepaper Cavnar and Trenkle '94: N-Gram-Based Text Categorization'.
is_reliable calculated?It is based on the following factors:
* How many unique trigrams are in the given text
* How big is the difference between the first and the second(not returned) detected languages? This metric is called rate in the code base.
Therefore, it can be presented as 2d space with threshold functions, that splits it into "Reliable" and "Not reliable" areas. This function is a hyperbola and it looks like the following one:
For more details, please check a blog article Introduction to Rust Whatlang Library and Natural Language Identification Algorithms.
This is mostly useful to test performance optimizations.
cargo bench
| | Whatlang | CLD2 | CLD3 | | ------------------------- | ---------- | ----------- | -------------- | | Implementation language | Rust | C++ | C++ | | Languages | 87 | 83 | 107 | | Algorithm | trigrams | quadgrams | neural network | | Supported Encoding | UTF-8 | UTF-8 | ? | | HTML support | no | yes | ? |
You can support the project by donating NEAR tokens.
Our NEAR wallet address is whatlang.near
Whatlang is a derivative work from Franc (JavaScript, MIT) by Titus Wormer.