Its task is simple: It tells you which language some text is written in. This is very useful as a preprocessing step for linguistic data in natural language processing applications such as text classification and spell checking. Other use cases, for instance, might include routing e-mails to the right geographically located customer service department, based on the e-mails' languages.
Language detection is often done as part of large machine learning frameworks or natural language processing applications. In cases where you don't need the full-fledged functionality of those systems or don't want to learn the ropes of those, a small flexible library comes in handy.
So far, other comprehensive open source libraries in the Rust ecosystem for this task are CLD2, Whatlang and Whichlang. Unfortunately, most of them have two major drawbacks:
Lingua aims at eliminating these problems. She nearly does not need any configuration and yields pretty accurate results on both long and short text, even on single words and phrases. She draws on both rule-based and statistical methods but does not use any dictionaries of words. She does not need a connection to any external API or service either. Once the library has been downloaded, it can be used completely offline.
Compared to other language detection libraries, Lingua's focus is on quality over quantity, that is, getting detection right for a small set of languages first before adding new ones. Currently, the following 75 languages are supported:
Lingua is able to report accuracy statistics for some bundled test data available for each supported language. The test data for each language is split into three parts:
Both the language models and the test data have been created from separate documents of the Wortschatz corpora offered by Leipzig University, Germany. Data crawled from various news websites have been used for training, each corpus comprising one million sentences. For testing, corpora made of arbitrarily chosen websites have been used, each comprising ten thousand sentences. From each test corpus, a random unsorted subset of 1000 single words, 1000 word pairs and 1000 sentences has been extracted, respectively.
Given the generated test data, I have compared the detection results of Lingua, CLD2, Whatlang and Whichlang running over the data of Lingua's supported 75 languages. Languages that are not supported by the other libraries are simply ignored for the respective library during the detection process.
Each of the following sections contains four plots. The bar plots show the detailed accuracy results for each supported language. The box plots illustrate the distributions of the accuracy values for each classifier. The boxes themselves represent the areas which the middle 50 % of data lie within. Within the colored boxes, the horizontal lines mark the median of the distributions.
The first two plots in each section show the results for all supported languages in each classifier, respectively. The last two plots are restricted to the common subset of currently 16 languages that is supported by all compared classifiers. This distinction makes sense because the first box plot creates the impression that Whichlang is the most accurate classifier, but it is not. Whichlang supports only 16 languages whereas Lingua supports 75 languages. For the second box plot, the supported languages in Whatlang and Lingua have been restricted to those 16 languages supported by Whichlang. This provides for a more accurate comparison and shows that overall, Lingua is the most accurate language detection library in this comparison.
Bar plot
Bar plot
Bar plot
Bar plot
Bar plot
Bar plot
Bar plot
Bar plot
The tables below show detailed statistics for each language and classifier including mean, median and standard deviation.
Open table
Language
Average
Single Words
Word Pairs
Sentences
Lingua
(high accuracy mode)Lingua
(low accuracy mode)Whichlang
Whatlang
CLD2
Lingua
(high accuracy mode)Lingua
(low accuracy mode)Whichlang
Whatlang
CLD2
Lingua
(high accuracy mode)Lingua
(low accuracy mode)Whichlang
Whatlang
CLD2
Lingua
(high accuracy mode)Lingua
(low accuracy mode)Whichlang
Whatlang
CLD2
Afrikaans
78 64
-
51 55 58
37 - 22
12 80 62 - 41 56
96 93 - 92 96
Albanian
87 79 - - 64 68 54 - - 18 94 86 - - 76 99 98 - - 99
Arabic
98 94 100 93 66 96 88 100 88 19 99 95 100 92 81 99 99 100 99 99
Armenian
100 100 - - 100 100 100 - - 100 100 100 - - 100 100 100 - - 100
Azerbaijani
89 82 - 79 71 77 71 - 61 34 92 78 - 81 81 99 96 - 97 99
Basque
83 74 - - 61 71 55 - - 22 87 76 - - 69 92 91 - - 91
Belarusian
96 91 - 91 75 91 80 - 81 42 99 95 - 94 87 99 99 - 99 98
Bengali
100 100 - 99 62 100 100 - 100 19 100 100 - 100 69 100 100 - 99 99
Bokmal
58 49 - 49 - 38 27 - 24 - 58 47 - 44 - 76 74 - 80 -
Bosnian
34 29 - - 18 29 22 - - 4 34 28 - - 15 40 36 - - 36
Bulgarian
86 77 - 69 65 70 56 - 44 31 91 80 - 67 72 98 96 - 96 92
Catalan
70 58 - 51 37 50 33 - 29 4 73 60 - 45 29 86 81 - 80 79
Chinese
100 100 97 100 33 100 100 93 100 - 100 100 98 100 2 100 100 100 100 98
Croatian
72 59 - 61 51 53 36 - 34 33 74 57 - 54 46 90 85 - 94 72
Czech
80 70 - 63 73 65 54 - 42 50 84 71 - 66 79 91 87 - 82 90
Danish
81 70 - 53 59 61 45 - 31 26 83 70 - 45 56 97 95 - 84 94
Dutch
77 63 73 47 47 55 35 53 22 10 80 61 70 37 41 96 94 98 82 90
English
80 62 66 49 55 54 29 34 17 11 88 62 65 35 55 99 96 98 94 99
Esperanto
83 65 - 58 50 67 43 - 32 7 85 60 - 53 45 98 92 - 90 97
Estonian
91 83 - 67 65 79 62 - 43 23 96 87 - 62 73 99 99 - 96 99
Finnish
96 90 - 76 77 90 77 - 53 43 98 94 - 78 89 99 99 - 98 99
French
89 77 68 63 51 74 51 36 35 12 94 82 70 59 47 99 97 99 97 93
Ganda
91 83 - - 61 79 64 - - 23 95 86 - - 62 100 99 - - 98
Georgian
99 99 - 99 100 100 100 - 100 100 100 100 - 100 100 99 99 - 99 100
German
89 79 79 69 63 73 56 55 44 26 94 83 82 66 66 99 99 99 98 98
Greek
100 100 - 99 100 100 100 - 100 100 100 100 - 100 100 100 100 - 99 100
Gujarati
99 99 - 99 99 99 99 - 100 99 100 100 - 100 100 100 100 - 99 100
Hebrew
99 99 - 95 - 100 100 - 92 - 100 100 - 95 - 99 99 - 99 -
Hindi
73 32 100 68 76 60 11 100 54 55 64 19 100 54 75 94 67 100 96 99
Hungarian
94 90 - 72 75 86 77 - 50 40 97 93 - 70 85 100 99 - 98 99
Icelandic
93 87 - - 65 82 71 - - 25 96 92 - - 73 99 99 - - 99
Indonesian
60 47 - 69 62 39 25 - 42 36 60 45 - 69 62 82 71 - 95 88
Irish
90 85 - - 66 81 70 - - 29 94 90 - - 77 95 94 - - 92
Italian
86 71 85 56 44 69 41 68 25 6 91 73 87 48 32 99 98 99 96 93
Japanese
100 100 99 99 33 100 100 98 98 - 100 100 100 100 - 100 100 99 100 100
Kazakh
91 90 - - 76 79 78 - - 43 96 92 - - 87 99 98 - - 99
Korean
99 99 99 99 99 100 100 100 100 100 100 100 100 100 100 99 99 99 99 99
Latin
87 73 - 55 46 72 49 - 36 8 92 75 - 51 42 97 94 - 79 88
Latvian
93 87 - 85 72 84 74 - 70 33 96 90 - 86 84 98 96 - 97 98
Lithuanian
94 87 - 79 70 86 75 - 59 29 97 88 - 81 81 99 98 - 97 99
Macedonian
83 72 - 66 60 65 51 - 43 27 86 70 - 60 69 98 95 - 96 83
Malay
30 30 - - 17 26 21 - - 9 38 35 - - 21 28 35 - - 23
Maori
92 83 - - 60 84 63 - - 12 92 87 - - 71 99 97 - - 98
Marathi
85 39 - 78 83 73 15 - 59 65 84 29 - 80 86 96 71 - 94 98
Mongolian
96 95 - - 78 92 89 - - 43 98 97 - - 92 99 99 - - 99
Nynorsk
65 51 - - 54 40 25 - - 18 65 49 - - 50 90 81 - - 93
Persian
90 80 - 72 61 77 61 - 49 12 93 80 - 68 72 99 97 - 99 99
Polish
94 89 - 81 74 85 76 - 63 37 98 93 - 81 86 99 99 - 99 99
Portuguese
81 68 70 60 53 59 41 42 30 19 85 69 70 52 47 98 95 99 97 94
Punjabi
99 99 - 99 100 100 100 - 100 100 100 100 - 100 100 99 99 - 99 100
Romanian
86 72 - 66 53 68 49 - 45 11 91 73 - 62 53 99 94 - 90 96
Russian
89 78 100 67 60 76 59 100 51 26 94 83 100 66 67 97 92 100 84 86
Serbian
87 77 - 66 68 73 61 - 46 29 90 79 - 63 77 99 91 - 89 99
Shona
91 80 - 76 64 77 56 - 55 23 95 86 - 76 71 100 99 - 98 99
Slovak
84 75 - 64 71 64 49 - 40 37 90 78 - 60 75 98 97 - 93 99
Slovene
82 66 - 52 47 61 39 - 29 8 86 67 - 43 42 98 93 - 83 92
Somali
92 84 - - 69 81 64 - - 26 95 90 - - 83 99 99 - - 99
Sotho
85 71 - - 53 66 42 - - 12 90 74 - - 54 99 97 - - 95
Spanish
69 56 62 47 43 43 25 35 18 12 68 48 54 32 33 97 93 98 91 84
Swahili
80 69 - - 57 60 43 - - 15 84 68 - - 58 98 96 - - 97
Swedish
83 72 71 61 53 64 45 47 34 14 88 75 70 57 52 98 94 97 92 93
Tagalog
77 66 - - 50 52 35 - - 9 83 66 - - 44 98 96 - - 96
Tamil
100 100 - 100 100 100 100 - 100 100 100 100 - 100 100 100 100 - 100 100
Telugu
100 100 - 99 100 100 100 - 100 100 100 100 - 100 100 100 100 - 99 100
Thai
99 99 - 99 99 100 100 - 100 100 100 100 - 100 100 99 99 - 99 99
Tsonga
84 72 - - 61 66 46 - - 19 89 73 - - 67 98 96 - - 97
Tswana
84 71 - - 56 65 44 - - 17 88 73 - - 57 98 96 - - 94
Turkish
93 87 92 63 66 83 70 83 39 30 97 91 94 59 71 99 99 99 91 97
Ukrainian
92 86 - 85 77 84 74 - 69 45 97 91 - 88 87 95 92 - 98 98
Urdu
90 79 - 68 61 80 64 - 45 8 94 78 - 63 75 97 96 - 95 99
Vietnamese
90 87 85 91 63 78 75 64 75 - 94 87 91 97 89 99 98 99 99 99
Welsh
91 82 - - 72 78 60 - - 34 95 87 - - 85 99 98 - - 98
Xhosa
82 68 - - 71 63 44 - - 45 85 67 - - 70 98 93 - - 97
Yoruba
74 62 - - 36 50 33 - - 1 76 60 - - 22 96 92 - - 87
Zulu
80 70 - 76 54 62 45 - 53 18 83 71 - 77 51 97 94 - 99 93
Mean
86 77 84 74 64 74 60 69 57 34 89 78 84 71 68 96 93 99 94 94
Median
89.0
79.0
85.0
69.0
63.0
74.0
56.0
66.0
49.5
26.0
93.0
80.0
89.0
67.5
71.0
99.0
96.0
99.0
97.0
98.0
Standard Deviation
13.15
17.33
14.21
17.27
18.54
18.47
25.01
26.6
27.23
28.81
13.24
19.09
15.66
21.07
22.74
11.06
11.85
0.85
6.04
12.16
Open table
Language
Average
Single Words
Word Pairs
Sentences
Lingua
(high accuracy mode)Lingua
(low accuracy mode)Whichlang
Whatlang
CLD2
Lingua
(high accuracy mode)Lingua
(low accuracy mode)Whichlang
Whatlang
CLD2
Lingua
(high accuracy mode)Lingua
(low accuracy mode)Whichlang
Whatlang
CLD2
Lingua
(high accuracy mode)Lingua
(low accuracy mode)Whichlang
Whatlang
CLD2
Arabic
100 100 100 100 66 100 100 100 100 19 100 100 100 100 81 100 100 100 100 99
Chinese
100 100 97 100 33 100 100 93 100 - 100 100 98 100 2 100 100 100 100 98
Dutch
87 80 73 63 47 73 60 53 39 10 89 81 70 54 41 99 98 98 96 90
English
87 76 66 57 55 70 52 34 29 11 93 77 65 47 55 99 98 98 95 99
French
92 84 68 74 51 82 65 36 50 12 96 88 70 73 47 99 98 99 98 93
German
92 86 79 76 63 82 70 55 54 26 95 89 82 75 66 99 99 99 98 98
Hindi
100 99 100 99 76 100 100 100 100 55 100 100 100 100 75 100 99 100 99 99
Italian
92 82 85 68 44 82 63 68 45 6 95 84 87 63 32 99 99 99 97 93
Japanese
100 100 99 99 33 100 100 98 98 - 100 100 100 100 - 100 100 99 100 100
Korean
99 99 99 99 99 100 100 100 100 100 100 100 100 100 100 99 99 99 99 99
Portuguese
85 76 70 67 53 69 55 42 42 19 88 77 70 62 47 99 97 99 98 94
Russian
100 100 100 100 60 100 100 100 100 26 100 100 100 100 67 100 100 100 100 86
Spanish
78 69 62 54 43 59 46 35 29 12 78 65 54 41 33 98 95 98 93 84
Swedish
93 86 71 74 53 83 69 47 51 14 96 90 70 74 52 99 98 97 97 93
Turkish
97 95 92 81 66 94 89 83 66 30 99 97 94 78 71 100 99 99 98 97
Vietnamese
94 91 85 92 63 86 83 64 79 - 96 92 91 98 89 99 99 99 99 99
Mean
94 89 84 81 57 86 78 69 68 26 95 90 84 79 57 99 99 99 98 95
Median
93.5
88.5
85.0
78.5
54.0
84.5
76.5
66.0
60.0
19.0
96.0
91.0
89.0
76.5
55.0
99.0
99.0
99.0
98.0
97.5
Standard Deviation
6.62
10.44
14.21
16.91
16.44
13.51
20.23
26.6
28.33
25.57
6.03
10.76
15.66
21.17
25.16
0.6
1.31
0.85
1.98
4.95
Various benchmarks for all classifiers can be run via:
cargo bench --features benchmark
The benchmarks measure the processing time for classifying 2,000 sentences available in the common 16 languages that are supported by each language detector. The results below have been produced on an iMac 3.6 Ghz 8-Core Intel Core i9 with 40 GB RAM. Whichlang has the shortest processing time, Lingua the longest.
| | Single Thread | Multiple Threads | |--------------------------------------------------|-------------------|----------------------| | Lingua / high accuracy mode / all languages | 622.00 ms | 96.648 ms | | Lingua / high accuracy mode / common languages | 333.31 ms | 37.347 ms | | Lingua / low accuracy mode / all languages | 373.15 ms | 48.182 ms | | Lingua / low accuracy mode / common languages | 180.54 ms | 24.550 ms | | Whichlang | 2.0458 ms | 351.03 µs | | Whatlang / all languages | 113.08 ms | 12.992 ms | | Whatlang / common languages | 47.742 ms | 5.6070 ms | | CLD 2 | 8.9223 ms | 2.0528 ms |
Every language detector uses a probabilistic n-gram model trained on the character distribution in some training corpus. Most libraries only use n-grams of size 3 (trigrams) which is satisfactory for detecting the language of longer text fragments consisting of multiple sentences. For short phrases or single words, however, trigrams are not enough. The shorter the input text is, the less n-grams are available. The probabilities estimated from such few n-grams are not reliable. This is why Lingua makes use of n-grams of sizes 1 up to 5 which results in much more accurate prediction of the correct language.
A second important difference is that Lingua does not only use such a statistical model, but also a rule-based engine. This engine first determines the alphabet of the input text and searches for characters which are unique in one or more languages. If exactly one language can be reliably chosen this way, the statistical model is not necessary anymore. In any case, the rule-based engine filters out languages that do not satisfy the conditions of the input text. Only then, in a second step, the probabilistic n-gram model is taken into consideration. This makes sense because loading less language models means less memory consumption and better runtime performance.
In general, it is always a good idea to restrict the set of languages to be considered in the classification process using the respective api methods. If you know beforehand that certain languages are never to occur in an input text, do not let those take part in the classifcation process. The filtering mechanism of the rule-based engine is quite good, however, filtering based on your own knowledge of the input text is always preferable.
If you want to reproduce the accuracy results above, you can generate the test reports yourself for all classifiers and all languages by doing:
cargo run --release --bin accuracy_reports --features accuracy-reports
It is important to use the --release flag here because loading the language models in debug mode takes too much time.
For each detector and language, a test report file is then written into
/accuracy-reports,
to be found next to the src directory. As an example, here is the current output of the Lingua German report:
```
Accuracy on average: 89.23%
Detection of 1000 single words (average length: 9 chars) Accuracy: 73.9% Erroneously classified as Dutch: 2.3%, Danish: 2.1%, English: 2%, Latin: 1.9%, Bokmal: 1.6%, Basque: 1.2%, Esperanto: 1.2%, French: 1.2%, Italian: 1.2%, Swedish: 1%, Afrikaans: 0.8%, Tsonga: 0.7%, Nynorsk: 0.6%, Spanish: 0.6%, Yoruba: 0.6%, Finnish: 0.5%, Sotho: 0.5%, Welsh: 0.5%, Estonian: 0.4%, Irish: 0.4%, Polish: 0.4%, Swahili: 0.4%, Tagalog: 0.4%, Tswana: 0.4%, Bosnian: 0.3%, Icelandic: 0.3%, Romanian: 0.3%, Albanian: 0.2%, Catalan: 0.2%, Croatian: 0.2%, Indonesian: 0.2%, Lithuanian: 0.2%, Maori: 0.2%, Turkish: 0.2%, Xhosa: 0.2%, Zulu: 0.2%, Latvian: 0.1%, Malay: 0.1%, Slovak: 0.1%, Slovene: 0.1%, Somali: 0.1%
Detection of 1000 word pairs (average length: 18 chars) Accuracy: 94.1% Erroneously classified as Dutch: 0.9%, Latin: 0.8%, English: 0.7%, Swedish: 0.6%, Danish: 0.5%, French: 0.4%, Bokmal: 0.3%, Irish: 0.2%, Tagalog: 0.2%, Afrikaans: 0.1%, Esperanto: 0.1%, Estonian: 0.1%, Finnish: 0.1%, Italian: 0.1%, Maori: 0.1%, Nynorsk: 0.1%, Somali: 0.1%, Swahili: 0.1%, Tsonga: 0.1%, Turkish: 0.1%, Welsh: 0.1%, Zulu: 0.1%
Detection of 1000 sentences (average length: 111 chars) Accuracy: 99.7% Erroneously classified as Dutch: 0.2%, Latin: 0.1% ```
Add Lingua to your Cargo.toml file like so:
toml
[dependencies]
lingua = "1.5.0"
By default, this will download the language model dependencies for all 75 supported languages,
a total of approximately 90 MB. If your bandwidth or hard drive space is limited, or you simply
do not need all languages, you can specify a subset of the language models to be downloaded as
separate features in your Cargo.toml:
toml
[dependencies]
lingua = { version = "1.5.0", default-features = false, features = ["french", "italian", "spanish"] }
In order to build the source code yourself, you need the stable Rust toolchain installed on your machine so that cargo, the Rust package manager is available.
git clone https://github.com/pemistahl/lingua-rs.git
cd lingua-rs
cargo build
The source code is accompanied by an extensive unit test suite. To run them, simply say:
cargo test
```rust use lingua::{Language, LanguageDetector, LanguageDetectorBuilder}; use lingua::Language::{English, French, German, Spanish};
let languages = vec![English, French, German, Spanish];
let detector: LanguageDetector = LanguageDetectorBuilder::fromlanguages(&languages).build();
let detectedlanguage: Option
asserteq!(detectedlanguage, Some(English)); ```
By default, Lingua returns the most likely language for a given input text. However, there are certain words that are spelled the same in more than one language. The word prologue, for instance, is both a valid English and French word. Lingua would output either English or French which might be wrong in the given context. For cases like that, it is possible to specify a minimum relative distance that the logarithmized and summed up probabilities for each possible language have to satisfy. It can be stated in the following way:
```rust use lingua::LanguageDetectorBuilder; use lingua::Language::{English, French, German, Spanish};
let detector = LanguageDetectorBuilder::fromlanguages(&[English, French, German, Spanish]) .withminimumrelativedistance(0.9) .build(); let detectedlanguage = detector.detectlanguage_of("languages are awesome");
asserteq!(detectedlanguage, None); ```
Be aware that the distance between the language probabilities is dependent on the length of the
input text. The longer the input text, the larger the distance between the languages. So if you
want to classify very short text phrases, do not set the minimum relative distance too high.
Otherwise None will be
returned most of the time as in the example above. This is the return value for cases where
language detection is not reliably possible.
Knowing about the most likely language is nice but how reliable is the computed likelihood? And how less likely are the other examined languages in comparison to the most likely one? These questions can be answered as well:
```rust use lingua::Language::{English, French, German, Spanish}; use lingua::LanguageDetectorBuilder;
fn main() {
let languages = vec![English, French, German, Spanish];
let detector = LanguageDetectorBuilder::fromlanguages(&languages).build();
let confidencevalues = detector
.computelanguageconfidencevalues("languages are awesome")
.intoiter()
.map(|(language, confidence)| (language, (confidence * 100.0).round() / 100.0))
.collect::
assert_eq!(
confidence_values,
vec![
(English, 0.93),
(French, 0.04),
(German, 0.02),
(Spanish, 0.01)
]
);
} ```
In the example above, a vector of two-element tuples is returned containing all possible languages sorted by their confidence value in descending order. Each value is a probability between 0.0 and 1.0. The probabilities of all languages will sum to 1.0. If the language is unambiguously identified by the rule engine, the value 1.0 will always be returned for this language. The other languages will receive a value of 0.0.
There is also a method for returning the confidence value for one specific language only:
```rust use lingua::Language::{English, French, German, Spanish}; use lingua::LanguageDetectorBuilder;
fn main() { let languages = vec![English, French, German, Spanish]; let detector = LanguageDetectorBuilder::fromlanguages(&languages).build(); let confidence = detector.computelanguageconfidence("languages are awesome", French); let roundedconfidence = (confidence * 100.0).round() / 100.0;
assert_eq!(rounded_confidence, 0.04);
} ```
The value that this method computes is a number between 0.0 and 1.0. If the language is unambiguously identified by the rule engine, the value 1.0 will always be returned. If the given language is not supported by this detector instance, the value 0.0 will always be returned.
By default, Lingua uses lazy-loading to load only those language models on demand which are considered relevant by the rule-based filter engine. For web services, for instance, it is rather beneficial to preload all language models into memory to avoid unexpected latency while waiting for the service response. If you want to enable the eager-loading mode, you can do it like this:
rust
LanguageDetectorBuilder::from_all_languages().with_preloaded_language_models().build();
Multiple instances of LanguageDetector share the same language models in memory which are
accessed asynchronously by the instances.
Lingua's high detection accuracy comes at the cost of being noticeably slower than other language detectors. The large language models also consume significant amounts of memory. These requirements might not be feasible for systems running low on resources. If you want to classify mostly long texts or need to save resources, you can enable a low accuracy mode that loads only a small subset of the language models into memory:
rust
LanguageDetectorBuilder::from_all_languages().with_low_accuracy_mode().build();
The downside of this approach is that detection accuracy for short texts consisting of less than 120 characters will drop significantly. However, detection accuracy for texts which are longer than 120 characters will remain mostly unaffected.
In high accuracy mode (the default), the language detector consumes approximately 970 MB of memory if all language models are loaded. In low accuracy mode, memory consumption is reduced to approximately 72 MB. The goal is to further reduce memory consumption in later releases.
An alternative for a smaller memory footprint and faster performance is to reduce the set of languages when building the language detector. In most cases, it is not advisable to build the detector from all supported languages. When you have knowledge about the texts you want to classify you can almost always rule out certain languages as impossible or unlikely to occur.
In contrast to most other language detectors, Lingua is able to detect multiple languages in mixed-language texts. This feature can yield quite reasonable results, but it is still in an experimental state and therefore the detection result is highly dependent on the input text. It works best in high-accuracy mode with multiple long words for each language. The shorter the phrases and their words are, the less accurate are the results. Reducing the set of languages when building the language detector can also improve accuracy for this task if the languages occurring in the text are equal to the languages supported by the respective language detector instance.
```rust use lingua::DetectionResult; use lingua::Language::{English, French, German}; use lingua::LanguageDetectorBuilder;
fn main() { let languages = vec![English, French, German]; let detector = LanguageDetectorBuilder::from_languages(&languages).build(); let sentence = "Parlez-vous français? \ Ich spreche Französisch nur ein bisschen. \ A little bit is better than nothing.";
let results: Vec<DetectionResult> = detector.detect_multiple_languages_of(sentence);
if let [first, second, third] = &results[..] {
assert_eq!(first.language(), French);
assert_eq!(
&sentence[first.start_index()..first.end_index()],
"Parlez-vous français? "
);
assert_eq!(second.language(), German);
assert_eq!(
&sentence[second.start_index()..second.end_index()],
"Ich spreche Französisch nur ein bisschen. "
);
assert_eq!(third.language(), English);
assert_eq!(
&sentence[third.start_index()..third.end_index()],
"A little bit is better than nothing."
);
}
} ```
In the example above, a vector of DetectionResult
is returned. Each entry in the vector describes a contiguous single-language text section,
providing start and end indices of the respective substring.
There might be classification tasks where you know beforehand that your language data is definitely not written in Latin, for instance (what a surprise :-). The detection accuracy can become better in such cases if you exclude certain languages from the decision process or just explicitly include relevant languages:
```rust use lingua::{LanguageDetectorBuilder, Language, IsoCode6391, IsoCode6393};
// Include all languages available in the library. LanguageDetectorBuilder::fromalllanguages();
// Include only languages that are not yet extinct (= currently excludes Latin). LanguageDetectorBuilder::fromallspoken_languages();
// Include only languages written with Cyrillic script. LanguageDetectorBuilder::fromalllanguageswithcyrillic_script();
// Exclude only the Spanish language from the decision algorithm. LanguageDetectorBuilder::fromalllanguages_without(&[Language::Spanish]);
// Only decide between English and German. LanguageDetectorBuilder::from_languages(&[Language::English, Language::German]);
// Select languages by ISO 639-1 code. LanguageDetectorBuilder::fromisocodes6391(&[IsoCode6391::EN, IsoCode6391::DE]);
// Select languages by ISO 639-3 code. LanguageDetectorBuilder::fromisocodes6393(&[IsoCode6393::ENG, IsoCode6393::DEU]); ```
This library can be compiled to WebAssembly (WASM) which allows to use Lingua in any JavaScript-based project, be it in the browser or in the back end running on Node.js.
The easiest way to compile is to use wasm-pack. After the installation,
you can, for instance, build the library with the web target so that it can be directly used in the browser:
wasm-pack build --target web
By default, all 75 supported languages are included in the compiled wasm file which has a size of 74 MB, approximately.
If you only need a subset of certain languages, you can tell wasm-pack which ones to include:
wasm-pack build --target web -- --no-default-features --features "french,italian,spanish"
This creates a directory named pkg on the top-level of this repository, containing the compiled wasm files
and JavaScript and TypeScript bindings. In an HTML file, you can then call Lingua like the following, for instance:
```html
```
There are also some integration tests available for Node.js and all major browsers. To run them, simply say:
wasm-pack test --node --headless --chrome --firefox --safari
The output of wasm-pack will be hosted in a separate repository which
allows to add further JavaScript-related configuration, tests and documentation. Lingua will then be added to the
npm registry as well, allowing for an easy download and installation within every JavaScript
or TypeScript project.
Take a look at the planned issues.
Josh Rotenberg has written a wrapper for using Lingua with the Elixir programming language.
Simon Liang has written a wrapper for using Lingua with NodeJS.
Any contributions to Lingua are very much appreciated. Please read the instructions
in CONTRIBUTING.md
for how to add new languages to the library.