icu_segmenter

🚧 [Experimental] Segment strings by lines, graphemes, word, and sentences.

This module is published as its own crate (icu_segmenter) and as part of the icu crate. See the latter for more details on the ICU4X project.

This module contains segmenter implementation for the following rules.

Line breaker that is compatible with Unicode Standard Annex #14 and CSS properties.
Grapheme cluster breaker, word breaker, and sentence breaker that are compatible with Unicode Standard Annex #29.

🚧 This code is experimental; it may change at any time, in breaking or non-breaking ways, including in SemVer minor releases. It can be enabled with the "experimental" Cargo feature of the icu meta-crate. Use with caution. #2259

Examples

Line Break

Segment a string with default options:

```rust use icu::segmenter::LineSegmenter;

let segmenter = LineSegmenter::trynewunstable(&icu_testdata::unstable()) .expect("Data exists");

let breakpoints: Vec = segmenter.segmentstr("Hello World").collect(); asserteq!(&breakpoints, &[6, 11]); ```

See [LineSegmenter] for more examples.

Grapheme Cluster Break

See [GraphemeClusterSegmenter] for examples.

Word Break

Segment a string:

```rust use icu::segmenter::WordSegmenter;

let segmenter = WordSegmenter::trynewunstable(&icu_testdata::unstable()) .expect("Data exists");

let breakpoints: Vec = segmenter.segmentstr("Hello World").collect(); asserteq!(&breakpoints, &[0, 5, 6, 11]); ```

See [WordSegmenter] for more examples.

Sentence Break

See [SentenceSegmenter] for examples.

More Information

For more information on development, authorship, contributing etc. please visit ICU4X home page.