# CREATURE FEATUR(ization) A crate for polymorphic ML & NLP featurization that leverages zero-cost abstraction. It provides composable n-gram combinators that are ergonomic and bare-metal fast. Although created with NLP in mind, it's very general and can be applied in a plethera of domains such as computer vision.

There are many n-gram crates, but the majority force heap allocation or lock you into a concrete type that doesn’t fit your use-case or performance needs. In most benchmarks, creature_feature is anywhere between 4x - 60x faster.

# See a live demo

Here is a live demo of creature_feature using WASM

# A Swiss Army Knife of Featurization ```rust use creaturefeature::traits::Ftzr; use creaturefeature::ftzrs::{bigram, bislice, foreach, whole}; use creaturefeature::HashedAs; use creature_feature::convert::Bag; use std::collections::{HashMap, HashSet, BTreeMap, LinkedList, BTreeSet, BinaryHeap, VecDeque};

let intdata = &[1, 2, 3, 4, 5]; let strdata = "one fish two fish red fish blue fish";

// notice how the left-hand side remains almost unchanged.

// we're using 'bislice' right now (which is a 2-gram of referenced data), 'ftzrs::bigram' would yield owned data instead of references

let reffeats: Vec<&str> = bislice().featurize(strdata); let reffeats: LinkedList<&[u8]> = bislice().featurize(strdata); let refbag: Bag> = bislice().featurize(intdata); let reftrigrambag: Bag> = foreach(whole()).featurize(strdata.splitasciiwhitespace()); let hashedtrigrams: BTreeSet> = trislice().featurize(intdata); The above five will have the following values, respectively. ["on", "ne", "e ", " f", "fi", "is", "sh", "h ", " t", "tw", "wo", "o ", " f", "fi", "is", "sh", "h ", " r", "re", "ed", "d ", " f", "fi", "is", "sh", "h ", " b", "bl", "lu", "ue", "e ", " f", "fi", "is", "sh"]

[[111, 110], [110, 101], [101, 32], [32, 102], [102, 105], [105, 115], [115, 104], [104, 32], [32, 116], [116, 119], [119, 111], [111, 32], [32, 102], [102, 105], [105, 115], [115, 104], [104, 32], [32, 114], [114, 101], [101, 100], [100, 32], [32, 102], [102, 105], [105, 115], [115, 104], [104, 32], [32, 98], [98, 108], [108, 117], [117, 101], [101, 32], [32, 102], [102, 105], [105, 115], [115, 104]]

Bag({[2, 3, 4]: 1, [3, 4, 5]: 1, [1, 2, 3]: 1})

Bag({"blue": 1, "fish": 4, "one": 1, "red": 1, "two": 1})

{HashedAs(3939941806544028562), HashedAs(7191405660579021101), HashedAs(16403185381100005216)} ```

Here are more examples of what's possible:

```rust // let's now switch to 'bigram' let ownedfeats: BTreeSet<[u8; 2]> = bigram().featurize(strdata); let ownedfeats: Vec = bigram().featurize(strdata); let ownedfeats: HashSet> = bigram().featurize(intdata); let ownedbag: Bag, u16>> = bigram().featurize(intdata);

let hashedfeats: BinaryHeap> = bislice().featurize(strdata); let hashedfeats: VecDeque> = bigram().featurize(intdata);

let sentence = strdata.splitasciiwhitespace(); let bagofwords: Bag> = foreach(bigram()) .featurize(sentence.clone()); let bagofwords: Bag> = for_each(bislice()).featurize(sentence.clone());

// and many, MANY more posibilities ```

### We can even produce multiple outputs while still only featurizing the input once rust let (set, list): (BTreeSet<HashedAs<u64>>, Vec<&str>) = bislice().featurize_x2(str_data);

# creature_feature provides three general flavors of featurizers:

1) NGram<const N: usize> provides n-grams over copied data and produces owned data or multiple [T;N]. Examples include ftzrs::n_gram, ftzrs::bigram and ftzrs::trigram.

2) SliceGram provides n-grams over referenced data and produces owned data or multiple &[T]. Examples include ftzrs::n_slice, ftzrs::bislice and ftzrs::trislice.

3) Combinators that compose one or more featurizers and return a new featurizer with different behavior. Examples include ftzrs::for_each, ftzrs::gap_gram, featurizers! and ftzrs::bookends.

# WHY POLYMORPHISM == PERFORMANCE Here is a small quiz to show why polymorphic featurization and FAST featurization go hand-in-hand.

Here are four different ways to featurize a string that are basically equivalent. But, which one of the four is fastest? By how much?

```rust let sentence = "It is a truth universally acknowledged that Jane Austin must be used in nlp examples";

let one: Vec = trigram().featurize(sentence); let two: Vec<[u8;3]> = trigram().featurize(sentence); let three: Vec<&str> = trislice().featurize(sentence); // same performance as &[u8] let four: Vec> = trislice().featurize(sentence); // could have used trigram ```

Trigrams of String, HashedAs<u64>, &str and [u8; 3] each have their place depending on your use-case. But there can be roughly two orders of magnitude of difference in performance between the fastest and the slowest. If you choose the wrong one for your needs (or use a less polymorphic crate), you're losing out on speed!

# What type should I use to represent my features? * use Collection<[T; N]> via ftzrs::n_gram if both T and N are small. This is most of the time.

• use Collection<&[T]> (or Collection<&str>) via ftzrs::n_slice if [T; N] would be larger (in bytes) than (usize, usize). This is more common if N is large or you're using char instead of u8. This is also depends on the lifetime of the original data vs the lifetime of the features produced.

• HashedAs<u64> has the opposite time complexity as &[T], linear creation and O(1) equality. If you're ok with one-in-a-millionish hash collisions, this can be a great compromise.

• Never use Collection<String> or Collection<Vec<T>> in a performance critical section.

  Where does creature_feature fit in with other tokenizers?

  creature_feature is very flexible, and traits::Ftzr/traits::IterFtzr can be easily implemented with a newtype for whatever other tokenizer/featurizer you please. Anything could be featurized: images, documents, time-series data, etc.

  Example: Featurizing books

  Consider a custom struct to represent a book ```rust struct Book { author: String, genre: Genre, sub_genre: SubGenre, year: u16, }

[derive(Hash)]

enum Genre { Fiction, NonFiction, Religion, }

[derive(Hash)]

enum SubGenre { Romance, History, DataScience, }

impl Book { fn decade(&self) -> u8 { unimplemented!() } } We can easily make a custom featurizer for `Book` by visitation with `traits::Ftzr`. rust use creaturefeature::ftzrs::whole; use creaturefeature::traits::*; use creature_feature::HashedAs;

struct BookFtzr;

impl<'a> Ftzr<&'a Book> for BookFtzr { type TokenGroup = HashedAs; fn pushtokenstokensfrom(&book.author, push); push(FeatureFrom::from(&book.genre)); push(FeatureFrom::from(&book.subgenre)); push(FeatureFrom::from(book.year)); push(FeatureFrom::from(book.decade())); } } `` Now we could easily implement a similarity metric forBookviaVec>`, like cosine or jaccard.