This crate is designed to create a general tooklit for natural language processing, a current deficiency in the Rust ecosystem.
Project can be found on crates.io.
Check out the examples folder to see how to create a sentiment lexicon and get the arousal level for a term.
The sentiment analysis was originally designed by Dr. Christopher Healey and then ported to Rust for the purpose of this project.
Basic tokenization is supported right now (string to sentences, string to tokens, term frequencies), but there are plans to expand this to include stop word removal as well.
Stemming currently uses modified code from rust-stem, but this may switch to the rust-stemmers crate after further research.
More information on the stemming algorithm can be found here.
Term frequency–inverse document frequency (TF-IDF) is an algorithm used to find document similarity. Creating a TF-IDF matrix takes place over two steps: 1. Apply a weight, $w{i,j}$, for every term, $ti$, in the document, $Dj$. $w{i,j}$ is defined as $tf{i,j} \times idfi$, where $tf{i,j}$ is the number of occurrences of $ti$ in $Dj$, and $idfi$ is the log of inverse fraction of documents $ni$ that contain at least one occurrence of $ti, idfi = ln(\frac{n}{ni})$. 1. Take the weighted matrix and then normalize each document vector in order to remove the influence of document length.
The weighted, normalized matrix can then be used to find the cosine similarity between documents. Normally, calculating the cosine similarity of two document vectors would look like $\cos \theta = \frac{Di \cdot Dj}{|Di| |Dj|}$. Since the matrix is already normalized, this simplifies to $\cos \theta = Di \cdot Dj$.
The resulting $MxM$ matrix, where $M$ is the number of columns from the TF-IDF matrix, has 1's along the diagonal since the similarity of a document with itself is 1. The intersections of rows and columns, $M{i,j}$, is the cosine similarity value between $Di$ and $D_j$.