tiktoken-rust

STATUS: Under development.

tiktoken is a fast BPE tokeniser for use with OpenAI's models. It provides Python interface to interact with it.

This project is a fork of original repo, bring the capability to rust world.

```rust use tiktoken_rust as tt;

let enc = tt::getencoding("cl100kbase").unwrap();

asserteq!( "hello world", enc.decode(&enc.encodeordinary("hello world"), tt::DecodeMode::Strict).unwrap() ) ```