tiktoken-rs

Ready-made tokenizer library for working with GPT and tiktoken

Usage

1. Install this tool locally with cargo

sh cargo add tiktoken-rs

See the examples in the repo for other use-cases.

Encountered any bugs?

If you encounter any bugs or have any suggestions for improvements, please open an issue on the repository.

License

This project is licensed under the MIT License.