TinKV is a simple and fast key-value storage engine written in Rust. Inspired by basho/bitcask, written after attending the Talent Plan courses.

Notes: - Do not use it in production. - Operations like set/remove/compact are not thread-safe currently.

Happy hacking~

Features

• Embeddable (use tinkv as a library);
• Builtin CLI (tinkv);
• Builtin Redis compatible server (WIP);
• Predictable read/write performance.

Usage

As a library

shell $ cargo add tinkv

Full example usage can be found in examples/basic.rs.

```rust use prettyenvlogger; use tinkv::{self, Store};

fn main() -> tinkv::Result<()> { prettyenvlogger::init(); let mut store = Store::open("/path/to/tinkv")?; store.set("hello".asbytes(), "tinkv".asbytes())?;

let value = store.get("hello".as_bytes())?;
assert_eq!(value, Some("tinkv".as_bytes().to_vec()));

store.remove("hello".as_bytes())?;

let value_not_found = store.get("hello".as_bytes())?;
assert_eq!(value_not_found, None);

Ok(())

} ```

Open with custom options

```rust use prettyenvlogger; use tinkv::{self, Store};

fn main() -> tinkv::Result<()> { let mut store = tinkv::OpenOptions::new() .maxdatafilesize(1024 * 1024) .maxkeysize(128) .maxvaluesize(128) .sync(true) .open(".tinkv")?; store.set("hello".asbytes(), "world".as_bytes())?; Ok(()) } ```

APIs

Public APIs of tinkv store are very easy to use: | API | Description | |--------------------------|---------------------------------------------------------------| |Store::open(path) | Open a new or existing datastore. The directory must be writeable and readable for tinkv store.| |tinkv::OpenOptions()| Open a new or existing datastore with custom options. | |store.get(key)| Get value by key from datastore.| |store.set(key, value)| Store a key value pair into datastore.| |store.remove(key, value)| Remove a key from datastore.| |store.compact()| Merge data files into a more compact form. drop stale segments to release disk space. Produce hint files after compaction for faster startup.| |store.keys()| Return all the keys in database.| |store.len()| Return total number of keys in database.| |store.for_each(f: impl Fn(key, value))| Iterate all keys in database and call functionffor each entry.| |store.stas()| Get current statistics of database.| |store.sync()| Force any writes to datastore.| |store.close()` | Close datastore, sync all pending writes to disk.|

Run examples

shell $ RUST_LOG=trace cargo run --example basic

RUST_LOG level can be one of [trace, debug, info, error].

CLI

Install tinkv executable binaries.

shell $ cargo install tinkv

```shell $ tinkv --help ... USAGE: tinkv [FLAGS]

FLAGS: -h, --help Prints help information -q, --quiet Pass many times for less log output -V, --version Prints version information -v, --verbose Pass many times for more log output

ARGS: Path to tinkv datastore

SUBCOMMANDS: compact Compact data files in datastore and reclaim disk space del Delete a key value pair from datastore get Retrive value of a key, and display the value help Prints this message or the help of the given subcommand(s) keys List all keys in datastore scan Perform a prefix scanning for keys set Store a key value pair into datastore stats Display statistics of the datastore ```

Example usages: ```shell $ tinkv /tmp/db set hello world $ tinkv /tmp/db get hello world

Change verbosity level (info).

$ tinkv /tmp/db -vvv compact 2020-06-20T10:32:45.582Z INFO tinkv::store > open store path: tmp/db 2020-06-20T10:32:45.582Z INFO tinkv::store > build key dirfrom data file /tmp/db/000000000001.tinkv.data 2020-06-20T10:32:45.583Z INFO tinkv::store > build key dirfrom data file /tmp/db/000000000002.tinkv.data 2020-06-20T10:32:45.583Z INFO tinkv::store > build keydirdone, got 1 keys. current stats: Stats { sizeofstaleentries:0, totalstaleentries: 0, totalactiveentries: 1,totaldatafiles: 2, sizeofalldata_files: 60 } 2020-06-20T10:32:45.583Z INFO tinkv::store > there are 3 datafiles need to be compacted ```

Client & Server

WIP...

Redis-compatible protocol?

Note: not all the redis commands are available, only a few of them are supported by tinkv.

• get <key>
• set <key> <value>
• del <key>

About Compaction

Compation process will be triggered if size_of_stale_entries >= config::COMPACTION_THRESHOLD after each call of set/remove. Compaction steps are very simple and easy to understand: 1. Freeze current active segment, and switch to another one. 2. Create a compaction segment file, then iterate all the entries in keydir (in-memory hash table), copy related data entries into compaction file and update keydir. 3. Remove all the stale segment files.

Hint files (for fast startup) of corresponding data files will be generated after each compaction.

You can call store.compact() method to trigger compaction process if nessesary.

```rust use prettyenvlogger; use tinkv::{self, Store};

fn main() -> tinkv::Result<()> { prettyenvlogger::init(); let mut store = Store::open("/path/to/tinkv")?; store.compact()?;

Ok(())

} ```

Structure of Data Directory

shell .tinkv ├── 000000000001.tinkv.hint -- related index/hint file, for fast startup ├── 000000000001.tinkv.data -- immutable data file └── 000000000002.tinkv.data -- active data file

Refs

I'm not familiar with erlang, but I found some implementations in other languages worth learning.

1. Go: prologic/bitcask
2. Go: prologic/bitraft
3. Python: turicas/pybitcask
4. Rust: dragonquest/bitcask

Found another simple key-value database based on Bitcask model, please refer xujiajun/nutsdb.