libdeflater

Rust bindings to libdeflate, a high-performance library for working with gzip/zlib/deflate data.

libdeflater = "0.7.3"

libdeflater is a thin wrapper library around libdeflate. Libdeflate is optimal in applications that have the input data up-front, or when (large) input datasets can be split into smaller chunks (e.g. genomic bam files, some object stores, specialized backends, game netcode packets).

This is a thin library around libdeflate that:

Builds libdeflate from source (see libdeflate-sys's build.rs file)
Binds to libdeflate's C API with rust bindings (see lib.rs)
Contains high-level integration tests to ensure the bindings work (see integration_test.rs)
Contains usage examples and a benchmark suite. The benchmark suite indicates a 2-3x speedup accross the Calgary and Canterbury corpuses (see Benchmarks section)

⚠️ Warning: libdeflate is best-suited for specialized use-cases where you know the rough size range of your input up-front. You should use something like flate2 if you want a general-purpose deflate library that supports streaming.

Examples

Example source here. To run the examples:

bash cargo run --example gz_compress.rs cargo run --example gz_decompress.rs

Benchmarks

Benchmark data is from both the Calgary Corpus, and the Canterbury Corpus. The benchmark tables below were made with this set of steps:

```bash wget http://www.data-compression.info/files/corpora/largecalgarycorpus.zip unzip -d benchdata largecalgarycorpus.zip wget http://corpus.canterbury.ac.nz/resources/cantrbry.zip unzip -d benchdata cantrbry.zip

runs benchmarks against all files in `bench_data`

cargo bench scripts/process-bench.rb encode scripts/process-bench.rb decode ```

Compression

Avg. speedup (on this corpus) is around 2-3x

bench size [KB] speedup flate2 [us] libdeflate [us] alice29.txt 152 3.2 9883 3132 asyoulik.txt 125 3.0 7792 2572 bib 111 2.8 5380 1907 book1 768 3.2 55512 17588 book2 610 2.9 33751 11719 cp.html 24 2.0 571 280 fields.c 11 2.1 229 108 geo 102 8.4 12142 1441 grammar.lsp 3 1.7 64 38 kennedy.xls 1029 6.4 62130 9695 lcet10.txt 426 3.1 23727 7733 news 377 2.7 15814 5898 obj1 21 2.5 543 220 obj2 246 3.7 12202 3336 paper1 53 2.5 2149 865 paper2 82 2.7 4410 1613 paper3 46 2.5 2179 873 paper4 13 2.0 341 174 paper5 11 1.9 269 144 paper6 38 2.3 1350 584 pic 513 2.6 11364 4426 plrabn12.txt 481 3.4 38339 11266 progc 39 2.4 1401 590 progl 71 2.6 2301 897 progp 49 2.5 1322 531 ptt5 513 2.6 11379 4455 sum 38 3.7 1606 435 trans 93 2.5 2403 958 xargs.1 4 1.7 72 42

Decompression

Avg. speedup (on this corpus) is around 2x.

bench size [KB] speedup flate2 [us] libdeflate [us] alice29.txt 152 2.5 601 244 asyoulik.txt 125 2.2 519 234 bib 111 2.5 413 165 book1 768 2.2 3204 1478 book2 610 2.4 2405 987 cp.html 24 2.0 81 41 fields.c 11 2.1 35 17 geo 102 1.8 553 315 grammar.lsp 3 1.8 16 9 kennedy.xls 1029 2.1 2352 1105 lcet10.txt 426 2.5 1505 607 news 377 2.1 1570 738 obj1 21 1.9 94 50 obj2 246 2.3 971 431 paper1 53 2.3 205 90 paper2 82 2.4 341 143 paper3 46 2.1 192 90 paper4 13 1.7 54 32 paper5 11 1.9 52 28 paper6 38 2.1 140 68 pic 513 3.0 1140 386 plrabn12.txt 481 2.2 1973 880 progc 39 2.2 153 70 progl 71 2.5 214 86 progp 49 2.3 135 58 ptt5 513 3.0 1144 386 sum 38 2.1 147 71 trans 93 2.4 246 104 xargs.1 4 1.7 19 11

Benchmark Notes

All benchmarks are single-threaded
IO/streaming overhead is not considered. The decompressed data is read into memory before performing the comparison
Comparison made against flate2 with no feature flags (i.e. miniz implementation). flate2 was chosen because it's the most popular.
Comparisons with other flate2 backends are available on the bench-flate2-miniz-oxide and bench-flate2-zlib branches. The zlib backend is ~8 % faster on some of the corpus entries.
Compression performed with default compression setting in both cases
Corpus entries were compressed with flate2 at default compression level

Compile-time features

You can enable the following features to customise the build: - use_rust_alloc: Makes libdeflate use Rust's allocator instead of the libc one. This is useful when Rust is preconfigured to use a custom global allocator (e.g. pool-based, or a tracking one, or something else entirely). - freestanding: Builds libdeflate in a freestanding mode (no reliance on libc). This is useful for targets that don't have a C stdlib (e.g. wasm32-unknown-unknown) as otherwise they would fail to compile. Implies use_rust_alloc.