lz4_flex

Fastest LZ4 implementation in Rust. Originally based on redox-os' lz4 compression, but now a complete rewrite. The results in the table are from a benchmark in this project (66Kb JSON) on with the block format.

| Compressor | Compression | Decompression | Ratio | |----------------------|-------------|---------------|---------------| | lz4flex unsafe | 898 MiB/s | 4417 MiB/s | 0.2289 | | lz4flex safe | 724 MiB/s | 2014 MiB/s | 0.2289 | | lzzz (lz4 1.9.3) | 993 MiB/s | 4193 MiB/s | 0.2283 | | lz4_fear | 456 MiB/s | 976 MiB/s | 0.2283 | | snap | 861 MiB/s | 941 MiB/s | 0.2242 |

Features

• Very good logo
• LZ4 Block format
• LZ4 Frame format (thanks @arthurprs)
• High performance
• 1,5s clean release build time
• Feature flags to configure safe/unsafe code usage
• no-std support (thanks @coolreader18)
• 32-bit support

Usage:

Compression and decompression uses no usafe via the default feature flags "safe-encode" and "safe-decode". If you need more performance you can disable them (e.g. with no-default-features).

Safe: lz4_flex = { version = "0.8.0" }

Performance: lz4_flex = { version = "0.8.0", default-features = false }

Warning: If you don't trust your input and your are using the Block format, use checked-decode in order to avoid out of bounds access. When using the Frame format make sure to enable checksums. lz4_flex = { version = "0.8.0", default-features = false, features = ["checked-decode"] }

```rust use lz4flex::{compressprependsize, decompresssize_prepended};

fn main(){ let input: &[u8] = b"Hello people, what's up?"; let compressed = compressprependsize(input); let uncompressed = decompresssizeprepended(&compressed).unwrap(); assert_eq!(input, uncompressed); } ```

Benchmarks

The benchmark is run with criterion, the test files are in the benches folder.

Currently 4 implementations are compared, this one, lz-fear and the c++ version via rust bindings and snappy. The lz4-flex version is tested with the feature flags safe-decode and safe-encode switched on and off.

• lz4_cpp: https://crates.io/crates/lz4
• lz-fear: https://github.com/main--/rust-lz-fear
• snap: https://github.com/burntsushi/rust-snappy

Results v0.8.0 17-05-2021 (safe-decode and safe-encode off)

cargo bench --no-default-features

Executed on Core i7-6700 Linux Mint.

Results v0.8.0 17-05-2021 (safe-decode and safe-encode on)

cargo bench

Executed on Core i7-6700 Linux Mint.

Miri

Miri can be used to find issues related to incorrect unsafe usage:

MIRIFLAGS="-Zmiri-disable-isolation -Zmiri-disable-stacked-borrows" cargo miri test --no-default-features

Fuzzer

This fuzz target generates corrupted data for the decompressor. Make sure to switch to the checked_decode version in fuzz/Cargo.toml before testing this. cargo fuzz run fuzz_decomp_corrupt_block and cargo fuzz run fuzz_decomp_corrupt_frame

This fuzz target asserts that a compression and decompression rountrip returns the original input. cargo fuzz run fuzz_roundtrip and cargo fuzz run fuzz_roundtrip_frame

This fuzz target asserts compression with cpp and decompression with lz4_flex returns the original input. cargo fuzz run fuzz_roundtrip_cpp_compress

TODO

• High compression