Direct no-stdlib port of the C brotli compressor to Rust
no dependency on the Rust stdlib: this library would be ideal for decompressing within a rust kernel among other things.
This is useful to see how C and Rust compare in an apples-to-apples comparison where the same algorithms and data structures and optimizations are employed.
Rust brotli currently supports compression levels 0 - 11 They should be bitwise identical to the brotli C compression engine at compression levels 0-9 Recommended lgwindowsize is between 20 and 22
rust
let mut input = brotli::CompressorReader::new(&mut io::stdin(), 4096 /* buffer size */,
quality as u32, lg_window_size as u32);
then you can simply read input as you would any other io::Read class
rust
let mut writer = brotli::Compressor::new(&mut io::stdout(), 4096 /* buffer size */,
quality as u32, lg_window_size as u32);
There are also methods to build Compressor Readers or Writers using the with_params static function
eg:
rust
let params = BrotliEncoderParams::default();
// modify params to fit the application needs
let mut writer = brotli::Compressor::with_params(&mut io::stdout(), 4096 /* buffer size */,
params);
or for the reader
rust
let params = BrotliEncoderParams::default();
// modify params to fit the application needs
let mut writer = brotli::CompressorReader::with_params(&mut io::stdin(), 4096 /* buffer size */,
params);
rust
match brotli::BrotliCompress(&mut io::stdin(), &mut io::stdout(), &brotli_encoder_params) {
Ok(_) => {},
Err(e) => panic!("Error {:?}", e),
}
rust
let mut input = brotli::Decompressor::new(&mut io::stdin(), 4096 /* buffer size */);
then you can simply read input as you would any other io::Read class
rust
let mut writer = brotli::DecompressorWriter::new(&mut io::stdout(), 4096 /* buffer size */);
rust
match brotli::BrotliDecompress(&mut io::stdin(), &mut io::stdout()) {
Ok(_) => {},
Err(e) => panic!("Error {:?}", e),
}
There are 3 steps to using brotli without stdlib
in Detail
```rust // at global scope declare a MemPool type -- in this case we'll choose the heap to // avoid unsafe code, and avoid restrictions of the stack size
declarestackallocator_struct!(MemPool, heap);
// at local scope, make a heap allocated buffers to hold uint8's uint32's and huffman codes
let mut u8buffer = defineallocatormemorypool!(4096, u8, [0; 32 * 1024 * 1024], heap);
let mut u32buffer = defineallocatormemorypool!(4096, u32, [0; 1024 * 1024], heap);
let mut hcbuffer = defineallocatormemorypool!(4096, HuffmanCode, [0; 4 * 1024 * 1024], heap);
let heapu8allocator = HeapPrealloc::
// At this point no more syscalls are going to be needed since everything can come from the allocators.
// Feel free to activate SECCOMP jailing or other mechanisms to secure your application if you wish.
// Now it's possible to setup the decompressor state let mut brotlistate = BrotliState::new(heapu8allocator, heapu32allocator, heaphc_allocator);
// at this point the decompressor simply needs an input and output buffer and the ability to track // the available data left in each buffer loop { result = BrotliDecompressStream(&mut availablein, &mut inputoffset, &input.slice(), &mut availableout, &mut outputoffset, &mut output.slicemut(), &mut written, &mut brotlistate);
// just end the decompression if result is BrotliResult::ResultSuccess or BrotliResult::ResultFailure
} ```
This interface is the same interface that the C brotli decompressor uses
Also feel free to use custom allocators that invoke Box directly. This example illustrates a mechanism to avoid subsequent syscalls after the initial allocation
rust-brotli is a drop-in replacement for the official http://github.com/google/brotli/ C implementation. That means you can use it from any place that supports that library. To build rust-brotli in this manner enter the c subdirectory and run make there
cd c && make
this should build c/target/release/libbrotli.so and should build the vanilla command line tool in C for compressing and decompressing any brotli file.
the libbrotli.so in c/target/release should be able to replace any other libbrotli.so file, but with all the advantages of using safe rust (except in the FFI bindings)
The code also allows a wider range of options, including forcing the prediction mode (eg UTF8 vs signed vs MSB vs LSB) and changing the weight of the literal cost from 540 to other values.
Additionally the CATABLE and APPENDABLE options are exposed and allow concatenation of files created in this manner.
Specifically CATABLE files can be concatenated in any order using the catbrotli tool and APPENDABLE files can be the first file in a sequence of catable files... eg you can combine appendable.br catable1.br catable2.br catable3.br
or simply catable0.br catable1.br catable2.br catable3.br