Fuzzcheck is a structure-aware and coverage-guided fuzzing engine for Rust functions.
Given a function test: (T) -> bool, you can use fuzzcheck to find a value of
type T that fails the test or leads to a crash.
Fuzzcheck works by maintaining a pool of test inputs and ranking them using
the uniqueness of the code coverage caused by running test(input).
From that pool, it selects a high-ranking input, mutates it, and runs the test
function again. If the new mutated input has an interesting code coverage then
it is added to the pool, otherwise, fuzzcheck tries again with a different
input and mutation.
In pseudocode:
```rust loop { let input = pool.select(); mutate(&mut input);
let analysis = analyze(test, &input);
match analysis {
Failed => reportFailure(input),
Interesting(score) => pool.add(input, score),
NotInteresting => continue
}
} ```
Fuzzcheck is unique because, unlike other coverage-guided fuzzing engines, it
doesn't work with bitstrings but instead works with values of any type T
directly. The complexity of the inputs and the way to mutate them is given by
functions defined by the user.
Rust nightly is required. You can install it with:
rustup toolchain install nightly
While it is not strictly necessary, installing the cargo-fuzzcheck
executable will make it easier to run fuzzcheck.
bash
cargo install cargo-fuzzcheck
In you Cargo.toml file, add fuzzcheck as a dev dependency:
toml
[dev-dependencies]
fuzzcheck = "0.7"
Then, we need a way to serialize values. By default, fuzzcheck uses serde_json for that purpose (but it can be changed).
That means our data types should implement serde's traits. In Cargo.toml, add:
[dependencies]
serde = { version = "1.0", features = ["derive"] }
Below is an example of how to use fuzz test. Note:
1. every code related to fuzzcheck is conditional on #[cfg(test)] because we
don't want to carry the fuzzcheck dependency in normal builds
2. the #![cfg_attr(test, feature(no_coverage))] that is required by fuzzcheck’s procedural macros
3. the use of derive(fuzzcheck::DefaultMutator) to make a custom type fuzzable
```rust // this nightly feature is required by fuzzcheck’s procedural macros
// The DefaultMutator macro creates a mutator for a custom type
// The mutator is accessible via SampleStruct::
// the fuzzer needs to serialize and deserialize test cases, // we use serde by default, but that can be changed
struct SampleStruct
enum SampleEnum { A(u16), B, C { x: bool, y: bool }, }
fn shouldnotcrash(xs: &[SampleStruct
// fuzz tests reside along your other tests and have the #[test] attribute
mod tests {
use super::*;
use fuzzcheck::{FuzzerBuilder, DefaultMutator, SerdeSerializer};
#[test]
fn testfunctionshouldntcrash() {
FuzzerBuilder::test(shouldnotcrash) // first give the function to test
// second, the mutator to generate the function’s inputs
.mutator(
We can now use cargo-fuzzcheck to launch the test, using Rust nightly:
```sh
rustup override set nightly
cargo fuzzcheck tests::testfunctionshouldntcrash fuzz --artifacts fuzz/artifacts ```
This starts a loop that will stop when a failing test has been found.
A line will be printed whenever a newsworthy event happened, along with some statistics. For example:
NEW 7825 score: 18.70 pool: 7 exec/s: 728516 cplx: 41.29
NEW means that a new input was added to the pool of interesting inputs7825 is the number of iterations that were performed so farscore: 18.70 is a measure of the total code coverage caused by all inputs
in the poolpool: 7 is the number of inputs in the poolexec/s: 728516 is the average number of iterations performed every secondcplx: 41.29 is the average complexity of the inputs in the poolWhen a failing test has been found, the following is printed:
================ TEST FAILED ================
13024 score: 20.90 pool: 7 exec/s: 1412576 cplx: 41.29
Saving at "fuzz/artifacts/59886edc1de2dcc1.json"
Here, the path to the artifact file is
fuzz/artifacts/59886edc1de2dcc1.json.
It contains the JSON-encoded input that failed the test.
json
[
{
"x": 100,
"y": {
"C": {
"x": false,
"y": true
}
}
},
{
"x": 55,
"y": {
"C": {
"x": true,
"y": false
}
}
},
..
]
Moreover, the fuzzer can maintain a copy of its input pool in the file system
by passing the argument --out-corpus <folder path>. Fuzzing corpora
are useful to kick-start a fuzzing process by providing a list of known
interesting inputs through the option --in-corpus <folder path>.
Fuzzcheck can also be used to minify a large input that fails a test.
Let's say you have a file crash.json containing an input that you would like
to minify. Launch cargo fuzzcheck <exact name of fuzz test> with the tmin command
and an --input-file option.
bash
cargo fuzzcheck "tests::test_function_shouldn_t_crash" tmin --input-file "crash.json"
This will repeatedly launch the fuzzer in “minify” mode and save the
artifacts in the folder artifacts/crash.minified. The name of each artifact
will be prefixed with the complexity of its input. For example,
crash.minified/800--fe958d4f003bd4f5.json has a complexity of 8.00.
You can stop the minifying fuzzer at any point and look for the least complex
input in the crash.minified folder.
If you would like to fuzz-test your own custom type T without the
DefaultMutator derive attribute or the make_mutator! procedural macro,
you will have to create a type that conforms to the Mutator<T> trait.
Ask for help in the Github issues or send me an email if you would like some help or advice on how to write a good mutator.
As far as I know, evolutionary, coverage-guided fuzzing engines were
popularized by American Fuzzy Lop (AFL).
Fuzzcheck is also evolutionary and coverage-guided.
Later on, LLVM released its own fuzzing engine,
libFuzzer, which is based on the
same ideas as AFL, but it uses Clang’s
SanitizerCoverage and is
in-process (it lives in the same process as the program being fuzz-tested.
Fuzzcheck is also in-process and also uses SanitizerCoverage.
Both AFL and libFuzzer work by manipulating bitstrings (e.g. 1011101011).
However, many programs work on structured data, and mutations at the
bitstring level may not map to meaningful mutations at the level of the
structured data. This problem can be partially addressed by using a compact
binary encoding such as protobuf and providing custom mutation functions to
libFuzzer that work on the structured data itself. This is a way to perform
“structure-aware fuzzing” (talk,
tutorial).
An alternative way to deal with structured data is to use generators just like
QuickCheck’s Arbitrary trait. And then to “treat the raw byte buffer input
provided by the coverage-guided fuzzer as a sequence of random values and
implement a “random” number generator around it.”
(cited blog post by @fitzgen).
The tool cargo-fuzz has
recently
implemented that approach.
Fuzzcheck is also structure-aware, but unlike previous attempts at
structure-aware fuzzing, it doesn't use an intermediary binary encoding such as
protobuf nor does it use Quickcheck-like generators.
Instead, it directly mutates the typed values in-process.
This is better many ways. First, it is faster because there is no
need to encode and decode inputs at each iteration. Second, the complexity of
the input is given by a user-defined function, which will be more accurate than
counting the bytes of the protobuf encoding.
Finally, and most importantly, the mutations are faster and more meaningful
than those done on protobuf or Arbitrary’s byte buffer-based RNG.
A detail that I particularly like about fuzzcheck, and that is possible only
because it mutates typed values, is that every mutation is done in-place
and is reversable. That means that generating a new test case is super fast,
and can often even be done with zero allocations.
As I was developing Fuzzcheck for Swift, a few researchers developed Fuzzchick
for Coq (paper). It
is a coverage-guided property-based testing tool implemented as an extension to
Quickchick. As far as I know, it is the only other tool with the same philosophy
as fuzzcheck. The similarity between the names fuzzcheck and Fuzzchick is a
coincidence.