Run tests with statements and method calls removed to help identify broken tests
Install from [crates.io]:
cargo install necessist --version=^0.1.0-beta
If you require [Foundry] support, install from [github.com]:
cargo install --git https://github.com/trailofbits/necessist --branch release
The following hypothetical test verifies that a login mechanism works. Suppose the test would pass if session.send_password(...) were removed. This could indicate that the wrong condition is checked thereafter. Or worse, it could indicate a bug in the login mechanism.
```rust
fn loginworks() { let session = Session::new(URL); session.sendusername(USERNAME).unwrap(); session.send_password(PASSWORD).unwrap(); // <-- Test passes without this assert!(session.read().unwrap().contains(WELCOME)); } ```
Necessist iteratively removes statements and method calls from tests and then runs them to help identify such cases.
Generally speaking, Necessist will not attempt to remove a statement if it is one the following:
for loop)let binding)break, continue, or returnAlso, for some frameworks, certain statements and methods are ignored (see below).
``` Usage: necessist [OPTIONS] [TEST_FILES]...
Arguments: [TEST_FILES]... Test files to mutilate (optional)
Options:
--allow --allow all silences all warnings
--default-config Create a default necessist.toml file in the project's root directory (experimental)
--deny --deny all treats all warnings as errors
--dump Dump sqlite database contents to the console
--framework passed
-h, --help Print help information
-V, --version Print version information
```
By default, Necessist outputs to both the console and to an sqlite database. For the latter, a tool like sqlitebrowser can be used to filter/sort the results.
By default, Necessist outputs only when tests pass. Passing --verbose causes Necessist to instead output all of the removal outcomes below.
| Outcome | Meaning (With the statement/method call removed...) | | -------------------------------------------- | --------------------------------------------------- | | passed | The test(s) built and passed. | | timed-out | The test(s) built but timed-out. | | failed | The test(s) built but failed. | | nonbuildable | The test(s) did not build. |
In addition to the below, the Foundry framework ignores:
vm.prank or any form of vm.expect (e.g., vm.expectRevert)assert (e.g., assertEq)expectEmitexpectRevertprankstartPrankstopPrankassert (e.g., assert.equal)expectshould (e.g., should.equal)to (e.g., to.equal)toNumbertoStringassertassert_eqassert_neeprinteprintlnpanicprintprintlnunimplementedunreachableas_bytesas_mutas_mut_sliceas_mut_stras_os_stras_pathas_refas_sliceas_strborrowborrow_mutcloneclonedcopiedderefderef_mutinto_boxed_bytesinto_boxed_os_strinto_boxed_pathinto_boxed_sliceinto_boxed_strinto_bytesinto_os_stringinto_ownedinto_path_bufinto_stringinto_veciteriter_mutleaksuccessto_os_stringto_ownedto_path_bufto_stringto_vecunwrapunwrap_err* This list is essentially the watched trait and inherent methods of Dylint's [unnecessary_conversion_for_trait] lint, with the following additions:
clone (e.g. [std::clone::Clone::clone])cloned (e.g. [std::iter::Iterator::cloned])copied (e.g. [std::iter::Iterator::copied])into_owned (e.g. [std::borrow::Cow::into_owned])success (e.g. [assert_cmd::assert::Assert::success])unwrap (e.g. [std::option::Option::unwrap])unwrap_err (e.g. [std::result::Result::unwrap_err])Configuration files are experimental and their behavior could change at any time.
A configuration file allows one to tailor Necessist's behavior with respect to a project. The file must be named necessist.toml, appear in the project's root directory, and be [toml] encoded. The file may contain one more of the options listed below.
ignored_functions: A list of strings. Functions whose names appear in the list are ignored.ignored_macros: A list of strings. Macros whose names appear in the list are ignored.cding into the project's directory and typing necessist (with no arguments) should produce meaningful output.