Run tests with statements and method calls removed to help identify broken tests
Install from [crates.io]:
cargo install necessist --version=^0.1.0-beta
Install from [github.com]:
cargo install --git https://github.com/trailofbits/necessist --branch release
The following hypothetical test verifies that a login mechanism works. Suppose the test would pass if session.send_password(...) were removed. This could indicate that the wrong condition is checked thereafter. Or worse, it could indicate a bug in the login mechanism.
```rust
fn loginworks() { let session = Session::new(URL); session.sendusername(USERNAME).unwrap(); session.send_password(PASSWORD).unwrap(); // <-- Test passes without this assert!(session.read().unwrap().contains(WELCOME)); } ```
Necessist iteratively removes statements and method calls from tests and then runs them to help identify such cases.
Generally speaking, Necessist will not attempt to remove a statement if it is one the following:
for loop)let binding)break, continue, or returnAlso, for some frameworks, certain statements and methods are ignored (see below).
```
Usage: necessist [OPTIONS] [TEST_FILES]... [--
Arguments: [TEST_FILES]... Test files to mutilate (optional) [ARGS]... Additional arguments to pass to each test command
Options:
--allow --allow all silences all warnings
--default-config Create a default necessist.toml file in the project's root directory (experimental)
--deny --deny all treats all warnings as errors
--dump Dump sqlite database contents to the console
--framework passed
-h, --help Print help
-V, --version Print version
```
By default, Necessist outputs to both the console and to an sqlite database. For the latter, a tool like sqlitebrowser can be used to filter/sort the results.
By default, Necessist outputs only when tests pass. Passing --verbose causes Necessist to instead output all of the removal outcomes below.
| Outcome | Meaning (With the statement/method call removed...) | | -------------------------------------------- | --------------------------------------------------- | | passed | The test(s) built and passed. | | timed-out | The test(s) built but timed-out. | | failed | The test(s) built but failed. | | nonbuildable | The test(s) did not build. |
Click on a framework to see its specifics.
Foundry
In addition to the below, the Foundry framework ignores:
vm.prank or any form of vm.expect (e.g., vm.expectRevert)emit statementassert (e.g., assertEq)expectEmitexpectRevertprankstartPrankstopPrankGolang
In addition to the below, the Golang framework ignores:
assert (e.g., assert.Equal)require (e.g., require.Equal)defer statementsCloseErrorErrorfFailFailNowFatalFatalfLogLogfParallel* This list is based primarily on [testing.T]'s methods. However, some methods with commonplace names are omitted to avoid colliding with other types' methods.
Hardhat TS
assert (e.g., assert.equal)expectshould (e.g., should.equal)to (e.g., to.equal)toNumbertoStringRust
assertassert_eqassert_matchesassert_neeprinteprintlnpanicprintprintlnunimplementedunreachableas_bytesas_mutas_mut_os_stras_mut_os_stringas_mut_sliceas_mut_stras_os_stras_pathas_refas_sliceas_strborrowborrow_mutcloneclonedcopiedderefderef_mutinto_boxed_bytesinto_boxed_os_strinto_boxed_pathinto_boxed_sliceinto_boxed_strinto_bytesinto_os_stringinto_ownedinto_path_bufinto_stringinto_veciteriter_mutsuccessto_os_stringto_ownedto_path_bufto_stringto_vecunwrapunwrap_err* This list is essentially the watched trait and inherent methods of Dylint's [unnecessary_conversion_for_trait] lint, with the following additions:
clone (e.g. [std::clone::Clone::clone])cloned (e.g. [std::iter::Iterator::cloned])copied (e.g. [std::iter::Iterator::copied])into_owned (e.g. [std::borrow::Cow::into_owned])success (e.g. [assert_cmd::assert::Assert::success])unwrap (e.g. [std::option::Option::unwrap])unwrap_err (e.g. [std::result::Result::unwrap_err])Configuration files are experimental and their behavior could change at any time.
A configuration file allows one to tailor Necessist's behavior with respect to a project. The file must be named necessist.toml, appear in the project's root directory, and be [toml] encoded. The file may contain one more of the options listed below.
ignored_functions: A list of strings. Functions whose names appear in the list are ignored.ignored_macros: A list of strings. Macros whose names appear in the list are ignored.Slow. Modifying tests requires them to be rebuilt. Running Necessist on even moderately sized codebases can take several hours.
Diagnosis requires intimate knowledge of the source code. Generally speaking, Necessist does not produce "obvious" bugs. In our experience, deciding whether a test statement should be necessary requires intimate knowledge of the code under test. Necessist is best run on codebases for which one has (or intends to have) such knowledge.
cding into the project's directory and typing necessist (with no arguments) should produce meaningful output.Necessist is licensed and distributed under the AGPLv3 license. Contact us if you're looking for an exception to the terms.