This crate provides a simple language testing framework designed to help when
you are testing things like compilers and virtual machines. It allows users to
embed simple tests for process success/failure and for stderr/stdout inside a
source file. It is loosely based on the
compiletest_rs crate, but is much
simpler (and hence sometimes less powerful), and designed to be used for
testing non-Rust languages too.
For example, a Rust language tester, loosely in the spirit of
compiletest_rs, looks as follows:
```rust use std::{path::PathBuf, process::Command};
use lang_tester::LangTester; use tempdir::TempDir;
fn main() {
// We use rustc to compile files into a binary: we store those binary files
// into tempdir. This may not be necessary for other languages.
let tempdir = TempDir::new("rustlangtester").unwrap();
LangTester::new()
.testdir("examples/rustlangtester/langtests")
// Only use files named *.rs as test files.
.testfilefilter(|p| p.extension().unwrap().tostr().unwrap() == "rs")
// Extract the first sequence of commented line(s) as the tests.
.testextract(|s| {
Some(
s.lines()
// Skip non-commented lines at the start of the file.
.skipwhile(|l| !l.startswith("//"))
// Extract consecutive commented lines.
.takewhile(|l| l.startswith("//"))
.map(|l| &l[2..])
.collect::Compiler: runs rustc.
// * Run-time: if rustc does not error, and the Compiler tests
// succeed, then the output binary is run.
.testcmds(move |p| {
// Test command 1: Compile x.rs into tempdir/x.
let mut exe = PathBuf::new();
exe.push(&tempdir);
exe.push(p.filestem().unwrap());
let mut compiler = Command::new("rustc");
compiler.args(&["-o", exe.tostr().unwrap(), p.tostr().unwrap()]);
// Test command 2: run tempdir/x.
let runtime = Command::new(exe);
vec![("Compiler", compiler), ("Run-time", runtime)]
})
.run();
}
```
This defines a lang tester that uses all *.rs files in a given directory as
test files, running two test commands against them: Compiler (i.e. rustc);
and Run-time (the compiled binary).
Users can then write test files such as the following:
rust
// Compiler:
// status: success
// stderr:
// warning: unused variable: `x`
// ...unused_var.rs:12:9
// ...
//
// Run-time:
// status: success
// stdout: Hello world
fn main() {
let x = 0;
println!("Hello world");
}
Test data is specified with a two-level indentation syntax: the outer most level of indentation defines a test command (multiple command names can be specified, as in the above); each test command can then define sub-tests for one or more of:
status: <success|failure|<int>>, where success and failure map to
platform specific notions of a command completing successfully or
unsuccessfully respectively and <int> is a signed integer checking for a
specific exit code on platforms that support it.stderr: [<string>], stdout: [<string>] are matched strings against a
command's stderr or stdout. The special string ... can be used as a
simple wildcard: if a line consists solely of ..., it means "match zero
or more lines"; if a line begins with ..., it means "match the remainder
of the line only"; if a line ends with ..., it means "match the start of
the line only". A line may start and end with .... Note that
stderr/stdout matches ignore leading/trailing whitespace and newlines,
but are case sensitive.The above file thus contains 4 actual tests: the Compiler should succeed
(e.g. return a 0 exit code when run on Unix), and its stderr output should
warn about an unused variable on line 12; and the resulting binary should
succeed and produce Hello world on stdout.
Potential sub-tests that are not mentioned are not tested: for example, the
above file does not state whether the Compilers stdout should have content
or not (but note that the line stdout: on its own would state that the
Compiler should have no content at all).
lang_tester's output is deliberately similar to Rust's normal testing output.
Running the example rust_lang_tester in this crate produces the following
output:
``text
$ cargo run --example=rust_lang_tester
Compiling lang_tester v0.1.0 (/home/ltratt/scratch/softdev/lang_tester)
Finished dev [unoptimized + debuginfo] target(s) in 3.49s
Runningtarget/debug/examples/rustlangtester`
running 4 tests test langtests::nomain ... ok test langtests::unknownvar ... ok test langtests::unusedvar ... ok test langtests::exitcode ... ok
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out ```
If you want to run a subset of tests, you can specify simple filters which use substring match to run a subset of tests:
``text
$ cargo run --example=rust_lang_tester var
Compiling lang_tester v0.1.0 (/home/ltratt/scratch/softdev/lang_tester)
Finished dev [unoptimized + debuginfo] target(s) in 3.37s
Runningtarget/debug/examples/rustlangtester var`
running 2 tests test langtests::unknownvar ... ok test langtests::unusedvar ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 2 filtered out ```