Testing with pep-test

pep-test builds the [Test] functions of Pepper files and runs them, each in a child process of its own, reporting which pass. Writing tests is covered in Unit tests.

pepper test                          # every test under the current folder
pepper test src/words_test.pep       # the tests of one file
pepper test -f words src             # only tests whose name contains "words"
pepper test --list src               # list them without running them
pepper test -I . geometry/tests      # tests that import the module geometry

running 3 tests
test counts_the_words ... ok (4 ms)
test an_empty_string_has_no_words ... ok (3 ms)
test reading_past_the_last_word_panics ... ok (5 ms)
3 passed, 0 failed

pep-test is written in Pepper (tools/pep-test). Build it with the compiler and put it next to pepper and pep-build, so that pepper test finds it and it finds the compiler:

target/bin/pep-build tools/pep-test/main.pep -o target/bin/pep-test

Usage and options

pep-test [options] [path]...

The exit status is 0 when every test program builds and every test passes, 1 when a test fails or a program does not build, and 2 for a usage error.

What it does

1. Finds the files. A file named on the command line is tested as it is. In a folder, and the folders below it (except those whose names start with .), it takes each .pep file with a line that starts with [Test].
2. Builds one test program per folder, of that folder’s test files, with pep-build --test (found next to pep-test, else on the PATH), in parallel. Each file keeps the scope it has in any program, so test files in a module’s folder see each other’s names, and a helper two of them declare must be private (see Unit tests). A program that does not build is reported with the compiler’s errors, and the other folders’ tests still run.
3. Lists each program’s tests with --list and keeps those that match --filter. With --list it prints them and stops: the test’s name, or file::name when the tests come from more than one file.
4. Runs each test as program --run FILE::NAME --timeout SECONDS, each in its own process, up to --jobs at once, and prints test NAME ... ok (TIME) or ... FAILED (TIME) as each ends (in order with --jobs 1). TIME is how long the test’s process ran: whole milliseconds (12 ms), or seconds with two decimals from one second up (1.25 s). A test passes when its process exits with 0; a [ShouldPanic] test passes when its process panics with a message that contains the attribute’s text.
5. Reports the failures, each with what the test printed (the panic’s message and location) and why it failed, then a summary: 2 passed, 1 failed, with the number of tests --filter left out and of test programs that did not build when there are any.

The test programs are built in a temporary folder that is removed at the end. A test runs in the current folder, with PEPPER_BACKTRACE=0 unless PEPPER_BACKTRACE is set, so a panic prints its message without a backtrace. Each test gets a temporary folder of its own (TMPDIR, and TMP and TEMP for Windows, name it), removed when the test ends, so what a test that panics or times out leaves there, such as a TempFolder whose destructor never ran, is removed too.

The test program

pep-build --test builds a folder’s test program (see pep-build), which pep-test runs with one of two requests:

• --list prints a line per test, its fields separated by tabs: the name, file:line, returns or panics (a [ShouldPanic] test), and the text the panic must contain.
• --run NAME [--timeout SECONDS] runs one test (NAME, or FILE::NAME with the file as --list gives it) and exits with 0 when it returns having released everything it allocated; a test that leaves an allocation behind (or releases one too many) exits with 1 and says so on standard error. A panic aborts the program; a test that runs past the timeout ends it with exit status 124. However the program ends (the test returns, panics, crashes or times out), every process the test started ends with it (a job object on Windows, a process group elsewhere): one left running would hold the pipes to its output open, and pep-test would wait on it.

On Windows, a test program turns off Windows Error Reporting for its crashes and abort(), so a test that panics or crashes ends at once, without a dialog.