X-Git-Url: http://nitlanguage.org diff --git a/tests/README.md b/tests/README.md index c1b4cff..8941587 100644 --- a/tests/README.md +++ b/tests/README.md @@ -17,6 +17,8 @@ This directory contains small Nit programs and useful scripts to test the non re Useful before committing something. * `listfull.sh` lists tests that are run by `testfull.sh`. It is used by other scripts but can be used by human to see what is tested. + Each path is quoted using apostrophes, with `'\''` escaping an apostrophe. + This syntax is compatible with `xargs` and the POSIX shell. Currently, all files in `tests/`, `lib/` and `examples/` are executed, some of `contrib/` and the main programs of `src/`. To update the list of tested files, just edit this script. @@ -154,11 +156,14 @@ It is a failure, and analogous to the standard `fail`. `[todo] out/zzz_test_todo.res -> not yet implemented` -The produced result file contains the magic string "NOT YET IMPLEMENTED". +The produced result file contains a magic string, like `NOT YET IMPLEMENTED`. Those are considered the same as expected errors (like a fixme) It is a success. -Some engines, libraries or program just print this to simplify the management of tests. +The magic strings are listed in `todo` files in the root and `sav` directories. +They are used by engines, libraries or program just print this to simplify the management of tests. + +Magic strings are used with `grep -f`, so each line is a pattern that is searched within the res files. ### Skipped @@ -170,12 +175,18 @@ It is a success. The `$engine.skip` files (where `$engine` is an engine name, see below) describe tests that are skipped completely on a given engine. Usually it used with then engine `niti` because tests are too long. -The `cc.skip` file describes tests that are analysed but no executable is generated. -Usually it is because expected CC errors or missing C libraries. +The `cc.skip` file describes tests that are analyzed but no executable is generated. +Usually it is because of expected CC errors or missing C libraries. The `exec.skip` file describes tests that compiled but not executed. Usually it is because the programs are interactive or run some kind of server. +The `$os.skip` file describes tests that are to be skipped completely on the given OS. +Usually it is because of OS specific libraries. + +The `turing.skip` file describes tests that are to be skipped completely on the Turing cluster doing continuous testing over MPI. +Usually it is because of an unavailable library or a large work which would not benefit from parallelization. + These `*.skip` files contain a list of patterns that will be used against test names. A single substring can thus be used to skip a full family of tests. @@ -202,18 +213,18 @@ Use the various skipping or controls to try to produce reproducible results. Engines are selected with the option `--engine`. ~~~ -./tests.sh --engine nitg-e base_class_name.nit +./tests.sh --engine nitce base_class_name.nit ~~~ Current engines are: -* `nitg-s`, for `nitg --separate` (this is the default) -* `nitg-e`, for `nitg --erasure` -* `nitg-sg`, for `nitg --separate --semi-global` -* `nitg-g`, for `nitg --global` +* `nitcs`, for `nitc --separate` (this is the default) +* `nitce`, for `nitc --erasure` +* `nitcsg`, for `nitc --separate --semi-global` +* `nitcg`, for `nitc --global` * `niti`, for `nit`, the interpreter -* `nitvm`, for `nitvm` (not automatically executed by `testall.sh`) -* `emscripten`, for `nitg --semi-global -m emscripten` (not automatically executed by `testall.sh`) +* `nitvm`, for `nit --vm`, the virtual machine +* `emscripten`, for `nitc --semi-global -m emscripten` (not automatically executed by `testall.sh`) Engines control: @@ -313,6 +324,11 @@ $ cat out/zzz_test_post_proc.res The environment variable `NIT_TESTING` is set to `true` during the execution of program tests. Some libraries of programs can use it to produce specific reproducible results ; or just to exit their executions. +`NIT_TESTING_TESTS_SH` is set to `true` for tests ran by `tests.sh` only, not by nitunit. + +`NIT_TESTING_ID` is set to an integer value unique between concurrent executions of `tests.sh`. +Use it to name files, database entries, open sockets in order to avoid race conditions. + ~~~ $ cat zzz_tests/zzz_test_envvar.nit $ ./tests.sh zzz_tests/zzz_test_envvar.nit