1 # Regression test for the Nit project
3 ## Content of the directory
5 This directory contains small Nit programs and useful scripts to test the non regression of the nit tools.
9 * `tests.sh` is the main test script used to run some specific tests.
10 * `testall.sh` runs some tests on all engines.
11 (No more excuse to forget to check your new tests with an obscure engine).
12 * `testfull.sh` runs all tests on a single engine.
13 (go and grab a coffee)
14 * `search_test.sh` lists tests given a testname, a basename, or a `.res` file.
15 Useful when `foo failed` but you do not know where the program `foo` is.
16 * `search_tests_git.sh` lists tests that seem modified between 2 git revisions.
17 Useful before committing something.
18 * `listfull.sh` lists tests that are run by `testfull.sh`.
19 It is used by other scripts but can be used by human to see what is tested.
20 Each path is quoted using apostrophes, with `'\''` escaping an apostrophe.
21 This syntax is compatible with `xargs` and the POSIX shell.
23 Currently, all files in `tests/`, `lib/` and `examples/` are executed, some of `contrib/` and the main programs of `src/`.
24 To update the list of tested files, just edit this script.
26 ### Small Nit programs
28 They are divided in categories:
30 * `base` are minimal and fundamental tests.
31 They are small and test the fundamental language specifications.
32 * `bench` are memory and time tests
33 * `error` are invalid programs
34 They test the error message and the good behavior of the tool.
35 Nowadays, alternatives (see below) are more used to test erroneous programs.
36 * `example` are examples
37 Nowadays, examples are developed in the `examples/` root directory.
38 * `test` are others tests that usually depend on standard or other libraries.
39 Nowadays, nitunit are a better place for them.
40 * `shootout` are benchmarks from http://shootout.alioth.debian.org/
44 The .gitignore contains specific artefacts produced by some tests.
45 However, it is better to produce these artifacts in the `out` directory instead,
46 because `out` is cleaned before running tests (so that old artefacts do not
47 interfere with new executions of tests)
49 The `sav/` directory contains reference result files for tests (see below)
51 The `alt/` and `out/` directory are transient directories used during the tests.
53 The `zzz_tests/` directory contains tests to test the test system.
54 Execute `./tests.ss zzz_tests/*.nit` if you want.
59 ./tests.sh filepath.nit...
60 ./tests.sh base_attr*.nit
63 Will execute the program filepath for all its alternative (see below) and all its arguments (see below) for the default engine (see below).
65 The output is something like
67 => test_args: . [ok] out/test_args.res sav/test_args.res
70 where in order, there is
72 * The test description (`test_args` in the example).
73 It is the basename of the file + information about alternatives and arguments.
74 * The build status: `.`, `!`, or `nocc`, or `_`.
75 They respectively mean "compilation OK", "compilation error", "compilation skipped", "compilation OK but execution skipped".
76 * The test status (see below) where `out/*.res` is the produced result file, and `sav/*.res` is the reference result file.
83 `[ok] out/zzz_test_ok.res sav/zzz_test_ok.res`
85 The produced result file correspond to the reference result file (according to `diff`).
90 `[0k] out/zzz_test_0k.res is empty`
92 The produced result file is empty and no reference result file is present.
95 If you introduced a new library, it is likely that you get this.
96 It basically means that the module compile and does noting, and its fine.
100 `[======= fail out/zzz_test_fail.res sav/zzz_test_fail.res =======]`
102 The produced result file do not correspond to the reference result file.
105 Did something break? or does the reference result file need to be updated?
109 `[=== no sav ===] out/zzz_test_nosav.res is not empty`
111 The produced result file is not empty but no reference result file is present.
114 Did something break? or does the reference result file need to be created?
115 If you introduced a new program, it is likely that you get this.
116 It basically means the program compiles and produces something.
117 Please create a new reference result file for it.
121 `[======= soso out/zzz_test_soso.res sav//zzz_test_soso.res =======]`
123 The produced result file do not correspond to the reference result file.
124 But the difference in only in errors or warnings.
127 Usually, some lines in error messages changed.
128 So just update the reference result file.
132 `[fixme] out/zzz_test_fixme.res sav//fixme/zzz_test_fixme.res`
134 The produced result file correspond to a *fixme* reference result file.
135 It is an expected error. So not a regression.
137 Reference result files in `sav/**/fixme/` are considered expected errors.
138 The distinction with standard reference result files is only for final statistics.
140 ### Ok, but fixme remains
142 `[*ok*] out/zzz_test_fixme_remains.res sav//zzz_test_fixme_remains.res - but sav//fixme/zzz_test_fixme_remains.res remains!`
144 There is a fixme reference result file and a standard reference result file. But the produced result file correspond to the standard one.
145 Usually it means that a bug was fixed. But is considered a regression until the *fixme* reference result file is removed.
150 `[======= changed out/zzz_test_fixme_changed.res sav//fixme/zzz_test_fixme_changed.res ======]`
152 The produced result file do not correspond to the *fixme* reference result file.
153 It is a failure, and analogous to the standard `fail`.
155 ### Todo, not yet implemented
157 `[todo] out/zzz_test_todo.res -> not yet implemented`
159 The produced result file contains a magic string, like `NOT YET IMPLEMENTED`.
160 Those are considered the same as expected errors (like a fixme)
163 The magic strings are listed in `todo` files in the root and `sav` directories.
164 They are used by engines, libraries or program just print this to simplify the management of tests.
166 Magic strings are used with `grep -f`, so each line is a pattern that is searched within the res files.
175 The `$engine.skip` files (where `$engine` is an engine name, see below) describe tests that are skipped completely on a given engine.
176 Usually it used with then engine `niti` because tests are too long.
178 The `cc.skip` file describes tests that are analyzed but no executable is generated.
179 Usually it is because of expected CC errors or missing C libraries.
181 The `exec.skip` file describes tests that compiled but not executed.
182 Usually it is because the programs are interactive or run some kind of server.
184 The `$os.skip` file describes tests that are to be skipped completely on the given OS.
185 Usually it is because of OS specific libraries.
187 The `turing.skip` file describes tests that are to be skipped completely on the Turing cluster doing continuous testing over MPI.
188 Usually it is because of an unavailable library or a large work which would not benefit from parallelization.
190 These `*.skip` files contain a list of patterns that will be used against test names.
191 A single substring can thus be used to skip a full family of tests.
193 ## Update reference result files
195 To update the reference result files, just create/remove/update files in the `sav/` directory.
197 HINT: for easy management, just copy-paste parts of the test status to build your command.
198 Example `cp <copy-paste out/zzz_test_fail.res sav/zzz_test_fail.res>` to update a file.
201 If a reference result file contains only `UNDEFINED`, then the produced result file is always considered successful.
202 It is used to store system-dependant *fixme* reference result files.
203 Once the problem is solved, the status will become `Ok but fixme remains`
206 Note: `UNDEFINED` still gives a success even if the tests is uncompilable.
207 So do not use it for standard reference result files.
208 Use the various skipping or controls to try to produce reproducible results.
213 Engines are selected with the option `--engine`.
216 ./tests.sh --engine nitce base_class_name.nit
221 * `nitcs`, for `nitc --separate` (this is the default)
222 * `nitce`, for `nitc --erasure`
223 * `nitcsg`, for `nitc --separate --semi-global`
224 * `nitcg`, for `nitc --global`
225 * `niti`, for `nit`, the interpreter
226 * `nitvm`, for `nit --vm`, the virtual machine
227 * `emscripten`, for `nitc --semi-global -m emscripten` (not automatically executed by `testall.sh`)
231 * How tests are run: compiled or interpreted? which binary? what options?
232 * Where to find the reference result files.
233 The sav/ directory contains subdirectories per engine that are used to store specific reference result files.
234 Specific reference result files override the generic ones.
239 See `README_alternatives.md`
242 ## Controlling the execution of program tests
247 Argument files `*.args` are used to run program tests with specific command line arguments.
249 * `foo.args` are used for the test file basenamed `foo`.
250 * `foo_*alt*.args` are used for specific alternatives if they exists.
252 Each line of an argument file produces an additional distinct test, with its own testname and reference result files.
255 The first test, with a short arrow `=>`, is the test executed without any arguments.
256 Other tests are arguments tests and have a longer arrow `==>`.
257 If the first test does not produce an executapel, arguments tests are not run.
262 $ tail zzz_test_args.args zzz_tests/zzz_test_args.nit
263 $ ./tests.sh zzz_tests/zzz_test_args.nit
264 $ tail out/zzz_test_args*.res
270 By default, stdin is read from `/dev/null.`
271 Specific tests may need prepared inputs.
273 The `*.inputs` files are used to provide inputs during the execution of program tests.
275 * `foo.inputs` are used for the test file basenamed `foo`
276 * `foo_*alt*_args*.args` are used for specific alternatives and/or arguments if they exists.
281 $ tail zzz_test_in.inputs zzz_tests/zzz_test_in.nit
282 $ ./tests.sh zzz_tests/zzz_test_in.nit
283 $ cat out/zzz_test_in.res
287 ### Output and generated files
289 The stdout and stderr are redirected to the produced result file.
291 Sometimes, files are also produced by the programs.
292 In order to check those files, a special path by testname is watched.
293 Its name is `out/foo.write` where `foo` is the complete testname (with alts and args).
295 The shell variable `$WRITE` can be used in `*.args` file in order to give the correct path to the program.
297 If it exists, the content of the `$WRITE` file is appended at the end of the produced result file.
298 If `$WRITE` is created as a directory, then the names of files in this directory is appended.
303 $ tail zzz_test_write.args zzz_tests/zzz_test_write.nit
304 $ ./tests.sh zzz_tests/zzz_test_write.nit
305 $ cat out/zzz_test_write.out
311 Some simple post-processing can be executed after each tests associated to a line in a `.args` file.
313 In `*.args` files, semicolons can be used to introduce and separate additional shell commands.
316 $ tail zzz_test_post_proc.args zzz_tests/zzz_test_post_proc.nit
317 $ ./tests.sh zzz_tests/zzz_test_post_proc.nit
318 $ cat out/zzz_test_post_proc.res
322 ### Environment variable
324 The environment variable `NIT_TESTING` is set to `true` during the execution of program tests.
325 Some libraries of programs can use it to produce specific reproducible results ; or just to exit their executions.
327 `NIT_TESTING_TESTS_SH` is set to `true` for tests ran by `tests.sh` only, not by nitunit.
329 `NIT_TESTING_ID` is set to an integer value unique between concurrent executions of `tests.sh`.
330 Use it to name files, database entries, open sockets in order to avoid race conditions.
333 $ cat zzz_tests/zzz_test_envvar.nit
334 $ ./tests.sh zzz_tests/zzz_test_envvar.nit
335 $ cat out/zzz_test_post_proc.res