--- /dev/null
+# Regression test for the Nit project
+
+## Content of the directory
+
+This directory contains small Nit programs and useful scripts to test the non regression of the nit tools.
+
+### Useful scripts
+
+* `tests.sh` is the main test script used to run some specific tests.
+* `testall.sh` runs some tests on all engines.
+ (No more excuse to forget to check your new tests with an obscure engine).
+* `testfull.sh` runs all tests on a single engine.
+ (go and grab a coffee)
+* `search_test.sh` lists tests given a testname, a basename, or a `.res` file.
+ Useful when `foo failed` but you do not know where the program `foo` is.
+* `search_tests_git.sh` lists tests that seem modified between 2 git revisions.
+ Useful before committing something.
+* `listfull.sh` lists tests that are run by `testfull.sh`.
+ It is used by other scripts but can be used by human to see what is tested.
+
+ Currently, all files in `tests/`, `lib/` and `examples/` are executed, some of `contrib/` and the main programs of `src/`.
+ To update the list of tested files, just edit this script.
+
+### Small Nit programs
+
+They are divided in categories:
+
+* `base` are minimal and fundamental tests.
+ They are small and test the fundamental language specifications.
+* `bench` are memory and time tests
+* `error` are invalid programs
+ They test the error message and the good behavior of the tool.
+ Nowadays, alternatives (see below) are more used to test erroneous programs.
+* `example` are examples
+ Nowadays, examples are developed in the `examples/` root directory.
+* `test` are others tests that usually depend on standard or other libraries.
+ Nowadays, nitunit are a better place for them.
+* `shootout` are benchmarks from http://shootout.alioth.debian.org/
+
+### Other stuff
+
+The .gitignore contains specific artefacts produced by some tests.
+However, it is better to produce these artifacts in the `out` directory instead,
+because `out` is cleaned before running tests (so that old artefacts do not
+interfere with new executions of tests)
+
+The `sav/` directory contains reference result files for tests (see below)
+
+The `alt/` and `out/` directory are transient directories used during the tests.
+
+The `zzz_tests/` directory contains tests to test the test system.
+Execute `./tests.ss zzz_tests/*.nit` if you want.
+
+## Running tests
+
+~~~
+./tests.sh filepath.nit...
+./tests.sh base_attr*.nit
+~~~
+
+Will execute the program filepath for all its alternative (see below) and all its arguments (see below) for the default engine (see below).
+
+The output is something like
+~~~
+=> test_args: . [ok] out/test_args.res sav/test_args.res
+~~~
+
+where in order, there is
+
+* The test description (`test_args` in the example).
+ It is the basename of the file + information about alternatives and arguments.
+* The build status: `.`, `!`, or `nocc`, or `_`.
+ They respectively mean "compilation OK", "compilation error", "compilation skipped", "compilation OK but execution skipped".
+* The test status (see below) where `out/*.res` is the produced result file, and `sav/*.res` is the reference result file.
+
+
+## Test statuses
+
+### Ok
+
+`[ok] out/zzz_test_ok.res sav/zzz_test_ok.res`
+
+The produced result file correspond to the reference result file (according to `diff`).
+It is a success.
+
+### 0k, is empty
+
+`[0k] out/zzz_test_0k.res is empty`
+
+The produced result file is empty and no reference result file is present.
+It is a success.
+
+If you introduced a new library, it is likely that you get this.
+It basically means that the module compile and does noting, and its fine.
+
+### Fail
+
+`[======= fail out/zzz_test_fail.res sav/zzz_test_fail.res =======]`
+
+The produced result file do not correspond to the reference result file.
+It is a failure.
+
+Did something break? or does the reference result file need to be updated?
+
+### No sav
+
+`[=== no sav ===] out/zzz_test_nosav.res is not empty`
+
+The produced result file is not empty but no reference result file is present.
+It is a failure.
+
+Did something break? or does the reference result file need to be created?
+If you introduced a new program, it is likely that you get this.
+It basically means the program compiles and produces something.
+Please create a new reference result file for it.
+
+### Soso
+
+`[======= soso out/zzz_test_soso.res sav//zzz_test_soso.res =======]`
+
+The produced result file do not correspond to the reference result file.
+But the difference in only in errors or warnings.
+It is a failure.
+
+Usually, some lines in error messages changed.
+So just update the reference result file.
+
+### Fixme
+
+`[fixme] out/zzz_test_fixme.res sav//fixme/zzz_test_fixme.res`
+
+The produced result file correspond to a *fixme* reference result file.
+It is an expected error. So not a regression.
+
+Reference result files in `sav/**/fixme/` are considered expected errors.
+The distinction with standard reference result files is only for final statistics.
+
+### Ok, but fixme remains
+
+`[*ok*] out/zzz_test_fixme_remains.res sav//zzz_test_fixme_remains.res - but sav//fixme/zzz_test_fixme_remains.res remains!`
+
+There is a fixme reference result file and a standard reference result file. But the produced result file correspond to the standard one.
+Usually it means that a bug was fixed. But is considered a regression until the *fixme* reference result file is removed.
+It is a failure.
+
+### Changed fixme
+
+`[======= changed out/zzz_test_fixme_changed.res sav//fixme/zzz_test_fixme_changed.res ======]`
+
+The produced result file do not correspond to the *fixme* reference result file.
+It is a failure, and analogous to the standard `fail`.
+
+### Todo, not yet implemented
+
+`[todo] out/zzz_test_todo.res -> not yet implemented`
+
+The produced result file contains the magic string "NOT YET IMPLEMENTED".
+Those are considered the same as expected errors (like a fixme)
+It is a success.
+
+Some engines, libraries or program just print this to simplify the management of tests.
+
+### Skipped
+
+`[skip]`
+
+The test is skipped.
+It is a success.
+
+The `$engine.skip` files (where `$engine` is an engine name, see below) describe tests that are skipped completely on a given engine.
+Usually it used with then engine `niti` because tests are too long.
+
+The `cc.skip` file describes tests that are analysed but no executable is generated.
+Usually it is because expected CC errors or missing C libraries.
+
+The `exec.skip` file describes tests that compiled but not executed.
+Usually it is because the programs are interactive or run some kind of server.
+
+These `*.skip` files contain a list of patterns that will be used against test names.
+A single substring can thus be used to skip a full family of tests.
+
+## Update reference result files
+
+To update the reference result files, just create/remove/update files in the `sav/` directory.
+
+HINT: for easy management, just copy-paste parts of the test status to build your command.
+Example `cp <copy-paste out/zzz_test_fail.res sav/zzz_test_fail.res>` to update a file.
+
+
+If a reference result file contains only `UNDEFINED`, then the produced result file is always considered successful.
+It is used to store system-dependant *fixme* reference result files.
+Once the problem is solved, the status will become `Ok but fixme remains`
+
+
+Note: `UNDEFINED` still gives a success even if the tests is uncompilable.
+So do not use it for standard reference result files.
+Use the various skipping or controls to try to produce reproducible results.
+
+
+## Engines
+
+Engines are selected with the option `--engine`.
+
+~~~
+./tests.sh --engine nitg-e base_class_name.nit
+~~~
+
+Current engines are:
+
+* `nitg-s`, for `nitg --separate` (this is the default)
+* `nitg-e`, for `nitg --erasure`
+* `nitg-sg`, for `nitg --separate --semi-global`
+* `nitg-g`, for `nitg --global`
+* `niti`, for `nit`, the interpreter
+* `nitvm`, for `nitvm` (not automatically executed by `testall.sh`)
+* `emscripten`, for `nitg --semi-global -m emscripten` (not automatically executed by `testall.sh`)
+
+Engines control:
+
+* How tests are run: compiled or interpreted? which binary? what options?
+* Where to find the reference result files.
+ The sav/ directory contains subdirectories per engine that are used to store specific reference result files.
+ Specific reference result files override the generic ones.
+
+
+## Alternatives
+
+See `README_alternatives.md`
+
+
+## Controlling the execution of program tests
+
+
+### Arguments
+
+Argument files `*.args` are used to run program tests with specific command line arguments.
+
+* `foo.args` are used for the test file basenamed `foo`.
+* `foo_*alt*.args` are used for specific alternatives if they exists.
+
+Each line of an argument file produces an additional distinct test, with its own testname and reference result files.
+
+
+The first test, with a short arrow `=>`, is the test executed without any arguments.
+Other tests are arguments tests and have a longer arrow `==>`.
+If the first test does not produce an executapel, arguments tests are not run.
+
+Example:
+
+~~~
+$ tail zzz_test_args.args zzz_tests/zzz_test_args.nit
+$ ./tests.sh zzz_tests/zzz_test_args.nit
+$ tail out/zzz_test_args*.res
+~~~
+
+
+### Input
+
+By default, stdin is read from `/dev/null.`
+Specific tests may need prepared inputs.
+
+The `*.inputs` files are used to provide inputs during the execution of program tests.
+
+* `foo.inputs` are used for the test file basenamed `foo`
+* `foo_*alt*_args*.args` are used for specific alternatives and/or arguments if they exists.
+
+Example:
+
+~~~
+$ tail zzz_test_in.inputs zzz_tests/zzz_test_in.nit
+$ ./tests.sh zzz_tests/zzz_test_in.nit
+$ cat out/zzz_test_in.res
+~~~
+
+
+### Output and generated files
+
+The stdout and stderr are redirected to the produced result file.
+
+Sometimes, files are also produced by the programs.
+In order to check those files, a special path by testname is watched.
+Its name is `out/foo.write` where `foo` is the complete testname (with alts and args).
+
+The shell variable `$WRITE` can be used in `*.args` file in order to give the correct path to the program.
+
+If it exists, the content of the `$WRITE` file is appended at the end of the produced result file.
+If `$WRITE` is created as a directory, then the names of files in this directory is appended.
+
+Example:
+
+~~~
+$ tail zzz_test_write.args zzz_tests/zzz_test_write.nit
+$ ./tests.sh zzz_tests/zzz_test_write.nit
+$ cat out/zzz_test_write.out
+~~~
+
+
+### Post-processing
+
+Some simple post-processing can be executed after each tests associated to a line in a `.args` file.
+
+In `*.args` files, semicolons can be used to introduce and separate additional shell commands.
+
+~~~
+$ tail zzz_test_post_proc.args zzz_tests/zzz_test_post_proc.nit
+$ ./tests.sh zzz_tests/zzz_test_post_proc.nit
+$ cat out/zzz_test_post_proc.res
+~~~
+
+
+### Environment variable
+
+The environment variable `NIT_TESTING` is set to `true` during the execution of program tests.
+Some libraries of programs can use it to produce specific reproducible results ; or just to exit their executions.
+
+~~~
+$ cat zzz_tests/zzz_test_envvar.nit
+$ ./tests.sh zzz_tests/zzz_test_envvar.nit
+$ cat out/zzz_test_post_proc.res
+~~~
--- /dev/null
+# Produces alternatives of text files
+
+alterner.pl is a simple tool that use alternation directives in a text file to produce variations (alternatives) of this text file.
+
+alterner.pl was primarily developed to help the generation of regression tests for [SableCC][1] but now it aims to be a more generic and versatile tool.
+
+ [1]: http://sablecc.org
+
+Principles of alterner.pl
+-------------------------
+
+alterner works line by line and produces alternatives files where some lines are altered.
+
+An line containing an alternation directive looks like
+
+ standard text //alt1 alternative text
+
+where `//alt1` is the alternation directive (note there is no space between the `//` and the `alt` nor between the `alt` and the number).
+
+Note: an alternation directive can also be terminated by a `//`.
+Eg.
+
+ standard text //alt1// alternative text
+
+The first principle is that the alternation is produced by swapping the right part of the directive with its left part.
+
+The basic idea behind alterner.pl is that if `//` marks the start of a comment for your input language then:
+
+* if the file is used unmodified, the useful part of the line will be `standard text`;
+* if the alternative version is used, the useful part of the line will be `alternative text`.
+
+Neither the left or the right part is mandatory.
+Therefore, directives are often used to make some lines appear or disappear in the alternatives:
+
+ this will disappear in the alternative //alt1
+ //alt1 this will appear in the alternative
+
+
+The second principle is that there could be many alternatives generated for a single base file.
+Each alternation directive is numbered (`//alt1`, `//alt2`, `//alt3`, etc.) and, for a given alternative, all the associated directives are applied together.
+
+Example:
+
+ //alt1 this will appear in alternative 1
+ this will disappear in alternative 2 //alt2
+ this is in base and in alternative 2 //alt1 but this is in alternative 1
+
+The last principle is that more than one directive can appear on a single line.
+The first principle still apply: for a given alternative version, the left and right parts of the associated directives are swapped.
+The trick is that the directives of other alternatives do not interfere.
+
+Example:
+
+ base version //alt1 version of alternative 1 //alt2 version of alternative 2
+
+
+Lists and ranges
+----------------
+
+A single directive can be associated to more than one alternative version.
+Directives for multiple alternatives are declared with a list (separated with a comma) or with a range (indicated with a dash).
+
+Example:
+
+ base version //alt1,3 version of alternative 1 and 3
+ base version //alt2-4 version of alternatives 2, 3 and 4
+
+Lists and ranges can be used together and multiple alternatives can appears on a single line.
+
+ base //alt1,3-4 version for 1, 3, and 4 //alt2,5 version for 2 and 5
+
+
+Combinations
+------------
+
+Each directive belong to a given group: either to the anonymous group (by default) or to a numbered group.
+The number of the group is noted before the `alt`.
+
+Example:
+
+ base version //1alt1 first alternative of the first group
+ base version //2alt1 first alternative of the second group
+ base version //2alt2 second alternative of the second group
+
+alterner.pl also produces alternative files by combining the groups.
+With two groups, is means that at least three alternative files are created: one with the first group, one with the second group, and one with both.
+
+If there is more than one alternative by group, even more alternative files are created.
+With the previous example, 5 alternative files are created:
+
+* one with 1alt1 activated
+* one with 2alt1 activated
+* one with 2alt2 activated
+* one with 1alt1 and 2alt1 activated
+* one with 1alt1 and 2alt2 activated
+
+Be careful with groups since the number of alternative file generated increase rapidly.
+For instance, with 4 groups of 4 alternatives each, 624 alternative files are generated.
+
+Usage of alterner.pl
+--------------------
+
+ alterner.pl [options] input-file.ext
+
+Produce alternatives of `input-file.ext` according to the alternation directives present in the file.
+By default, alternatives are named `input-file.altX.ext` (where `X` is a number) and are generated in a `alt/` sub-directory.
+If groups are used, the `altX` is replaced by `YaltX` (where `Y` is the number of the group) and the combination of multiple groups produce multiple `YaltX` (separated with dots).
+If the directory does not exist, it is created.
+The path of `input-file.ext` is not considered: only the base name of the file and its extension are used to name the alternatives.
+
+alterner.pl also outputs the path of each generated alternative (a path by line).
+This list can be used to know what are the generated alternatives.
+
+
+Options
+-------
+
+* `-d` directory
+
+ Generate the alternatives in the specified directory.
+ By default, the directory is `alt/`.
+ Use `.` for the current directory.
+
+* `--start` pattern
+
+ Alternation directives start with the specified pattern.
+ This option is used to indicate what starts comments in the language.
+ By default, the start pattern is `//`.
+ For example, `--start '#'` means that the alternatives are `#alt1`, `#alt2`, etc.
+
+ Note: an alternation directive can also be terminated by the start pattern.
+ For example, `--start '#'` means that the alternatives can also be `#alt1#`, `#alt2#`, etc.
+
+* `--end` pattern
+
+ Lines with alternation directives end with the specified pattern.
+ This option is used to indicate what end comments in the language when multiline comments are used.
+ By default, the end pattern is the empty string.
+ For example, start with `/*` and end with `*/` for using alternatives in a C file.
+
+ The principle of the end pattern is that:
+
+ 1. alternation directives are ignored if the end pattern is not present at the end of the line;
+ 2. the end pattern is not swapped when a directive is applied (it means the end pattern stay at the end of the line).
+
+* `--altsep` separator
+
+ Separator used in the name of the generated file betwen the basename of the original file and the alternative mark.
+ By default, it is `.`.
+
+Examples
+--------
+
+Example with default options
+
+ $ cat example.txt
+ foo //alt1 bar
+ foo //alt1 bar //alt2 baz
+
+ $ alterner.pl example.txt
+ alt/example.alt1.txt
+ alt/example.alt2.txt
+
+ $ echo alt/example.alt1.txt
+ bar //alt1 foo
+ bar //alt2 baz //alt1 foo
+
+ $ echo alt/example.alt2.txt
+ foo //alt1 bar
+ baz //alt2 foo //alt1 bar
+
+
+Example with directives in C comments.
+
+ $ cat input-file.c
+ base /*alt1 first /*alt2 second */
+
+ $ alterner.pl --start '/*' --end '*/' input-file.c
+ alt/input-file.alt1.c
+ alt/input-file.alt2.c
+
+ $ cat alt/input-file.alt1.c
+ first /*alt2 second /*alt1 base */
+
+ $ cat alt/input-file.alt2.c
+ second /*alt2 base /*alt1 first */
+
+
+Example with groups.
+
+ $ cat example.txt
+ first //alt1 second //alt2 third
+ first //2alt1 second
+
+ $ alterner.pl exaple.txt
+ example.alt1.txt
+ example.alt2.txt
+ example.2alt1.txt
+ example.alt1.2alt1.txt
+ example.alt2.2alt1.txt
nitlanguage / nit.git / commitdiff