X-Git-Url: http://nitlanguage.org diff --git a/share/man/nitunit.md b/share/man/nitunit.md index b43004f..728ecdb 100644 --- a/share/man/nitunit.md +++ b/share/man/nitunit.md @@ -1,5 +1,3 @@ -% NITUNIT(1) - # NAME nitunit - executes the unit tests from Nit source files. @@ -32,24 +30,28 @@ The execution can be verified using `assert`. Example with a class: - module foo - # var foo = new Foo - # assert foo.bar == 10 - class Foo - var bar = 10 - end +~~~ +module foo +# var foo = new Foo +# assert foo.bar == 10 +class Foo + var bar = 10 +end +~~~ Everything used in the test must be declared. To test a method you have to instantiate its class: - module foo +~~~ +module foo +# var foo = new Foo +# assert foo.bar == 10 +class Foo # var foo = new Foo - # assert foo.bar == 10 - class Foo - # var foo = new Foo - # assert foo.baz(1, 2) == 3 - fun baz(a, b: Int) do return a + b - end + # assert foo.baz(1, 2) == 3 + fun baz(a, b: Int) do return a + b +end +~~~ In a single piece of documentation, each docunit is considered a part of a single module, thus regrouped when tested. @@ -116,139 +118,354 @@ Finally, standard markdown documents can be checked with: $ nitunit foo.md +When testing, the environment variable `NIT_TESTING` is set to `true`. +This flag can be used by libraries and program to prevent (or limit) the execution of dangerous pieces of code. + +~~~~~ +# NIT_TESTING is automatically set. +# +# assert "NIT_TESTING".environ == "true" +~~~~ + ## Working with `TestSuites` -TestSuites are Nit files that define a set of TestCases for a particular module. +TestSuites are Nit modules that define a set of TestCases. + +A test suite is a module that uses the annotation `is test`. -The test suite must be called `test_` followed by the name of the module to test. -So for the module `foo.nit` the test suite will be called `test_foo.nit`. +It is common that a test suite focuses on testing a single module. +In this case, the name of the test suite is often `test_foo.nit` where `foo.nit` is the tested module. The structure of a test suite is the following: - # test suite for module `foo` - module test_foo - import foo # can be intrude to test private things - class TestFoo - # test case for `foo::Foo::baz` - fun test_baz do - var subject = new Foo - assert subject.baz(1, 2) == 3 - end +~~~~ +# test suite for module `foo` +module test_foo is test + +import foo # can be intrude to test private things + +class TestFoo + test + + # test case for `foo::Foo::baz` + fun baz is test do + var subject = new Foo + assert subject.baz(1, 2) == 3 end +end +~~~~ Test suite can be executed using the same `nitunit` command: $ nitunit foo.nit -`nitunit` will execute a test for each method named `test_*` in a class named `Test*` -so multiple tests can be executed for a single method: - - class TestFoo - fun test_baz_1 do - var subject = new Foo - assert subject.baz(1, 2) == 3 - end - fun test_baz_2 do - var subject = new Foo - assert subject.baz(1, -2) == -1 - end +`nitunit` will execute a test for each method with the `test` annotation in a class +also annotated with `test` so multiple tests can be executed for a single method: + +~~~~ +class TestFoo + test + + fun baz_1 is test do + var subject = new Foo + assert subject.baz(1, 2) == 3 + end + fun baz_2 is test do + var subject = new Foo + assert subject.baz(1, -2) == -1 end +end +~~~~ + +## Black Box Testing + +Sometimes, it is easier to validate a `TestCase` by comparing its output with a text file containing the expected result. + +For each TestCase `test_bar` of a TestSuite `test_mod.nit`, a corresponding file with the expected output is looked for: + +* "test_mod.sav/test_bar.res". I.e. test-cases grouped by test-suites. + + This is the default and is useful if there is a lot of test-suites and test-cases in a directory -`TestSuites` also provide methods to configure the test run: +* "sav/test_bar.res". I.e. all test-cases grouped in a common sub-directory. -`before_test` and `after_test`: methods called before/after each test case. + Useful if there is a lot of test-suites OR test-cases in a directory. + +* "test_bar.res" raw in the directory. + + Useful is there is a few test-suites and test-cases in a directory. + +All 3 are exclusive. If more than one exists, the test-case is failed. + +If a corresponding file then the output of the test-case is compared with the file. + +The `diff(1)` command is used to perform the comparison. +The test is failed if non-zero is returned by `diff`. + +~~~ +module test_mod is test + +class TestFoo + test + + fun bar is test do + print "Hello!" + end +end +~~~ + +Where `test_mod.sav/test_bar.res` contains + +~~~raw +Hello! +~~~ + +If no corresponding `.res` file exists, then the output of the TestCase is ignored. + +To helps the management of the expected results, the option `--autosav` can be used to automatically create and update them. + + +## Configuring TestSuites + +`TestSuite`s also provide annotations to configure the test run: +`before` and `after` annotations can be added to methods that must be called before/after each test case. They can be used to factorize repetitive tasks: - class TestFoo - var subject: Foo - # Mandatory empty init - init do end - # Method executed before each test - fun before_test do - subject = new Foo - end - fun test_baz_1 do - assert subject.baz(1, 2) == 3 - end - fun test_baz_2 do - assert subject.baz(1, -2) == -1 - end - end +~~~ +class TestFoo + test -When using custom test attributes, an empty `init` must be declared to allow automatic test running. + var subject: Foo is noinit -`before_module` and `after_module`: methods called before/after each test suite. -They have to be declared at top level: + # Method executed before each test + fun set_up is before do + subject = new Foo + end - module test_bdd_connector - import bdd_connector - # Testing the bdd_connector - class TestConnector - # test cases using a server - end - # Method executed before testing the module - fun before_module do - # start server before all test cases - end - # Method executed after testing the module - fun after_module do - # stop server after all test cases + fun baz_1 is test do + assert subject.baz(1, 2) == 3 + end + + fun baz_2 is test do + assert subject.baz(1, -2) == -1 + end +end +~~~ + +When using custom test attributes, a empty init must be declared to allow automatic test running. + +At class level, `before_all` and `after_all` annotations can be set on methods that must be called before/after all the test cases in the class: + +~~~ +class TestFoo + test + + var subject: Foo is noinit + + # Method executed before all tests in the class + fun set_up is before_all do + subject = new Foo + end + + fun baz_1 is test do + assert subject.baz(1, 2) == 3 + end + + fun baz_2 is test do + assert subject.baz(1, -2) == -1 + end +end +~~~ + +`before_all` and `after_all` annotations can also be set on methods that must be called before/after each test suite when declared at top level: + +~~~ +module test_bdd_connector + +import bdd_connector + +# Testing the bdd_connector +class TestConnector + test + # test cases using a server +end + +# Method executed before testing the module +fun setup_db is before_all do + # start server before all test cases +end + +# Method executed after testing the module +fun teardown_db is after_all do + # stop server after all test cases +end +~~~ + +When dealing with multiple test suites, niunit allows you to import other test suites to factorize your tests: + +~~~ +module test_bdd_users + +import test_bdd_connector + +# Testing the user table +class TestUsersTable + test + # test cases using the db server from `test_bdd_connector` +end + +fun setup_table is before_all do + # create user table +end + +fun teardown_table is after_all do + # drop user table +end +~~~ + +Methods with `before*` and `after*` annotations are linearized and called in different ways. + +* `before*` methods are called from the least specific to the most specific +* `after*` methods are called from the most specific to the least specific + +In the previous example, the execution order would be: + +1. `test_bdd_connector::setup_db` +2. `test_bdd_users::setup_table` +3. `all test cases from test_bdd_users` +4. `test_bdd_users::teardown_table` +5. `test_bdd_connector::teardown_db` + +## Accessing the test suite environment + +The `NIT_TESTING_PATH` environment variable contains the current test suite +file path. +Nitunit define this variable before the execution of each test suite. +It can be used to access files based on the current test suite location: + +~~~ +class TestWithPath + test + + fun test_suite_path do + assert "NIT_TESTING_PATH".environ != "" end +end +~~~ ## Generating test suites - Write test suites for big modules can be a repetitive and boring task... - To make it easier, `nitunit` can generate test skeletons for Nit modules: +Write test suites for big modules can be a repetitive and boring task... +To make it easier, `nitunit` can generate test skeletons for Nit modules: $ nitunit --gen-suite foo.nit - This will generate the test suite `test_foo` containing test case stubs for all public - methods found in `foo.nit`. +This will generate the test suite `test_foo` containing test case stubs for all public +methods found in `foo.nit`. # OPTIONS -`--full` -: Process also imported modules. +### `--full` +Process also imported modules. + +By default, only the modules indicated on the command line are tested. + +With the `--full` option, all imported modules (even those in standard) are also precessed. - By default, only the modules indicated on the command line are tested. +### `-o`, `--output` +Output name (default is 'nitunit.xml'). - With the `--full` option, all imported modules (even those in standard) are also precessed. +`nitunit` produces a XML file compatible with JUnit. -`-o`, `--output` -: Output name (default is 'nitunit.xml') +### `--dir` +Working directory (default is 'nitunit.out'). - `nitunit` produces a XML file comatible with JUnit. +In order to execute the tests, nit files are generated then compiled and executed in the giver working directory. -`--dir` -: Working directory (default is '.nitunit') +In case of success, the directory is removed. +In case of failure, it is kept as is so files can be investigated. - In order to execute the tests, nit files are generated then compiled and executed in the giver working directory. +### `--nitc` +nitc compiler to use. -`--no-act` -: Does not compile and run tests. +By default, nitunit tries to locate the `nitc` program with the environment variable `NITC` or heuristics. +The option is used to indicate a specific nitc binary. -`-p`, `--pattern` -: Only run test case with name that match pattern. Examples: `TestFoo`, `TestFoo*`, `TestFoo::test_foo`, `TestFoo::test_foo*`, `test_foo`, `test_foo*` +### `--no-act` +Does not compile and run tests. -`-t`, `--target-file` -: Specify test suite location. +### `-p`, `--pattern` +Only run test case with name that match pattern. + +Examples: `TestFoo`, `TestFoo*`, `TestFoo::test_foo`, `TestFoo::test_foo*`, `test_foo`, `test_foo*` + +### `--autosav` +Automatically create/update .res files for black box testing. + +If a black block test fails because a difference between the expected result and the current result then the expected result file is updated (and the test is passed). + +If a test-case of a test-suite passes but that some output is generated, then an expected result file is created. + +It is expected that the created/updated files are checked since the tests are considered passed. +A VCS like `git` is often a good tool to check the creation and modification of those files. + +### `--no-time` +Disable time information in XML. + +This is used to have reproducible XML results. + +This option is automatically activated if `NIT_TESTING` is set. ## SUITE GENERATION -`--gen-suite` -: Generate test suite skeleton for a module +### `--gen-suite` +Generate test suite skeleton for a module. + +### `-f`, `--force` +Force test generation even if file exists. + +Any existing test suite will be overwritten. + +### `--private` +Also generate test case for private methods. + +### `--only-show` +Only display the skeleton, do not write any file. + + +# ENVIRONMENT VARIABLES + +### `NITC` + +Indicate the specific Nit compiler executable to use. See `--nitc`. + +### `NIT_TESTING` + +The environment variable `NIT_TESTING` is set to `true` during the execution of program tests. +Some libraries of programs can use it to produce specific reproducible results; or just to exit their executions. + +Unit-tests may unset this environment variable to retrieve the original behavior of such piece of software. + +### `SRAND` + +In order to maximize reproducibility, `SRAND` is set to 0. +This make the pseudo-random generator no random at all. +See `Sys::srand` for details. + +To retrieve the randomness, unit-tests may unset this environment variable then call `srand`. + +### `NIT_TESTING_ID` -`-f`, `--force` -: Force test generation even if file exists. +Parallel executions can cause some race collisions on named resources (e.g. DB table names). +To solve this issue, `NIT_TESTING_ID` is initialized with a distinct integer identifier that can be used to give unique names to resources. - Any existing test suite will be overwritten. +Note: `rand` is not a recommended way to get a distinct identifier because its randomness is disabled by default. See `SRAND`. -`--private` -: Also generate test case for private methods. +### `NIT_TESTING_PATH` -`--only-show` -: Only display the skeleton, do not write any file. +Only available for test suites. +Contains the module test suite path. # SEE ALSO