X-Git-Url: http://nitlanguage.org diff --git a/share/man/nitunit.md b/share/man/nitunit.md index e01dbd2..728ecdb 100644 --- a/share/man/nitunit.md +++ b/share/man/nitunit.md @@ -118,22 +118,37 @@ Finally, standard markdown documents can be checked with: $ nitunit foo.md +When testing, the environment variable `NIT_TESTING` is set to `true`. +This flag can be used by libraries and program to prevent (or limit) the execution of dangerous pieces of code. + +~~~~~ +# NIT_TESTING is automatically set. +# +# assert "NIT_TESTING".environ == "true" +~~~~ + ## Working with `TestSuites` -TestSuites are Nit files that define a set of TestCases for a particular module. +TestSuites are Nit modules that define a set of TestCases. -The test suite must be called `test_` followed by the name of the module to test. -So for the module `foo.nit` the test suite will be called `test_foo.nit`. +A test suite is a module that uses the annotation `is test`. + +It is common that a test suite focuses on testing a single module. +In this case, the name of the test suite is often `test_foo.nit` where `foo.nit` is the tested module. The structure of a test suite is the following: ~~~~ # test suite for module `foo` -module test_foo +module test_foo is test + import foo # can be intrude to test private things + class TestFoo + test + # test case for `foo::Foo::baz` - fun test_baz do + fun baz is test do var subject = new Foo assert subject.baz(1, 2) == 3 end @@ -144,66 +159,199 @@ Test suite can be executed using the same `nitunit` command: $ nitunit foo.nit -`nitunit` will execute a test for each method named `test_*` in a class named `Test*` -so multiple tests can be executed for a single method: +`nitunit` will execute a test for each method with the `test` annotation in a class +also annotated with `test` so multiple tests can be executed for a single method: ~~~~ class TestFoo - fun test_baz_1 do + test + + fun baz_1 is test do var subject = new Foo assert subject.baz(1, 2) == 3 end - fun test_baz_2 do + fun baz_2 is test do var subject = new Foo assert subject.baz(1, -2) == -1 end end ~~~~ -`TestSuites` also provide methods to configure the test run: +## Black Box Testing + +Sometimes, it is easier to validate a `TestCase` by comparing its output with a text file containing the expected result. + +For each TestCase `test_bar` of a TestSuite `test_mod.nit`, a corresponding file with the expected output is looked for: + +* "test_mod.sav/test_bar.res". I.e. test-cases grouped by test-suites. + + This is the default and is useful if there is a lot of test-suites and test-cases in a directory + +* "sav/test_bar.res". I.e. all test-cases grouped in a common sub-directory. + + Useful if there is a lot of test-suites OR test-cases in a directory. + +* "test_bar.res" raw in the directory. + + Useful is there is a few test-suites and test-cases in a directory. + +All 3 are exclusive. If more than one exists, the test-case is failed. + +If a corresponding file then the output of the test-case is compared with the file. + +The `diff(1)` command is used to perform the comparison. +The test is failed if non-zero is returned by `diff`. + +~~~ +module test_mod is test + +class TestFoo + test + + fun bar is test do + print "Hello!" + end +end +~~~ + +Where `test_mod.sav/test_bar.res` contains -`before_test` and `after_test`: methods called before/after each test case. +~~~raw +Hello! +~~~ + +If no corresponding `.res` file exists, then the output of the TestCase is ignored. + +To helps the management of the expected results, the option `--autosav` can be used to automatically create and update them. + + +## Configuring TestSuites + +`TestSuite`s also provide annotations to configure the test run: +`before` and `after` annotations can be added to methods that must be called before/after each test case. They can be used to factorize repetitive tasks: -~~~~ +~~~ class TestFoo - var subject: Foo - # Mandatory empty init - init do end - # Method executed before each test - fun before_test do - subject = new Foo - end - fun test_baz_1 do - assert subject.baz(1, 2) == 3 - end - fun test_baz_2 do - assert subject.baz(1, -2) == -1 - end + test + + var subject: Foo is noinit + + # Method executed before each test + fun set_up is before do + subject = new Foo + end + + fun baz_1 is test do + assert subject.baz(1, 2) == 3 + end + + fun baz_2 is test do + assert subject.baz(1, -2) == -1 + end end -~~~~ +~~~ -When using custom test attributes, an empty `init` must be declared to allow automatic test running. +When using custom test attributes, a empty init must be declared to allow automatic test running. -`before_module` and `after_module`: methods called before/after each test suite. -They have to be declared at top level: +At class level, `before_all` and `after_all` annotations can be set on methods that must be called before/after all the test cases in the class: -~~~~ +~~~ +class TestFoo + test + + var subject: Foo is noinit + + # Method executed before all tests in the class + fun set_up is before_all do + subject = new Foo + end + + fun baz_1 is test do + assert subject.baz(1, 2) == 3 + end + + fun baz_2 is test do + assert subject.baz(1, -2) == -1 + end +end +~~~ + +`before_all` and `after_all` annotations can also be set on methods that must be called before/after each test suite when declared at top level: + +~~~ module test_bdd_connector + import bdd_connector + # Testing the bdd_connector class TestConnector - # test cases using a server + test + # test cases using a server end + # Method executed before testing the module -fun before_module do - # start server before all test cases +fun setup_db is before_all do + # start server before all test cases end + # Method executed after testing the module -fun after_module do - # stop server after all test cases +fun teardown_db is after_all do + # stop server after all test cases end -~~~~ +~~~ + +When dealing with multiple test suites, niunit allows you to import other test suites to factorize your tests: + +~~~ +module test_bdd_users + +import test_bdd_connector + +# Testing the user table +class TestUsersTable + test + # test cases using the db server from `test_bdd_connector` +end + +fun setup_table is before_all do + # create user table +end + +fun teardown_table is after_all do + # drop user table +end +~~~ + +Methods with `before*` and `after*` annotations are linearized and called in different ways. + +* `before*` methods are called from the least specific to the most specific +* `after*` methods are called from the most specific to the least specific + +In the previous example, the execution order would be: + +1. `test_bdd_connector::setup_db` +2. `test_bdd_users::setup_table` +3. `all test cases from test_bdd_users` +4. `test_bdd_users::teardown_table` +5. `test_bdd_connector::teardown_db` + +## Accessing the test suite environment + +The `NIT_TESTING_PATH` environment variable contains the current test suite +file path. +Nitunit define this variable before the execution of each test suite. +It can be used to access files based on the current test suite location: + +~~~ +class TestWithPath + test + + fun test_suite_path do + assert "NIT_TESTING_PATH".environ != "" + end +end +~~~ ## Generating test suites @@ -228,13 +376,22 @@ With the `--full` option, all imported modules (even those in standard) are also ### `-o`, `--output` Output name (default is 'nitunit.xml'). -### `nitunit` produces a XML file comatible with JUnit. +`nitunit` produces a XML file compatible with JUnit. ### `--dir` -Working directory (default is '.nitunit'). +Working directory (default is 'nitunit.out'). In order to execute the tests, nit files are generated then compiled and executed in the giver working directory. +In case of success, the directory is removed. +In case of failure, it is kept as is so files can be investigated. + +### `--nitc` +nitc compiler to use. + +By default, nitunit tries to locate the `nitc` program with the environment variable `NITC` or heuristics. +The option is used to indicate a specific nitc binary. + ### `--no-act` Does not compile and run tests. @@ -243,8 +400,22 @@ Only run test case with name that match pattern. Examples: `TestFoo`, `TestFoo*`, `TestFoo::test_foo`, `TestFoo::test_foo*`, `test_foo`, `test_foo*` -### `-t`, `--target-file` -Specify test suite location. +### `--autosav` +Automatically create/update .res files for black box testing. + +If a black block test fails because a difference between the expected result and the current result then the expected result file is updated (and the test is passed). + +If a test-case of a test-suite passes but that some output is generated, then an expected result file is created. + +It is expected that the created/updated files are checked since the tests are considered passed. +A VCS like `git` is often a good tool to check the creation and modification of those files. + +### `--no-time` +Disable time information in XML. + +This is used to have reproducible XML results. + +This option is automatically activated if `NIT_TESTING` is set. ## SUITE GENERATION @@ -262,6 +433,40 @@ Also generate test case for private methods. ### `--only-show` Only display the skeleton, do not write any file. + +# ENVIRONMENT VARIABLES + +### `NITC` + +Indicate the specific Nit compiler executable to use. See `--nitc`. + +### `NIT_TESTING` + +The environment variable `NIT_TESTING` is set to `true` during the execution of program tests. +Some libraries of programs can use it to produce specific reproducible results; or just to exit their executions. + +Unit-tests may unset this environment variable to retrieve the original behavior of such piece of software. + +### `SRAND` + +In order to maximize reproducibility, `SRAND` is set to 0. +This make the pseudo-random generator no random at all. +See `Sys::srand` for details. + +To retrieve the randomness, unit-tests may unset this environment variable then call `srand`. + +### `NIT_TESTING_ID` + +Parallel executions can cause some race collisions on named resources (e.g. DB table names). +To solve this issue, `NIT_TESTING_ID` is initialized with a distinct integer identifier that can be used to give unique names to resources. + +Note: `rand` is not a recommended way to get a distinct identifier because its randomness is disabled by default. See `SRAND`. + +### `NIT_TESTING_PATH` + +Only available for test suites. +Contains the module test suite path. + # SEE ALSO The Nit language documentation and the source code of its tools and libraries may be downloaded from