Merge: doc: fixed some typos and other misc. corrections

[nit.git] / share / man / nitunit.md

diff --git a/share/man/nitunit.md b/share/man/nitunit.md
index e01dbd2..728ecdb 100644 (file)
--- a/share/man/nitunit.md
+++ b/share/man/nitunit.md
@@ -118,22 +118,37 @@ Finally, standard markdown documents can be checked with:
 
     $ nitunit foo.md
 
 
     $ nitunit foo.md
 

+When testing, the environment variable `NIT_TESTING` is set to `true`.
+This flag can be used by libraries and program to prevent (or limit) the execution of dangerous pieces of code.
+
+~~~~~
+# NIT_TESTING is automatically set.
+#
+#     assert "NIT_TESTING".environ == "true"
+~~~~
+

 ## Working with `TestSuites`
 
 ## Working with `TestSuites`
 

-TestSuites are Nit files that define a set of TestCases for a particular module.
+TestSuites are Nit modules that define a set of TestCases.

 
 

-The test suite must be called `test_` followed by the name of the module to test.
-So for the module `foo.nit` the test suite will be called `test_foo.nit`.
+A test suite is a module that uses the annotation `is test`.
+
+It is common that a test suite focuses on testing a single module.
+In this case, the name of the test suite is often `test_foo.nit` where `foo.nit` is the tested module.

 
 The structure of a test suite is the following:
 
 ~~~~
 # test suite for module `foo`
 
 The structure of a test suite is the following:
 
 ~~~~
 # test suite for module `foo`

-module test_foo
+module test_foo is test
+

 import foo # can be intrude to test private things
 import foo # can be intrude to test private things

+

 class TestFoo
 class TestFoo

+    test
+

     # test case for `foo::Foo::baz`
     # test case for `foo::Foo::baz`

-    fun test_baz do
+    fun baz is test do

         var subject = new Foo
         assert subject.baz(1, 2) == 3
     end
         var subject = new Foo
         assert subject.baz(1, 2) == 3
     end

@@ -144,66 +159,199 @@ Test suite can be executed using the same `nitunit` command:
 
     $ nitunit foo.nit
 
 
     $ nitunit foo.nit
 

-`nitunit` will execute a test for each method named `test_*` in a class named `Test*`
-so multiple tests can be executed for a single method:
+`nitunit` will execute a test for each method with the `test` annotation in a class
+also annotated with `test` so multiple tests can be executed for a single method:

 
 ~~~~
 class TestFoo
 
 ~~~~
 class TestFoo

-    fun test_baz_1 do
+    test
+
+    fun baz_1 is test do

         var subject = new Foo
         assert subject.baz(1, 2) == 3
     end
         var subject = new Foo
         assert subject.baz(1, 2) == 3
     end

-    fun test_baz_2 do
+    fun baz_2 is test do

         var subject = new Foo
         assert subject.baz(1, -2) == -1
     end
 end
 ~~~~
 
         var subject = new Foo
         assert subject.baz(1, -2) == -1
     end
 end
 ~~~~
 

-`TestSuites` also provide methods to configure the test run:
+## Black Box Testing
+
+Sometimes, it is easier to validate a `TestCase` by comparing its output with a text file containing the expected result.
+
+For each TestCase `test_bar` of a TestSuite `test_mod.nit`, a corresponding file with the expected output is looked for:
+
+* "test_mod.sav/test_bar.res". I.e. test-cases grouped by test-suites.
+
+       This is the default and is useful if there is a lot of test-suites and test-cases in a directory
+
+* "sav/test_bar.res". I.e. all test-cases grouped in a common sub-directory.
+
+       Useful if there is a lot of test-suites OR test-cases in a directory.
+
+* "test_bar.res" raw in the directory.
+
+       Useful is there is a few test-suites and test-cases in a directory.
+
+All 3 are exclusive. If more than one exists, the test-case is failed.
+
+If a corresponding file then the output of the test-case is compared with the file.
+
+The `diff(1)` command is used to perform the comparison.
+The test is failed if non-zero is returned by `diff`.
+
+~~~
+module test_mod is test
+
+class TestFoo
+       test
+
+       fun bar is test do
+               print "Hello!"
+       end
+end
+~~~
+
+Where `test_mod.sav/test_bar.res` contains

 
 

-`before_test` and `after_test`: methods called before/after each test case.
+~~~raw
+Hello!
+~~~
+
+If no corresponding `.res` file exists, then the output of the TestCase is ignored.
+
+To helps the management of the expected results, the option `--autosav` can be used to automatically create and update them.
+
+
+## Configuring TestSuites
+
+`TestSuite`s also provide annotations to configure the test run:
+`before` and `after` annotations can be added to methods that must be called before/after each test case.

 They can be used to factorize repetitive tasks:
 
 They can be used to factorize repetitive tasks:
 

-~~~~
+~~~

 class TestFoo
 class TestFoo

-    var subject: Foo
-    # Mandatory empty init
-    init do end
-    # Method executed before each test
-    fun before_test do
-        subject = new Foo
-    end
-    fun test_baz_1 do
-        assert subject.baz(1, 2) == 3
-    end
-    fun test_baz_2 do
-        assert subject.baz(1, -2) == -1
-    end
+       test
+
+       var subject: Foo is noinit
+
+       # Method executed before each test
+       fun set_up is before do
+               subject = new Foo
+       end
+
+       fun baz_1 is test do
+               assert subject.baz(1, 2) == 3
+       end
+
+       fun baz_2 is test do
+               assert subject.baz(1, -2) == -1
+       end

 end
 end

-~~~~
+~~~

 
 

-When using custom test attributes, an empty `init` must be declared to allow automatic test running.
+When using custom test attributes, a empty init must be declared to allow automatic test running.

 
 

-`before_module` and `after_module`: methods called before/after each test suite.
-They have to be declared at top level:
+At class level, `before_all` and `after_all` annotations can be set on methods that must be called before/after all the test cases in the class:

 
 

-~~~~
+~~~
+class TestFoo
+       test
+
+       var subject: Foo is noinit
+
+       # Method executed before all tests in the class
+       fun set_up is before_all do
+               subject = new Foo
+       end
+
+       fun baz_1 is test do
+               assert subject.baz(1, 2) == 3
+       end
+
+       fun baz_2 is test do
+               assert subject.baz(1, -2) == -1
+       end
+end
+~~~
+
+`before_all` and `after_all` annotations can also be set on methods that must be called before/after each test suite when declared at top level:
+
+~~~

 module test_bdd_connector
 module test_bdd_connector

+

 import bdd_connector
 import bdd_connector

+

 # Testing the bdd_connector
 class TestConnector
 # Testing the bdd_connector
 class TestConnector

-    # test cases using a server
+       test
+       # test cases using a server

 end
 end

+

 # Method executed before testing the module
 # Method executed before testing the module

-fun before_module do
-    # start server before all test cases
+fun setup_db is before_all do
+       # start server before all test cases

 end
 end

+

 # Method executed after testing the module
 # Method executed after testing the module

-fun after_module do
-    # stop server after all test cases
+fun teardown_db is after_all do
+       # stop server after all test cases

 end
 end

-~~~~
+~~~
+
+When dealing with multiple test suites, niunit allows you to import other test suites to factorize your tests:
+
+~~~
+module test_bdd_users
+
+import test_bdd_connector
+
+# Testing the user table
+class TestUsersTable
+       test
+       # test cases using the db server from `test_bdd_connector`
+end
+
+fun setup_table is before_all do
+       # create user table
+end
+
+fun teardown_table is after_all do
+       # drop user table
+end
+~~~
+
+Methods with `before*` and `after*` annotations are linearized and called in different ways.
+
+* `before*` methods are called from the least specific to the most specific
+* `after*` methods are called from the most specific to the least specific
+
+In the previous example, the execution order would be:
+
+1. `test_bdd_connector::setup_db`
+2. `test_bdd_users::setup_table`
+3. `all test cases from test_bdd_users`
+4. `test_bdd_users::teardown_table`
+5. `test_bdd_connector::teardown_db`
+
+## Accessing the test suite environment
+
+The `NIT_TESTING_PATH` environment variable contains the current test suite
+file path.
+Nitunit define this variable before the execution of each test suite.
+It can be used to access files based on the current test suite location:
+
+~~~
+class TestWithPath
+    test
+
+    fun test_suite_path do
+        assert "NIT_TESTING_PATH".environ != ""
+    end
+end
+~~~

 
 ## Generating test suites
 
 
 ## Generating test suites
 

@@ -228,13 +376,22 @@ With the `--full` option, all imported modules (even those in standard) are also
 ### `-o`, `--output`
 Output name (default is 'nitunit.xml').
 
 ### `-o`, `--output`
 Output name (default is 'nitunit.xml').
 

-### `nitunit` produces a XML file comatible with JUnit.
+`nitunit` produces a XML file compatible with JUnit.

 
 ### `--dir`
 
 ### `--dir`

-Working directory (default is '.nitunit').
+Working directory (default is 'nitunit.out').

 
 In order to execute the tests, nit files are generated then compiled and executed in the giver working directory.
 
 
 In order to execute the tests, nit files are generated then compiled and executed in the giver working directory.
 

+In case of success, the directory is removed.
+In case of failure, it is kept as is so files can be investigated.
+
+### `--nitc`
+nitc compiler to use.
+
+By default, nitunit tries to locate the `nitc` program with the environment variable `NITC` or heuristics.
+The option is used to indicate a specific nitc binary.
+

 ### `--no-act`
 Does not compile and run tests.
 
 ### `--no-act`
 Does not compile and run tests.
 

@@ -243,8 +400,22 @@ Only run test case with name that match pattern.
 
 Examples: `TestFoo`, `TestFoo*`, `TestFoo::test_foo`, `TestFoo::test_foo*`, `test_foo`, `test_foo*`
 
 
 Examples: `TestFoo`, `TestFoo*`, `TestFoo::test_foo`, `TestFoo::test_foo*`, `test_foo`, `test_foo*`
 

-### `-t`, `--target-file`
-Specify test suite location.
+### `--autosav`
+Automatically create/update .res files for black box testing.
+
+If a black block test fails because a difference between the expected result and the current result then the expected result file is updated (and the test is passed).
+
+If a test-case of a test-suite passes but that some output is generated, then an expected result file is created.
+
+It is expected that the created/updated files are checked since the tests are considered passed.
+A VCS like `git` is often a good tool to check the creation and modification of those files.
+
+### `--no-time`
+Disable time information in XML.
+
+This is used to have reproducible XML results.
+
+This option is automatically activated if `NIT_TESTING` is set.

 
 ## SUITE GENERATION
 
 
 ## SUITE GENERATION
 

@@ -262,6 +433,40 @@ Also generate test case for private methods.
 ### `--only-show`
 Only display the skeleton, do not write any file.
 
 ### `--only-show`
 Only display the skeleton, do not write any file.
 

+
+# ENVIRONMENT VARIABLES
+
+### `NITC`
+
+Indicate the specific Nit compiler executable to use. See `--nitc`.
+
+### `NIT_TESTING`
+
+The environment variable `NIT_TESTING` is set to `true` during the execution of program tests.
+Some libraries of programs can use it to produce specific reproducible results; or just to exit their executions.
+
+Unit-tests may unset this environment variable to retrieve the original behavior of such piece of software.
+
+### `SRAND`
+
+In order to maximize reproducibility, `SRAND` is set to 0.
+This make the pseudo-random generator no random at all.
+See `Sys::srand` for details.
+
+To retrieve the randomness, unit-tests may unset this environment variable then call `srand`.
+
+### `NIT_TESTING_ID`
+
+Parallel executions can cause some race collisions on named resources (e.g. DB table names).
+To solve this issue, `NIT_TESTING_ID` is initialized with a distinct integer identifier that can be used to give unique names to resources.
+
+Note: `rand` is not a recommended way to get a distinct identifier because its randomness is disabled by default. See `SRAND`.
+
+### `NIT_TESTING_PATH`
+
+Only available for test suites.
+Contains the module test suite path.
+

 # SEE ALSO
 
 The Nit language documentation and the source code of its tools and libraries may be downloaded from <http://nitlanguage.org>
 # SEE ALSO
 
 The Nit language documentation and the source code of its tools and libraries may be downloaded from <http://nitlanguage.org>