Merge: Loose Tokens

Another step to simplify the clients of AST: loose tokens.

Some tokens, correctly lexed by the lexer, are discarded by the parser because of (thanks to?) sablecc transformations. Mainly comments and new-lines (EOL) are lost but some other tokens might be also discarded (currently, commas and dots are still lost).

Previously, the full sequence of tokens can still be accessed trough `next_token` and `prev_token` but this mean that tools like nitprettty and nitlight have to maintain two cursors, one on the AST and one on this linked list of tokens and try to keep these two cursors synchronized.

This PR names the concept of *loose* tokens that are lexed tokens but absent from the AST, and attach to each (non-loose) token in the AST a list of loose tokens that precede it (`prev_looses`) and that follow it (`next_looses`).
This way, a visit on the AST can be used to also easily process all the tokens from the source.

Future PRs will try to use this new thing to improve/simplify/whatever nitpretty and nitlight.

Pull-Request: #1346
Reviewed-by: Alexandre Terrasa <alexandre@moz-code.org>
Reviewed-by: Lucas Bajolet <r4pass@hotmail.com>

diff --combined src/parser/parser_nodes.nit
index 6ec146f,6d7233e..9131c3e
--- 1/src/parser/parser_nodes.nit
--- 2/src/parser/parser_nodes.nit
+++ b/src/parser/parser_nodes.nit
@@@ -311,6 -311,35 +311,35 @@@ abstract class Toke
        # May have disappeared in the AST
        var next_token: nullable Token = null
  
+       # Is `self` a token discarded from the AST?
+       #
+       # Loose tokens are not present in the AST.
+       # It means they were identified by the lexer but were discarded by the parser.
+       # It also means that they are not visited or manipulated by AST-related functions.
+       #
+       # Each loose token is attached to the non-loose token that precedes or follows it.
+       # The rules are the following:
+       #
+       # * tokens that follow a non-loose token on a same line are attached to it.
+       #   See `next_looses`.
+       # * other tokens, thus that precede a non-loose token on the same line or the next one,
+       # are attached to this one. See `prev_looses`.
+       #
+       # Loose tokens are mostly end of lines (`TEol`) and comments (`TComment`).
+       # Whitespace are ignored by the lexer, so they are not even considered as loose tokens.
+       # See `blank_before` to get the whitespace that separate tokens.
+       var is_loose = false
+ 
+       # Loose tokens that precede `self`.
+       #
+       # These tokens start the line or belong to a line with only loose tokens.
+       var prev_looses = new Array[Token] is lazy
+ 
+       # Loose tokens that follow `self`
+       #
+       # These tokens are on the same line than `self`.
+       var next_looses = new Array[Token] is lazy
+ 
        # The verbatim blank text between `prev_token` and `self`
        fun blank_before: String
        do
@@@ -946,36 -975,6 +975,36 @@@ class THexNumbe
        super TokenLiteral
  end
  
 +# A literal binary integer
 +class TBinNumber
 +      super TokenLiteral
 +end
 +
 +# A literal octal integer
 +class TOctNumber
 +      super TokenLiteral
 +end
 +
 +# A literal decimal byte
 +class TBytenum
 +      super TokenLiteral
 +end
 +
 +# A literal hexadecimal byte
 +class THexBytenum
 +      super TokenLiteral
 +end
 +
 +# A literal binary byte
 +class TBinBytenum
 +      super TokenLiteral
 +end
 +
 +# A literal octal byte
 +class TOctBytenum
 +      super TokenLiteral
 +end
 +
  # A literal floating point number
  class TFloat
        super TokenLiteral
@@@ -2492,59 -2491,6 +2521,59 @@@ class AHexIntExp
        var n_hex_number: THexNumber is writable, noinit
  end
  
 +# An integer literal in binary format
 +class ABinIntExpr
 +      super AIntExpr
 +
 +      # The binary token
 +      var n_bin_number: TBinNumber is writable, noinit
 +end
 +
 +# An integer literal in octal format
 +class AOctIntExpr
 +      super AIntExpr
 +
 +      # The octal token
 +      var n_oct_number: TOctNumber is writable, noinit
 +end
 +
 +# An byte literal
 +class AByteExpr
 +      super AExpr
 +end
 +
 +# An byte literal in decimal format
 +class ADecByteExpr
 +      super AByteExpr
 +
 +      # The decimal token
 +      var n_bytenum: TBytenum is writable, noinit
 +end
 +
 +# An byte literal in hexadecimal format
 +class AHexByteExpr
 +      super AByteExpr
 +
 +      # The hexadecimal token
 +      var n_hex_bytenum: THexBytenum is writable, noinit
 +end
 +
 +# An byte literal in binary format
 +class ABinByteExpr
 +      super AByteExpr
 +
 +      # The binary token
 +      var n_bin_bytenum: TBinBytenum is writable, noinit
 +end
 +
 +# An byte literal in octal format
 +class AOctByteExpr
 +      super AByteExpr
 +
 +      # The octal token
 +      var n_oct_bytenum: TOctBytenum is writable, noinit
 +end
 +
  # A float literal
  class AFloatExpr
        super AExpr