msgpack

package msgpack
MessagePack, an efficient binary serialization format

Concerns

  • msgpack: MessagePack, an efficient binary serialization format
    • serialization_common: Serialization services for serialization_write and serialization_read
    • write: Low-level write in MessagePack format to Writer streams
    • ext: Application specific MessagePack extension MsgPackExt
    • read: Low-level read MessagePack format from Reader streams
    • serialization_write: Serialize full Nit objects to MessagePack format
    • msgpack_to_json: Convert MessagePack format to JSON
    • serialization_read: Deserialize full Nit objects from MessagePack format
    • msgpack: MessagePack, an efficient binary serialization format

msgpack::serialization_write

Serialize full Nit objects to MessagePack format

There are 3 main entrypoint services:

  • Writer::serialize_msgpack adds an object to any stream writer.
  • Serializable::serialize_msgpack serializes the object to bytes.
  • MsgPackSerializer gives full control over the serialization of Nit objets to the MessagePack format.
Redefines
  • Serializable: Instances of this class can be passed to Serializer::serialize
  • MsgPackExt: Application specific MessagePack extension
  • Map: Maps are associative collections: key -> item.
  • Writer: A Stream that can be written to
  • Bool: Native Booleans.
  • SimpleCollection: Items can be added to these collections.
  • CString: C string char *
  • Char: Native characters.
  • Byte: Native bytes.
  • Float: Native floating point numbers.
  • Int: Native integer numbers.
  • Text: High-level abstraction for all text representations
  • Bytes: A buffer containing Byte-manipulation facilities

msgpack::serialization_read

Deserialize full Nit objects from MessagePack format

See the package msgpack for more details on the serialization of Nit objects.

Redefines
  • Map: Maps are associative collections: key -> item.
  • Reader: A Stream that can be read from
  • SimpleCollection: Items can be added to these collections.
  • Bytes: A buffer containing Byte-manipulation facilities

msgpack::msgpack

module msgpack

MessagePack, an efficient binary serialization format

This modules provides services at different levels:

  • Serialize Nit objects using either the quick and easy Serializable::serialize_msgpack and Write::serialize_msgpack, or the extensible MsgPackSerializer.

  • Deserialize MessagePack to Nit objects using the quick and easy methods Reader|Bytes::deserialize_msgpack, or inspect errors with the extensible MsgPackDeserializer.

  • Read and write MessagePack data at a low-level (for engines and the likes) by importing msgpack::read or msgpack::write. These services support only which support only core Nit types with a corresponding MessagePack type. See Reader::read_msgpack and services on Writer including write_msgpack_ext.

Here we discuss the recommended serialization services supporting core Nit and MessagePack types as well as full Nit objects.

Primitive MessagePack types

Most Nit core types are serialized to the smallest corresponding MessagePack type.

assert false.serialize_msgpack == b"\xC2"
assert b"\xC2".deserialize_msgpack == false

assert true.serialize_msgpack  == b"\xC3"
assert b"\xC3".deserialize_msgpack == true

assert 1.234.serialize_msgpack == b"\xCB\x3F\xF3\xBE\x76\xC8\xB4\x39\x58"
assert b"\xCB\x3F\xF3\xBE\x76\xC8\xB4\x39\x58".deserialize_msgpack == 1.234

assert "ABC".serialize_msgpack == b"\xA3ABC"
assert b"\xA3ABC".deserialize_msgpack == "ABC"

assert [0x11, 0x22, 0x33].serialize_msgpack(plain=true) == b"\x93\x11\x22\x33"
assert b"\x93\x11\x22\x33".deserialize_msgpack == [0x11, 0x22, 0x33]

var map = new Map[String, nullable Object]
map["i"] = 1
map["o"] = null
assert map.serialize_msgpack(plain=true) == b"\x82\xA1\x69\x01\xA1\x6F\xC0"

Ints are serialized to the smallest MessagePack type, so a small integer fits in a single byte and larger integers take more bytes as needed.

assert 1.serialize_msgpack            == b"\x01"
assert (-32).serialize_msgpack        == b"\xE0"
assert 0x7F.serialize_msgpack         == b"\x7F"
assert 0x80.serialize_msgpack         == b"\xCC\x80"
assert 0x1234.serialize_msgpack       == b"\xCD\x12\x34"
assert (-0x1234).serialize_msgpack    == b"\xD1\xED\xCC"
assert 0x12345678.serialize_msgpack   == b"\xCE\x12\x34\x56\x78"
assert 0x0123456789.serialize_msgpack == b"\xCF\x00\x00\x00\x01\x23\x45\x67\x89"

assert b"\x01".deserialize_msgpack                 == 1
assert b"\xE0".deserialize_msgpack                 == -32
assert b"\x7F".deserialize_msgpack                 == 0x7F
assert b"\xCC\x80".deserialize_msgpack             == 0x80
assert b"\xCD\x12\x34".deserialize_msgpack         == 0x1234
assert b"\xD1\xED\xCC".deserialize_msgpack         == -0x1234
assert b"\xCE\x12\x34\x56\x78".deserialize_msgpack == 0x12345678
assert b"\xCF\x00\x00\x00\x01\x23\x45\x67\x89".deserialize_msgpack == 0x0123456789

Primitive Nit type without a MessagePack equivalent

Chars are serialized as a string in plain mode.

assert 'A'.serialize_msgpack(plain=true)      == b"\xA1\x41"
assert b"\xA1\x41".deserialize_msgpack == "A"

Or, with metadata, chars are serialized to an ext with id 0x7C.

assert 'A'.serialize_msgpack                      == b"\xD4\x7C\x41"
assert b"\xD4\x7C\x41".deserialize_msgpack == 'A'

Byte instances are serialized as an integer in plain mode.

assert 0x01u8.serialize_msgpack(plain=true) == b"\x01"
assert b"\x01".deserialize_msgpack   == 1

Or, with metadata, byte instances are serialized to an ext with id 0x7E.

assert 0x01u8.serialize_msgpack                   == b"\xD4\x7E\x01"
assert b"\xD4\x7E\x01".deserialize_msgpack == 0x01u8

Full objects

Objects are serialized to a map in plain mode, replacing cycles by null values. This creates plain MessagePack easy to read for other non-Nit programs, but cycles and the dynamic type information are lost.

class A
    serialize

    var i = 1
    var o: nullable A = self

    redef fun ==(o) do return o isa A and o.i == i # Skip the cyclic `o`
end

var a = new A
var bytes = a.serialize_msgpack(plain=true)
assert bytes == b"\x82\xA1\x69\x01\xA1\x6F\xC0"
assert bytes.deserialize_msgpack isa Map[nullable Serializable, nullable Serializable]

Or, with metadata, the same object is serialized with information on object uniqueness (with an id and references) and its dynamic type. The whole object is contained in a MessagePack array:

  • The array holds the metadata and attributes or each object, here it is a fixarray of 3 items: 0x93
  • Define an object (ext type 0x7B) with the id 0, here a fixext1: 0xD47B00
  • The dynamic type name, here a fixstr with the letter 'A': 0xA141
  • The attributes as a map, here a fixmap of 2 items: 0x82
  • First attribute name, here a fixstr for "i": 0xA169
  • First attribute value, here a fixint for 1: 0x01
  • Second attribute name, here a fixstr for "o": 0xA16F
  • Second attribute value, a reference (ext type 0x7D) to object id 0, here a fixext1: 0xD47D00
bytes = a.serialize_msgpack
assert bytes == b"\x93\xD4\x7B\x00\xA1\x41\x82\xA1\x69\x01\xA1\x6F\xD4\x7D\x00"
assert bytes.deserialize_msgpack == a

References

Format description and other implementations: http://msgpack.org/

Format specification: https://github.com/msgpack/msgpack/blob/master/spec.md