MessagePack, an efficient binary serialization format

This modules provides services at different levels:

  • Serialize Nit objects using either the quick and easy Serializable::serialize_msgpack and Write::serialize_msgpack, or the extensible MsgPackSerializer.

  • Deserialize MessagePack to Nit objects using the quick and easy methods Reader|Bytes::deserialize_msgpack, or inspect errors with the extensible MsgPackDeserializer.

  • Read and write MessagePack data at a low-level (for engines and the likes) by importing read or write. These services support only which support only core Nit types with a corresponding MessagePack type. See Reader::read_msgpack and services on Writer including write_msgpack_ext.

Here we discuss the recommended serialization services supporting core Nit and MessagePack types as well as full Nit objects.

Primitive MessagePack types

Most Nit core types are serialized to the smallest corresponding MessagePack type.

assert false.serialize_msgpack == b"\xC2"
assert b"\xC2".deserialize_msgpack == false

assert true.serialize_msgpack  == b"\xC3"
assert b"\xC3".deserialize_msgpack == true

assert 1.234.serialize_msgpack == b"\xCB\x3F\xF3\xBE\x76\xC8\xB4\x39\x58"
assert b"\xCB\x3F\xF3\xBE\x76\xC8\xB4\x39\x58".deserialize_msgpack == 1.234

assert "ABC".serialize_msgpack == b"\xA3ABC"
assert b"\xA3ABC".deserialize_msgpack == "ABC"

assert [0x11, 0x22, 0x33].serialize_msgpack(plain=true) == b"\x93\x11\x22\x33"
assert b"\x93\x11\x22\x33".deserialize_msgpack == [0x11, 0x22, 0x33]

var map = new Map[String, nullable Object]
map["i"] = 1
map["o"] = null
assert map.serialize_msgpack(plain=true) == b"\x82\xA1\x69\x01\xA1\x6F\xC0"

Ints are serialized to the smallest MessagePack type, so a small integer fits in a single byte and larger integers take more bytes as needed.

assert 1.serialize_msgpack            == b"\x01"
assert (-32).serialize_msgpack        == b"\xE0"
assert 0x7F.serialize_msgpack         == b"\x7F"
assert 0x80.serialize_msgpack         == b"\xCC\x80"
assert 0x1234.serialize_msgpack       == b"\xCD\x12\x34"
assert (-0x1234).serialize_msgpack    == b"\xD1\xED\xCC"
assert 0x12345678.serialize_msgpack   == b"\xCE\x12\x34\x56\x78"
assert 0x0123456789.serialize_msgpack == b"\xCF\x00\x00\x00\x01\x23\x45\x67\x89"

assert b"\x01".deserialize_msgpack                 == 1
assert b"\xE0".deserialize_msgpack                 == -32
assert b"\x7F".deserialize_msgpack                 == 0x7F
assert b"\xCC\x80".deserialize_msgpack             == 0x80
assert b"\xCD\x12\x34".deserialize_msgpack         == 0x1234
assert b"\xD1\xED\xCC".deserialize_msgpack         == -0x1234
assert b"\xCE\x12\x34\x56\x78".deserialize_msgpack == 0x12345678
assert b"\xCF\x00\x00\x00\x01\x23\x45\x67\x89".deserialize_msgpack == 0x0123456789

Primitive Nit type without a MessagePack equivalent

Chars are serialized as a string in plain mode.

assert 'A'.serialize_msgpack(plain=true)      == b"\xA1\x41"
assert b"\xA1\x41".deserialize_msgpack == "A"

Or, with metadata, chars are serialized to an ext with id 0x7C.

assert 'A'.serialize_msgpack                      == b"\xD4\x7C\x41"
assert b"\xD4\x7C\x41".deserialize_msgpack == 'A'

Byte instances are serialized as an integer in plain mode.

assert 0x01u8.serialize_msgpack(plain=true) == b"\x01"
assert b"\x01".deserialize_msgpack   == 1

Or, with metadata, byte instances are serialized to an ext with id 0x7E.

assert 0x01u8.serialize_msgpack                   == b"\xD4\x7E\x01"
assert b"\xD4\x7E\x01".deserialize_msgpack == 1

Full objects

Objects are serialized to a map in plain mode, replacing cycles by null values. This creates plain MessagePack easy to read for other non-Nit programs, but cycles and the dynamic type information are lost.

class A

    var i = 1
    var o: nullable A = self

    redef fun ==(o) do return o isa A and o.i == i # Skip the cyclic `o`

var a = new A
var bytes = a.serialize_msgpack(plain=true)
assert bytes == b"\x82\xA1\x69\x01\xA1\x6F\xC0"
assert bytes.deserialize_msgpack isa Map[nullable Serializable, nullable Serializable]

Or, with metadata, the same object is serialized with information on object uniqueness (with an id and references) and its dynamic type. The whole object is contained in a MessagePack array:

  • The array holds the metadata and attributes or each object, here it is a fixarray of 3 items: 0x93
  • Define an object (ext type 0x7B) with the id 0, here a fixext1: 0xD47B00
  • The dynamic type name, here a fixstr with the letter 'A': 0xA141
  • The attributes as a map, here a fixmap of 2 items: 0x82
  • First attribute name, here a fixstr for "i": 0xA169
  • First attribute value, here a fixint for 1: 0x01
  • Second attribute name, here a fixstr for "o": 0xA16F
  • Second attribute value, a reference (ext type 0x7D) to object id 0, here a fixext1: 0xD47D00
bytes = a.serialize_msgpack
assert bytes == b"\x93\xD4\x7B\x00\xA1\x41\x82\xA1\x69\x01\xA1\x6F\xD4\x7D\x00"
assert bytes.deserialize_msgpack == a


Format description and other implementations:

Format specification:

All groups and modules

module ext

msgpack :: ext

Application specific MessagePack extension MsgPackExt
module msgpack

msgpack :: msgpack

MessagePack, an efficient binary serialization format
module msgpack_to_json

msgpack :: msgpack_to_json

Convert MessagePack format to JSON
module read

msgpack :: read

Low-level read MessagePack format from Reader streams
module serialization_common

msgpack :: serialization_common

Serialization services for serialization_write and serialization_read
module serialization_read

msgpack :: serialization_read

Deserialize full Nit objects from MessagePack format
module serialization_write

msgpack :: serialization_write

Serialize full Nit objects to MessagePack format
module write

msgpack :: write

Low-level write in MessagePack format to Writer streams
package_diagram msgpack msgpack serialization serialization msgpack->serialization json json msgpack->json binary binary msgpack->binary core core msgpack->core serialization->json poset poset serialization->poset meta meta serialization->meta binary->core ...poset ... ...poset->poset ...meta ... ...meta->meta ...json ... ...json->json ...core ... ...core->core gamnit gamnit gamnit->msgpack android android android->gamnit android... ... android...->android


package meta


Simple user-defined meta-level to manipulate types of instances as object.
package parser_base


Simple base for hand-made parsers of all kinds
package poset


Pre order sets and partial order set (ie hierarchies)


package binary


Read and write binary data with any Reader and Writer
package core


Nit common library of core classes and methods
package json


read and write JSON formatted text
package serialization


Abstract serialization services


package gamnit


Portable game and multimedia framework for Nit


package android


Android platform support and APIs
package app


app.nit, a framework for portable applications
package egl


Interface between rendering APIs (OpenGL, OpenGL ES, etc.) and the native windowing system.
package glesv2


OpenGL graphics rendering library for embedded systems, version 2.0
package ios


iOS support for app.nit
package linux


Implementation of app.nit for the Linux platform