MessagePack, an efficient binary serialization format

This modules provides services at different levels:

  • Serialize Nit objects using either the quick and easy Serializable::serialize_msgpack and Write::serialize_msgpack, or the extensible MsgPackSerializer.

  • Deserialize MessagePack to Nit objects using the quick and easy methods Reader|Bytes::deserialize_msgpack, or inspect errors with the extensible MsgPackDeserializer.

  • Read and write MessagePack data at a low-level (for engines and the likes) by importing read or write. These services support only which support only core Nit types with a corresponding MessagePack type. See Reader::read_msgpack and services on Writer including write_msgpack_ext.

Here we discuss the recommended serialization services supporting core Nit and MessagePack types as well as full Nit objects.

Primitive MessagePack types

Most Nit core types are serialized to the smallest corresponding MessagePack type.

assert false.serialize_msgpack == b"\xC2"
assert b"\xC2".deserialize_msgpack == false

assert true.serialize_msgpack  == b"\xC3"
assert b"\xC3".deserialize_msgpack == true

assert 1.234.serialize_msgpack == b"\xCB\x3F\xF3\xBE\x76\xC8\xB4\x39\x58"
assert b"\xCB\x3F\xF3\xBE\x76\xC8\xB4\x39\x58".deserialize_msgpack == 1.234

assert "ABC".serialize_msgpack == b"\xA3ABC"
assert b"\xA3ABC".deserialize_msgpack == "ABC"

assert [0x11, 0x22, 0x33].serialize_msgpack(plain=true) == b"\x93\x11\x22\x33"
assert b"\x93\x11\x22\x33".deserialize_msgpack == [0x11, 0x22, 0x33]

var map = new Map[String, nullable Object]
map["i"] = 1
map["o"] = null
assert map.serialize_msgpack(plain=true) == b"\x82\xA1\x69\x01\xA1\x6F\xC0"

Ints are serialized to the smallest MessagePack type, so a small integer fits in a single byte and larger integers take more bytes as needed.

assert 1.serialize_msgpack            == b"\x01"
assert (-32).serialize_msgpack        == b"\xE0"
assert 0x7F.serialize_msgpack         == b"\x7F"
assert 0x80.serialize_msgpack         == b"\xCC\x80"
assert 0x1234.serialize_msgpack       == b"\xCD\x12\x34"
assert (-0x1234).serialize_msgpack    == b"\xD1\xED\xCC"
assert 0x12345678.serialize_msgpack   == b"\xCE\x12\x34\x56\x78"
assert 0x0123456789.serialize_msgpack == b"\xCF\x00\x00\x00\x01\x23\x45\x67\x89"

assert b"\x01".deserialize_msgpack                 == 1
assert b"\xE0".deserialize_msgpack                 == -32
assert b"\x7F".deserialize_msgpack                 == 0x7F
assert b"\xCC\x80".deserialize_msgpack             == 0x80
assert b"\xCD\x12\x34".deserialize_msgpack         == 0x1234
assert b"\xD1\xED\xCC".deserialize_msgpack         == -0x1234
assert b"\xCE\x12\x34\x56\x78".deserialize_msgpack == 0x12345678
assert b"\xCF\x00\x00\x00\x01\x23\x45\x67\x89".deserialize_msgpack == 0x0123456789

Primitive Nit type without a MessagePack equivalent

Chars are serialized as a string in plain mode.

assert 'A'.serialize_msgpack(plain=true)      == b"\xA1\x41"
assert b"\xA1\x41".deserialize_msgpack == "A"

Or, with metadata, chars are serialized to an ext with id 0x7C.

assert 'A'.serialize_msgpack                      == b"\xD4\x7C\x41"
assert b"\xD4\x7C\x41".deserialize_msgpack == 'A'

Byte instances are serialized as an integer in plain mode.

assert 0x01u8.serialize_msgpack(plain=true) == b"\x01"
assert b"\x01".deserialize_msgpack   == 1

Or, with metadata, byte instances are serialized to an ext with id 0x7E.

assert 0x01u8.serialize_msgpack                   == b"\xD4\x7E\x01"
assert b"\xD4\x7E\x01".deserialize_msgpack == 1

Full objects

Objects are serialized to a map in plain mode, replacing cycles by null values. This creates plain MessagePack easy to read for other non-Nit programs, but cycles and the dynamic type information are lost.

class A
    serialize

    var i = 1
    var o: nullable A = self

    redef fun ==(o) do return o isa A and o.i == i # Skip the cyclic `o`
end

var a = new A
var bytes = a.serialize_msgpack(plain=true)
assert bytes == b"\x82\xA1\x69\x01\xA1\x6F\xC0"
assert bytes.deserialize_msgpack isa Map[nullable Serializable, nullable Serializable]

Or, with metadata, the same object is serialized with information on object uniqueness (with an id and references) and its dynamic type. The whole object is contained in a MessagePack array:

  • The array holds the metadata and attributes or each object, here it is a fixarray of 3 items: 0x93
  • Define an object (ext type 0x7B) with the id 0, here a fixext1: 0xD47B00
  • The dynamic type name, here a fixstr with the letter 'A': 0xA141
  • The attributes as a map, here a fixmap of 2 items: 0x82
  • First attribute name, here a fixstr for "i": 0xA169
  • First attribute value, here a fixint for 1: 0x01
  • Second attribute name, here a fixstr for "o": 0xA16F
  • Second attribute value, a reference (ext type 0x7D) to object id 0, here a fixext1: 0xD47D00
bytes = a.serialize_msgpack
assert bytes == b"\x93\xD4\x7B\x00\xA1\x41\x82\xA1\x69\x01\xA1\x6F\xD4\x7D\x00"
assert bytes.deserialize_msgpack == a

References

Format description and other implementations: http://msgpack.org/

Format specification: https://github.com/msgpack/msgpack/blob/master/spec.md

All class definitions

package_diagram msgpack::msgpack msgpack msgpack::serialization_write serialization_write msgpack::msgpack->msgpack::serialization_write msgpack::serialization_read serialization_read msgpack::msgpack->msgpack::serialization_read msgpack::serialization_common serialization_common msgpack::serialization_write->msgpack::serialization_common msgpack::write write msgpack::serialization_write->msgpack::write msgpack::ext ext msgpack::serialization_write->msgpack::ext msgpack::serialization_read->msgpack::serialization_common json json msgpack::serialization_read->json msgpack::read read msgpack::serialization_read->msgpack::read ...msgpack::serialization_common ... ...msgpack::serialization_common->msgpack::serialization_common ...msgpack::write ... ...msgpack::write->msgpack::write ...msgpack::ext ... ...msgpack::ext->msgpack::ext ...json ... ...json->json ...msgpack::read ... ...msgpack::read->msgpack::read gamnit::common common gamnit::common->msgpack::msgpack gamnit::client client gamnit::client->gamnit::common gamnit::server server gamnit::server->gamnit::common gamnit::client... ... gamnit::client...->gamnit::client gamnit::server... ... gamnit::server...->gamnit::server

Ancestors

module abstract_collection

core :: abstract_collection

Abstract collection classes and services.
module abstract_text

core :: abstract_text

Abstract class for manipulation of sequences of characters
module array

core :: array

This module introduces the standard array structure.
module binary

binary :: binary

Read and write binary data with any Reader and Writer
module bitset

core :: bitset

Services to handle BitSet
module bytes

core :: bytes

Services for byte streams and arrays
module caching

serialization :: caching

Services for caching serialization engines
module circular_array

core :: circular_array

Efficient data structure to access both end of the sequence.
module codec_base

core :: codec_base

Base for codecs to use with streams
module codecs

core :: codecs

Group module for all codec-related manipulations
module collection

core :: collection

This module define several collection classes.
module core

core :: core

Standard classes and methods used by default by Nit programs and libraries.
module engine_tools

serialization :: engine_tools

Advanced services for serialization engines
module environ

core :: environ

Access to the environment variables of the process
module error

json :: error

Intro JsonParseError which is exposed by all JSON reading APIs
module error

core :: error

Standard error-management infrastructure.
module exec

core :: exec

Invocation and management of operating system sub-processes.
module ext

msgpack :: ext

Application specific MessagePack extension MsgPackExt
module file

core :: file

File manipulations (create, read, write, etc.)
module fixed_ints

core :: fixed_ints

Basic integers of fixed-precision
module fixed_ints_text

core :: fixed_ints_text

Text services to complement fixed_ints
module flat

core :: flat

All the array-based text representations
module gc

core :: gc

Access to the Nit internal garbage collection mechanism
module hash_collection

core :: hash_collection

Introduce HashMap and HashSet.
module inspect

serialization :: inspect

Refine Serializable::inspect to show more useful information
module iso8859_1

core :: iso8859_1

Codec for ISO8859-1 I/O
module json

json :: json

Read and write JSON formatted text using the standard serialization services
module kernel

core :: kernel

Most basic classes and methods.
module list

core :: list

This module handle double linked lists
module math

core :: math

Mathematical operations
module meta

meta :: meta

Simple user-defined meta-level to manipulate types of instances as object.
module native

core :: native

Native structures for text and bytes
module numeric

core :: numeric

Advanced services for Numeric types
module parser_base

parser_base :: parser_base

Simple base for hand-made parsers of all kinds
module poset

poset :: poset

Pre order sets and partial order set (ie hierarchies)
module protocol

core :: protocol

module queue

core :: queue

Queuing data structures and wrappers
module range

core :: range

Module for range of discrete objects.
module re

core :: re

Regular expression support for all services based on Pattern
module read

msgpack :: read

Low-level read MessagePack format from Reader streams
module ropes

core :: ropes

Tree-based representation of a String.
module safe

serialization :: safe

Services for safer deserialization engines
module serialization

serialization :: serialization

General serialization services
module serialization_common

msgpack :: serialization_common

Serialization services for serialization_write and serialization_read
module serialization_core

serialization :: serialization_core

Abstract services to serialize Nit objects to different formats
module serialization_read

json :: serialization_read

Services to read JSON: deserialize_json and JsonDeserializer
module serialization_write

json :: serialization_write

Services to write Nit objects to JSON strings: serialize_to_json and JsonSerializer
module sorter

core :: sorter

This module contains classes used to compare things and sorts arrays.
module static

json :: static

Static interface to read Nit objects from JSON strings
module stream

core :: stream

Input and output streams of characters
module text

core :: text

All the classes and methods related to the manipulation of text entities
module time

core :: time

Management of time and dates
module union_find

core :: union_find

union–find algorithm using an efficient disjoint-set data structure
module utf8

core :: utf8

Codec for UTF-8 I/O
module write

msgpack :: write

Low-level write in MessagePack format to Writer streams

Parents

module serialization_read

msgpack :: serialization_read

Deserialize full Nit objects from MessagePack format
module serialization_write

msgpack :: serialization_write

Serialize full Nit objects to MessagePack format

Children

module common

gamnit :: common

Services common to the client and server modules

Descendants

module a_star-m

a_star-m

module client

gamnit :: client

Client-side network services for games and such
module network

gamnit :: network

Easy client/server logic for games and simple distributed applications
module server

gamnit :: server

Server-side network services for games and such
# MessagePack, an efficient binary serialization format
#
# This modules provides services at different levels:
#
# * Serialize Nit objects using either the quick and easy `Serializable::serialize_msgpack`
#   and `Write::serialize_msgpack`, or the extensible `MsgPackSerializer`.
#
# * Deserialize MessagePack to Nit objects using the quick and easy methods
#   `Reader|Bytes::deserialize_msgpack`, or inspect errors with the extensible
#   `MsgPackDeserializer`.
#
# * Read and write MessagePack data at a low-level (for engines and the likes)
#   by importing `msgpack::read` or `msgpack::write`. These services support
#   only which support only core Nit types with a corresponding MessagePack type.
#   See `Reader::read_msgpack` and services on `Writer` including `write_msgpack_ext`.
#
# Here we discuss the recommended serialization services supporting core
# Nit and MessagePack types as well as full Nit objects.
#
# ## Primitive MessagePack types
#
# Most Nit core types are serialized to the smallest corresponding MessagePack type.
#
# ~~~
# assert false.serialize_msgpack == b"\xC2"
# assert b"\xC2".deserialize_msgpack == false
#
# assert true.serialize_msgpack  == b"\xC3"
# assert b"\xC3".deserialize_msgpack == true
#
# assert 1.234.serialize_msgpack == b"\xCB\x3F\xF3\xBE\x76\xC8\xB4\x39\x58"
# assert b"\xCB\x3F\xF3\xBE\x76\xC8\xB4\x39\x58".deserialize_msgpack == 1.234
#
# assert "ABC".serialize_msgpack == b"\xA3ABC"
# assert b"\xA3ABC".deserialize_msgpack == "ABC"
#
# assert [0x11, 0x22, 0x33].serialize_msgpack(plain=true) == b"\x93\x11\x22\x33"
# assert b"\x93\x11\x22\x33".deserialize_msgpack == [0x11, 0x22, 0x33]
#
# var map = new Map[String, nullable Object]
# map["i"] = 1
# map["o"] = null
# assert map.serialize_msgpack(plain=true) == b"\x82\xA1\x69\x01\xA1\x6F\xC0"
# ~~~
#
# Ints are serialized to the smallest MessagePack type, so a small integer fits
# in a single byte and larger integers take more bytes as needed.
#
# ~~~
# assert 1.serialize_msgpack            == b"\x01"
# assert (-32).serialize_msgpack        == b"\xE0"
# assert 0x7F.serialize_msgpack         == b"\x7F"
# assert 0x80.serialize_msgpack         == b"\xCC\x80"
# assert 0x1234.serialize_msgpack       == b"\xCD\x12\x34"
# assert (-0x1234).serialize_msgpack    == b"\xD1\xED\xCC"
# assert 0x12345678.serialize_msgpack   == b"\xCE\x12\x34\x56\x78"
# assert 0x0123456789.serialize_msgpack == b"\xCF\x00\x00\x00\x01\x23\x45\x67\x89"
#
# assert b"\x01".deserialize_msgpack                 == 1
# assert b"\xE0".deserialize_msgpack                 == -32
# assert b"\x7F".deserialize_msgpack                 == 0x7F
# assert b"\xCC\x80".deserialize_msgpack             == 0x80
# assert b"\xCD\x12\x34".deserialize_msgpack         == 0x1234
# assert b"\xD1\xED\xCC".deserialize_msgpack         == -0x1234
# assert b"\xCE\x12\x34\x56\x78".deserialize_msgpack == 0x12345678
# assert b"\xCF\x00\x00\x00\x01\x23\x45\x67\x89".deserialize_msgpack == 0x0123456789
# ~~~
#
# ## Primitive Nit type without a MessagePack equivalent
#
# Chars are serialized as a string in plain mode.
#
# ~~~
# assert 'A'.serialize_msgpack(plain=true)      == b"\xA1\x41"
# assert b"\xA1\x41".deserialize_msgpack == "A" # Not a Char
# ~~~
#
# Or, with metadata, chars are serialized to an ext with id 0x7C.
#
# ~~~
# assert 'A'.serialize_msgpack                      == b"\xD4\x7C\x41"
# assert b"\xD4\x7C\x41".deserialize_msgpack == 'A'
# ~~~
#
# Byte instances are serialized as an integer in plain mode.
#
# ~~~
# assert 0x01u8.serialize_msgpack(plain=true) == b"\x01"
# assert b"\x01".deserialize_msgpack   == 1 # Not a Byte
# ~~~
#
# Or, with metadata, byte instances are serialized to an ext with id 0x7E.
#
# ~~~
# assert 0x01u8.serialize_msgpack                   == b"\xD4\x7E\x01"
# assert b"\xD4\x7E\x01".deserialize_msgpack == 1
# ~~~
#
# ## Full objects
#
# Objects are serialized to a map in plain mode, replacing cycles by `null` values.
# This creates plain MessagePack easy to read for other non-Nit programs,
# but cycles and the dynamic type information are lost.
#
# ~~~
# class A
#     serialize
#
#     var i = 1
#     var o: nullable A = self
#
#     redef fun ==(o) do return o isa A and o.i == i # Skip the cyclic `o`
# end
#
# var a = new A
# var bytes = a.serialize_msgpack(plain=true)
# assert bytes == b"\x82\xA1\x69\x01\xA1\x6F\xC0"
# assert bytes.deserialize_msgpack isa Map[nullable Serializable, nullable Serializable] # Not an A
# ~~~
#
# Or, with metadata, the same object is serialized with information on object
# uniqueness (with an id and references) and its dynamic type.
# The whole object is contained in a MessagePack array:
#
# * The array holds the metadata and attributes or each object,
#   here it is a fixarray of 3 items: 0x93
# * Define an object (ext type 0x7B) with the id 0, here a fixext1: 0xD47B00
# * The dynamic type name, here a fixstr with the letter 'A': 0xA141
# * The attributes as a map, here a fixmap of 2 items: 0x82
# * First attribute name, here a fixstr for "i": 0xA169
# * First attribute value, here a fixint for 1: 0x01
# * Second attribute name, here a fixstr for "o": 0xA16F
# * Second attribute value, a reference (ext type 0x7D) to object id 0,
#   here a fixext1: 0xD47D00
#
# ~~~
# bytes = a.serialize_msgpack
# assert bytes == b"\x93\xD4\x7B\x00\xA1\x41\x82\xA1\x69\x01\xA1\x6F\xD4\x7D\x00"
# assert bytes.deserialize_msgpack == a
# ~~~
#
# ## References
#
# Format description and other implementations: http://msgpack.org/
#
# Format specification: https://github.com/msgpack/msgpack/blob/master/spec.md
module msgpack

import serialization_write
import serialization_read
lib/msgpack/msgpack.nit:15,1--164,25