Native model

A thin wrapper around serialized data which add information of identity and version.

See concepts for more details.

Goals

Interoperability: Allows different applications to work together, even if they are using different versions of the data model.
Data Consistency: Ensure that we process the data expected model.
Flexibility: You can use any serialization format you want. More details here.
Performance: A minimal overhead. More details here.

Usage

Application 1 (DotV1) Application 2 (DotV1 and DotV2) | | Encode DotV1 |----------------------------------------> | Decode DotV1 to DotV2 | | Modify DotV2 Decode DotV1 | <----------------------------------------| Encode DotV2 back to DotV1 | |

```rust,skt-main // Application 1 let dot = DotV1(1, 2); let bytes = native_model::encode(&dot).unwrap();

// Application 1 sends bytes to Application 2.

// Application 2 // We are able to decode the bytes directly into a new type DotV2 (upgrade). let (mut dot, sourceversion) = nativemodel::decode::(bytes).unwrap(); asserteq!(dot, DotV2 { name: "".tostring(), x: 1, y: 2 }); dot.name = "Dot".tostring(); dot.x = 5; // For interoperability, we encode the data with the version compatible with Application 1 (downgrade). let bytes = nativemodel::encodedowngrade(dot, sourceversion).unwrap();

// Application 2 sends bytes to Application 1.

// Application 1 let (dot, ) = nativemodel::decode::(bytes).unwrap(); assert_eq!(dot, DotV1(5, 2)); ```

Full example here.

When to use it? - Your applications that interact with each other are written in Rust. - Your applications evolve independently need to read serialized data coming from each other. - Your applications store data locally and need to read it later by a newer version of the application. - Your systems need to be upgraded incrementally. Instead of having to upgrade the entire system at once, individual applications can be upgraded one at a time, while still being able to communicate with each other.

When not to use it? - Your applications that interact with each other are not all written in Rust. - Your applications need to communicate with other systems that you don't control. - You need to have a human-readable format. (You can use a human-readable format like JSON wrapped in a native model, but you have to unwrap it to see the data correctly.)

Status

Early development. Not ready for production.

Setup your serialization format

First, you need to set up your serialization format. You can use any serialization format.

Just define the following functions, so they must be imported in the scope where you use the native model.

```rust,ignore fn nativemodelencode_body(obj: &T) -> Result, E> { ... }

fn nativemodeldecode_body(data: Vec) -> Result { ... } ```

With T and E the type depending on the serialization format you use. Just E need to implement the std::error::Error trait.

Examples: - bincode with encode/decode - bincode with serde

Setup your data model

Define your model using the macro native_model.

Attributes: - id = u32: The unique identifier of the model. - version = u32: The version of the model. - from = type: Optional, the previous version of the model. - type: The previous version of the model that you use for the From implementation. - try_from = (type, error): Optional, the previous version of the model with error handling. - type: The previous version of the model that you use for the TryFrom implementation. - error: The error type that you use for the TryFrom implementation.

```rust,skt-define-models use nativemodel::nativemodel;

[derive(Encode, Decode, PartialEq, Debug)]

[native_model(id = 1, version = 1)]

struct DotV1(u32, u32);

[derive(Encode, Decode, PartialEq, Debug)]

[native_model(id = 1, version = 2, from = DotV1)]

struct DotV2 { name: String, x: u64, y: u64, }

// Implement the conversion between versions From for DotV2 and From for DotV1.

[derive(Encode, Decode, PartialEq, Debug)]

[nativemodel(id = 1, version = 3, tryfrom = (DotV2, anyhow::Error))]

struct DotV3 { name: String, cord: Cord, }

[derive(Encode, Decode, PartialEq, Debug)]

struct Cord { x: u64, y: u64, }

// Implement the conversion between versions From for DotV3 and From for DotV2. ```

Full example here.

Concepts

In order to understand how the native model works, you need to understand the following concepts.

Identity(id): The identity is the unique identifier of the model. It is used to identify the model and prevent to decode a model into the wrong type.
Version(version) The version is the version of the model. It is used to check the compatibility between two models.
Encode: The encode is the process of converting a model into a byte array.
Decode: The decode is the process of converting a byte array into a model.
Downgrade: The downgrade is the process of converting a model into a previous version of the model.
Upgrade: The upgrade is the process of converting a model into a newer version of the model.

Under the hood, the native model is a thin wrapper around serialized data. The id and the version are twice encoded with a little_endian::U32. That represents 8 bytes, that are added at the beginning of the data.

+------------------+------------------+------------------------------------+ | ID (4 bytes) | Version (4 bytes)| Data (indeterminate-length bytes) | +------------------+------------------+------------------------------------+

Performance

This crate is in an early stage of development, so the performance should be improved in the future. The goal is to have a minimal and constant overhead for all data sizes. It uses the zerocopy crate to avoid unnecessary copies.

Current performance: - Encode time: have overhead that evolves linearly with the data size. - Decode time: have overhead of ~162 ps for all data sizes.

Benchmark of the native model overhead here.

To know how much time it takes to encode/decode your data, you need to add this overhead to the time of your serialization format.