Native model

A thin wrapper around serialized data which add information of identity and version.

Goals

• Interoperability: Allows different applications to work together, even if they are using different versions of the data model.
• Data Consistency: Ensure that we process the data expected model.
• Flexibility: You can use any serialization format you want. More details here.
• Performance: A minimal overhead. More details here.

Usage

Application 1 (DotV1) Application 2 (DotV1 and DotV2) | | Encode DotV1 |----------------------------------------> | Decode DotV1 to DotV2 | | Modify DotV2 Decode DotV1 | <----------------------------------------| Encode DotV2 back to DotV1 | |

```rust,skt-main // Application 1 let dot = DotV1(1, 2); let bytes = native_model::encode(&dot).unwrap();

// Application 1 sends bytes to Application 2.

// Application 2 // We are able to decode the bytes directly into a new type DotV2. let (mut dot, sourceversion) = nativemodel::decode::(bytes).unwrap(); asserteq!(dot, DotV2 { name: "".tostring(), x: 1, y: 2 }); dot.name = "Dot".tostring(); dot.x = 5; // For interoperability, we encode the data with the version compatible with Application 1. let bytes = nativemodel::encodedowngrade(dot, sourceversion).unwrap();

// Application 2 sends bytes to Application 1.

// Application 1 let (dot, ) = nativemodel::decode::(bytes).unwrap(); assert_eq!(dot, DotV1(5, 2)); ```

Full example here.

When use it? - Your applications that interact with each other are written in Rust. - Your applications evolve independently need to read serialized data coming from each other. - Your applications store data locally and need to read it later by a newer version of the application. - Your systems need to be upgraded incrementally. Instead of having to upgrade the entire system at once, individual applications can be upgraded one at a time, while still being able to communicate with each other.

When not use it? - Your applications that interact with each other are not all written in Rust. - Your applications need to communicate with other systems that you don't control. - You need to have a human-readable format. (You can use a human-readable format like JSON wrapped in a native model, but you have to unwrap it to see the data correctly.)

Status

Early development. Not ready for production.

Setup your serialization format

First, you need to set up your serialization format. You can use any serialization format.

Just define the following functions, so they must be imported in the scope where you use the native model.

```rust,ignore fn nativemodelencode_body(obj: &T) -> Result, dyn Error> { ... }

fn nativemodeldecode_body(data: Vec) -> Result { ... } ``` Examples: - bincode with encode/decode - bincode with serde

Setup your data model

Define your model using the macro native_model.

Attributes: - id = u32: The unique identifier of the model. - version = u32: The version of the model. - from = type: Optional, the previous version of the model. - type: The previous version of the model that you use for the From implementation. - try_from = (type, error): Optional, the previous version of the model with error handling. - type: The previous version of the model that you use for the TryFrom implementation. - error: The error type that you use for the TryFrom implementation.

```rust,skt-define-models use nativemodel::nativemodel;

[derive(Encode, Decode, PartialEq, Debug)]

[native_model(id = 1, version = 1)]

struct DotV1(u32, u32);

[derive(Encode, Decode, PartialEq, Debug)]

[native_model(id = 1, version = 2, from = DotV1)]

struct DotV2 { name: String, x: u64, y: u64, }

// Implement the conversion between versions From for DotV2 and From for DotV1.

[derive(Encode, Decode, PartialEq, Debug)]

[nativemodel(id = 1, version = 3, tryfrom = (DotV2, anyhow::Error))]

struct DotV3 { name: String, cord: Cord, }

[derive(Encode, Decode, PartialEq, Debug)]

struct Cord { x: u64, y: u64, }

// Implement the conversion between versions From for DotV3 and From for DotV2. ```

Full example here.

Performance

This crate is in an early stage of development, so the performance should be improved in the future. The goal is to have a minimal and constant overhead for all data sizes. It uses the zerocopy crate to avoid unnecessary copies.

Current performance: - Encode time: have overhead that evolves linearly with the data size. - Decode time: have overhead of ~162 ps for all data sizes.

| data size | encode time (ns/ps/µs/ms) | decode time (ps) | |:---------------------:|:--------------------------:|:----------------:| | 1 B | 40.093 ns - 40.510 ns | 161.87 ps - 162.02 ps | | 1 KiB (1024 B) | 116.45 ns - 116.83 ns | 161.85 ps - 162.08 ps | | 1 MiB (1048576 B) | 66.697 µs - 67.634 µs | 161.87 ps - 162.18 ps | | 10 MiB (10485760 B) | 1.5670 ms - 1.5843 ms | 162.40 ps - 163.52 ps | | 100 MiB (104857600 B) | 63.778 ms - 64.132 ms | 162.71 ps - 165.10 ps |

Benchmark of the native model overhead here.

To know how much time it takes to encode/decode your data, you need to add this overhead to the time of your serialization format.