Alkahest is blazing-fast, zero-deps, zero-overhead, zero-unsafe, schema-based serialization library. It is suitable for broad range of use-cases, but tailored for custom high-performance network protocols.
This benchmark that mimics some game networking protocol.
| | alkahest | bincode | rkyv | speedy |
|:----------------|:-------------------------|:--------------------------------|:--------------------------------|:-------------------------------- |
| serialize | 10.73 us (✅ 1.00x) | 10.96 us (✅ 1.02x slower) | 12.21 us (❌ 1.14x slower) | 10.76 us (✅ 1.00x slower) |
| read | 1.43 us (✅ 1.00x) | 9.26 us (❌ 6.49x slower) | 2.13 us (❌ 1.49x slower) | 8.01 us (❌ 5.62x slower) |
See also benchmark results from https://github.com/djkoloski/rust_serialization_benchmark (in draft until 0.2 release).
Schema-based serialization.
Alkahest uses data schemas called [Formula]s to serialize and deserialize data.
Thus controlling data layout independently from data types that are serialized
or deserialized.
Support wide variety of formulas.
Integers, floats, booleans, tuples, arrays, slices, strings and
user-defined formulas with custom data layout using derive macro
that works for structs and enums of any complexity and supports generics.
Zero-overhead serialization of sequences.
Alkahest support serializing iterators directly into slice formulas.
No more allocation of a [Vec] to serialize and drop immediately.
Lazy deserialization.
Alkahest provides [Lazy<F>][Lazy] type to deserialize any formula F lazily.
[Lazy] can be used later to perform actual deserialization.\
[Lazy<[F]>][Lazy] can also produce iterator that deserializes elements on demand.\
Laziness is controlled on type level and can be applied to any element
of a larger formula.
Infallible serialization.
Given large enough or growing buffer any value that implements [Serialize]
can be serialized without error.
No more unnecessary unwraps or puzzles "what to do if serialization fails?".
The only error condition for serialization is "data doesn't fit".
Alkahest separates data schema definition (aka [Formula]) from
serialization and deserialization code.
Doing so, this library provides better guarantees for cases
when serializable data type and deserializable data type
are different.
It also supports serializing from iterators instead of collections
and deserialization into lazy wrappers that defers costly process
and may omit it entirely if value is never accessed.
User controls laziness on type level by choosing appropriate [Deserialize] impls.
For instance deserializing into [Vec<T>][Vec] is eager because
[Vec<T>][Vec] is constructed with all T instances
and memory allocated for them.
While [DeIter] obtained from [Lazy<[F]>][Lazy]
implements [Iterator] and deserializes
elements in Iterator::next and other methods.
And provides constant-time random access to any element for sized
formulas.
Flexibility comes at cost of using only byte slices for serialization and deserialization. And larger footprint of serialized data than some other binary formats.
Question about support of dense data packing is open. It may be desireable to control on type level.
The API is designed with following principles: Any value can be serialized successfully given large enough buffer. Data can't cause panic, incorrect implementation of a trait can.
There is zero unsafe code in the library on any code it generates.
No UB is possible given that std is not unsound.
No data schemas stays the same. New fields and variants are added, others are deprecated and removed.
There's set of rules that ensures forward compatibility between formulas. And another set or rules for backward compatibility.
Verification of compatibility is not implemented yet.
Forward compatibility is an ability to deserialize data that was serialized into newer formulas.
TODO: List all rules
Backward compatibility is an ability to deserialize data that was serialized into older formulas.
TODO: List all rules
The crate works using three fundamental traits.
[Formula], [Serialize] and [Deserialize].
There's also supporting trait - [BareFormula].
Alkahest provides derive macros for [Formula], [Serialize] and [Deserialize].
[Formula] trait is used to allow types to serve as data schemas.
Any value serialized into given formula should be deserializable from the same
formula. Sharing only [Formula] type allows modules and crates
easily communicate.
[Formula] dictates binary data layout and it must be platform-independent.
Potentially [Formula] types can be generated from separate files,
opening possibility for cross-language communication.
[Formula] is implemented for a number of types out-of-the-box.
Primitive types like [bool], integers and floating point types all implement [Formula].
This excludes [isize] and [usize].
In their place there's [FixedUsize] and [FixedIsize] types provided,
whose size is controlled by a feature-flag.
!Caveat!:
Sizes and addresses are serialized as [FixedUsize].
Truncating [usize] value if it was too large.
This may result in broken data generated and panic in debug.
Increase size of the [FixedUsize] if you encounter this.
It is also implemented for tuples, array and slice, [Option] and [Vec] (the later requires "alloc" feature).
The easiest way to define a new formula is to derive [Formula] trait for a struct or an enum.
Generics are supported, but may require complex bounds specified in attributes for
[Serialize] and [Deserialize] derive macros.
The only constrain is that all fields must implement [Formula].
[Serialize<F>][Serialize] trait is used to implement serialization
according to a specific formula F.
Serialization writes to mutable bytes slice and should not
perform any dynamic allocations.
Binary result of any type serialized into a formula must follow it.
At the end, if a stream of primitives serialized is the same,
binary result should be the same.
Types may be serializable into different formulas producing
different binary result.
[Serialize] is implemented for many types.
Most notably there's implementation T: Serialize<T>
and &T: Serialize<T> for all primitives T (except [usize] and [isize]).
Another important implementation is
Serialize<F> for I where I: IntoIterator, I::Item: Serialize<F>,
allowing serializing into slice directly from both iterators and collections.
Serialization with formula [Ref<F>][Ref] uses serialization into formula F
and then stores relative address and size. No dynamic allocations is required.
Deriving [Serialize] for a type will generate [Serialize] implementation,
formula is specified in attribute #[alkahest(FormulaRef)] or
#[alkahest(serialize(FormulaRef))]. FormulaRef is typically a type.
When generics are used it also contains generic parameters and bounds.
If formula is not specified - Self is assumed.
[Formula] should be derived for the type as well.
It is in-advised to derive [Serialize] for formulas with
manual [Formula] implementation,
[Serialize] derive macro generates code that uses non-public items
generated by [Formula] derive macro.
So either both should have manual implementation or both derived.
For structures [Serialize] derive macro requires that all fields
are present on both [Serialize] and [Formula] structure and has the same
order (trivially if this is the same structure).
For enums [Serialize] derive macro checks that for each variant there
exists variant on [Formula] enum.
Variants content is compared similar to structs.
Serialization inserts variant ID and serializes variant as struct.
The size of variants may vary. Padding is inserted by outer value serialization
if necessary.
[Serialize] can be derived for structure where [Formula] is an enum.
In this case variant should be specified using
#[alkahest(@variant_ident)] or #[alkahest(serialize(@variant_ident))]
and then [Serialize] derive macro will produce serialization code that works
as if this variant was a struct [Formula],
except that variant's ID will be serialized before fields.
[Serialize] can be derived for enum only if [Formula] is enum as well.
Serializable enum may omit some (or all) variants from [Formula].
It may not have variants missing in [Formula].
Each variant then follows rules for structures.
For convenience [Infallible] implements [Serialize] for enum formulas.
[Deserialize<'de, F>][Deserialize] trait is used to implement deserialization
according to a specific formula.
Deserialization reads from bytes slice constructs deserialized value.
Deserialization should not perform dynamic allocations except those
that required to construct and initialize deserialized value.
E.g. it is allowed to allocate when [Vec<T>] is produced if non-zero
number of T values are deserialized. It should not over-allocate.
Similar to [Serialize] alkahest provides a number of out-of-the-box
implementations of [Deserialize] trait.
[From<T>][From] types can be deserialized from primitive formula T.
Values that can be deserialized from formula F
can also deserialize from [Ref<F>][Ref], it reads address and length
and proceeds with formula F.
[Vec<T>][Vec] may deserialize from slice formula.
[Deserialize<'de, [F]>][Deserialize] is implemented for
[Lazy<'de, [F]>][Lazy] type that may produce
[DeIter<'de, F, T>][DeIter] - an [Iterator] that
lazily deserialize elements of type T: Deserialize<'de, F>.
[DeIter] is cloneable, can be iterated from both ends and
skips elements in constant time for sized formulas.
Deriving [Deserialize] for a type will generate [Deserialize] implementation,
formula is specified in attribute #[alkahest(FormulaRef)] or
#[alkahest(deserialize(<formula-ref>))]. <formula-ref> is typically a type.
When generics are used it also contains generic parameters and bounds.
If formula is not specified - Self is assumed.
[Formula] should be derived for the type as well.
It is in-advised to derive [Deserialize] for formulas with
manual [Formula] implementation,
[Deserialize] derive macro generates code that uses non-public items
generated by [Formula] derive macro.
So either both should have manual implementation or both derived.
serdeAlkahest is cool but [serde] is almost universally used, and for good reasons.
While designing a [Formula] it may be desireable to include existing type
that supports serialization serde, especially if it comes from another crate.
This crate provides [Bincode] and [Bincoded<T>][Bincoded] formulas to cover this.
Anything with [serde::Serialize] implementation can be serialized into [Bincode]
formula, naturally it will be serialized using bincode crate.
[Bincoded<T>][Bincoded] is a restricted version of Bincode that works only for T.
```rust // This requires two default features - "alloc" and "derive".
fn main() { use alkahest::{Formula, Serialize, Deserialize, serializetovec, deserialize};
// Define simple formula. Make it self-serializable.
#[derive(Clone, Debug, PartialEq, Eq, Formula, Serialize, Deserialize)]
struct MyDataType {
a: u32,
b: Vec
// Prepare data to serialize. let value = MyDataType { a: 1, b: vec![2, 3], };
// Use infallible serialization to Vec.
let mut data = Vec::new();
// Note that this value can be serialized by reference.
// This is default behavior for Serialized derive macro.
// Some types required ownership transfer for serialization.
// Notable example is iterators.
let size = serializetovec::
let (de, desize) = deserialize::
fn main() {} ```
Licensed under either of
at your option.
Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.