Airone is a library inspired from Aral Balkan's JSDB principles applied to a Rust library.
The name has nothing to do with "air" or "one", it simply comes from the Italian word "Airone", which means "Heron".
Hence, it has to be read [aiˈrone].
Table of Contents
[TOC]
Airone is aimed at helping small-tech to be produced, which is a way of building technology antithetical to the Silicon Valley model. Small-tech features close-to-zero data collection, a free and open source structure, no lust to scale exponentially, no desire in tracking people's behaviour while crunching tons of data and, overall, it has a desire to keep things simple and human.
Airone is developed to be used in situations where all of these prerequisites apply: - No big-data: the whole dataset should fit in memory - the dataset has to be persisted to disk - after some data is modified, the change needs to be written to disk in a fast way - we can slow down the program start time without any relevant consequence - no nested objects
These limits can be a problem for some usage scenarios. Existing databases add all sort of optimizations, caching mechanisms and are very complex, in order to be able to deal with such big amount of data.
However, when the above-mentioned prerequisites apply, we can leverage these limits to simplify the source code, making it more maintanable and getting rid of all the bloatware.
Moreover, limiting object nesting makes it compatible with CSV style files, ensuring you can manipulate data with standard UNIX tools such as grep and awk.
Here are two examples of good usage scenarios: - small web server: data may change fast and can entirely fit into memory. The length of the timespan neeeded to start the server is not important, a long as the server is performant while it's running. - an offline GUI/TUI program: you can store modifications of some data in a very fast way thanks to airone's append-only file architecture, so the interface freeze during the saving process is barely noticeable.
This library persists a list of structs from memory to disk. Every change in the data in memory is saved to disk automatically in a fast way, avoiding the full dump of the whole list.
As saving a long list of objects to a file each time a change happens may be cumbersome, airone writes each incremental change to an append-only file. This incremental transaction log is applied at the next restart to compact the base dump file; in this way, the heavy operation of writing the full dump of data from memory is executed only once at startup. After that, each change is written in a fast way to the append-only log and which will be compacted at the next instantiation.
This project is still a Work in Progress. It has been started only out of curiosity attempting to replicate JSDB behaviour with the bare minimum number of dependencies I could achieve (only one). It has unit tests, but it has not NOT been used in production so far.
OS: Windows and Mac are not supported nor endorsed and airone has not been tested on them. Beware that WSL is a compatibility layer and may unexpectedly break too. I encourage you to switch to an actually freedom-and-privacy respecting operating system such as GNU/Linux, where this library has been tested on more. Head to https://www.youtube.com/watch?v=Ag1AKIl_2GM for more information
Currently, it has been tested only with primitive types and Option<T> where T is a primitive type.
Data is written to two files, depending of the phase of execution.
The base dump file contains the full dump of data in memory. This is recreated whenever the program starts by using the old dump data file as a base point and applying each incremental change to it. Afterwards, the data is saved to the a new dump file and the old transaction log is deleted.
From this point, the program continues its execution saving changes to a new transaction log.
Both files follow this character convention:
- the \n character is used as a newline (no carriage return)
- the \t character is used as a field separator.
The base dump file is saved using a standard UNIX-style CSV text file.
Let's use this struct as an example:
rust
struct ExampleStruct
{
field1: String,
field2: f64,
field3: i32
}
Given a list of two ExampleStruct elements, the base dump file could look like this:
plain
abc 3.15 57
text2 47.89 -227
When the program is running, changes are written to the append-only transaction log. Each line of this file is formatted as it follows, depending on the applied operation.
The first letter A sets the operation to Add. The new object fields are serialized as in the base dump file, by writing each field's value in the proper order.
Structure:
plain
A field1 field2 field3
Example:
plain
A abc 3.15 57
The first letter D sets the operation to Delete. After that, it expects the index of the element to remove.
Structure:
plain
D index_of_element_to_remove
Example:
plain
D 2
The first letter E sets the operation to Edit. After that field, adhere to the following structure.
Structure:
plain
E variable_type index_of_element field_to_change new_value
Example
plain
E f64 0 field2 -57.5
Here are some basic instructions.
See Documentation for the complete generated crate documentation.
Add this line to your dependencies section of the Cargo.toml file.
toml
airone = "0.1.2"
If you're planning to use it in a Rocket webserver, you can enable airone optional "rocket" feature.
toml
airone = {version: "0.1.2", features=["rocket"]}
The core lies in the [airone_db!] macro. Pass a struct named as you wish to it (let's take Foo as an example) and the macro
will define a new struct named $structname+AironeListProxy (which expands to FooAironeListProxy in this example).
It acts as a proxy between you and the underlying list of data, automatically persisting
changes when they happen and providing methods to interact with them.
Given the example structure: ```
use airone::AironeError; use airone::Database;
// This generates a FooAironeListProxy struct
// to interact with the data while saving any change to disk.
airone_db!(
struct Foo
{
pub field1: f64,
pub field2: String,
field3: Option
{ // Open the database let mut db = FooAironeListProxy::new(); // Add one element using // a method from the Database trait db.push( Foo{ field1: 0.5, field2: "Hello world".tostring(), field3: Some(-45) } ); // Change a field using the generated setter method db.setfield3(0, None); // The database is closed automatically here } { // Open again, check the modified data // has been correctly persisted. let mut db = FooAironeListProxy::new(); asserteq!( *db.getfield3(0).unwrap(), None ); }
```
The generated struct implements all methods from the [Database] trait,
plus getter and setters for each variable to change the element
at the specified index in the form of:
rust,ignore
fn set_$field_name(&mut self, index: usize, new_value: $field_type) -> Result<(), AironeError>
fn get_$field_name(&self, index: usize) -> Result<$field_type, AironeError>
fn set_bulk_$field_name(&mut self, indices: Vec<usize>, value: $field_type) -> Result<(), AironeError>
You can use the [Query] struct to make queries using dot notation, chaining them one after another.
```rust
use airone::Query; let mut db = QueryExampleAironeListProxy::new(); // Fill in data how you want here // … //
// Use brackets to create an inner scope // so that the &mut db reference we pass to // the query object will be given back // when the query object is released. { let mut q = Query::new(&mut db); // Can now use dot notation chaining operations q.filter( |e| { e.getmytext() == "Test string" } ) .delete().unwrap();
// Query goes out of scope
// and mutable borrow is released
// to the outer scope
}
// Do something else with db object
```
You can serialize and deserialize your custom types by implementing [LoadableValue] and [PersistableValue] traits on them.
The serialized string must be on a single line and must escape any \t, \r and \n character to ensure CSV style compatibility.
For most types, you can simply use Rust's format!() and parse() features.
This is NOT public domain, make sure to respect the license terms. You can find the license text in the COPYING file.
Copyright © 2022 Massimo Gismondi
This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU Affero General Public License along with this program. If not, see https://www.gnu.org/licenses/.