acid-store

acid-store is a library for secure, deduplicated, transactional, and verifiable data storage.

This library provides high-level abstractions for data storage over a number of storage backends. The goal is to decouple how you access your data from where you store it. You can access your data as a virtual file system, an object store, or a persistent collection, regardless of where it's stored. Out of the box, this library supports the local file system, SQLite, Redis, and Amazon S3 as storage backends. Storage backends are easy to implement, and this library builds on top of them to provide features like encryption, compression, deduplication, locking, and atomic transactions.

For details and examples, see the documentation.

⚠️ This project is still immature and needs more testing. Testers are always appreciated, but please remember to back up your data! Also keep in mind that this code has not been audited for security. All the usual disclaimers apply.

Features

• Optional encryption of all data and metadata using XChaCha20-Poly1305 and Argon2, powered by libsodium
• Optional compression using DEFLATE, LZMA, OR LZ4
• Content-based deduplication using the ZPAQ chunking algorithm
• Integrity checking of data and metadata using checksums and (if encryption is enabled) AEAD
• Transactional operations providing atomicity, consistency, isolation, and durability (ACID)
• Copy-on-write semantics
• New storage backends are easy to implement

This library provides the following abstractions for data storage: - An object store which maps keys to seekable binary blobs - A virtual file system which can import and export files to the local OS file system - A persistent, heterogeneous, map-like collection - An object store with support for content versioning

This library provides the following storage backends out of the box: - Local file system directory - SQLite - Redis - Amazon S3 - In-Memory