s3d is an S3 daemon for the Edge written in Rust

The goal of s3d is to provide a daemon for edge platforms (e.g. factory servers 🏭 planes 🛩 ships 🚢 cars 🏎 laptops 💻 mobile devices 📱 wearables ⌚ ...) that connect to a central object storage (aka Hub).

Since edge applications operate in subideal environments (high latency network, disconnections, hardware failures, eavesdropping, ...), s3d aims to make the applications run smoothly while it handles the flows of data, security, networking, local capacity, metadata caching, write queueing, etc.

Worth mentioning that the choice of the Rust language is a natural fit for the edge systems, as it is a modern language with a focus on functionality, safety and performance. s3d builds with the rust toolchain into a single binary that loads a yaml config file, which makes it easy to set up and configure in standalone linux or containerized environments like Kubernetes.

Info

This project is still 🛸🛸🛸 Experimental 🚀🚀🚀

This means it's a great time to affect its direction!

If you find it interesting or want to contribute, please feel free to communicate using these options:

• Discord Chat - join to discussion channels
• Github Issues - for bugs/questions/disccussions/suggestions
• Github PR's - add features/fixes/improvements (prefer to have an issue open first)
• Github Projects - track roadmap/progress
• License - Apache-2.0

Docs

• Commands
• Config
• Examples

Quick start

To start using s3d you need to install it, run it, and configure your data workflows.

The following steps illustrate a basic flow of s3d usage:

```sh

Install the s3d binary using Rust toolchain

cargo install s3d

Sets up the configuration, hub connection, and local storage

s3d init

Run the daemon to serve S3 clients can connect to read and write objects

Start and stop will spawn/kill the daemon in the background

s3d run s3d start s3d stop

Fetch reads metadata from the hub and stores locally

This includes the list of objects and their metadata, but excludes objects contents

s3d fetch [bucket/prefix]

Pull is like fetch but includes objects contents.

s3d pull [bucket/prefix]

Simple access to objects from the CLI

s3d get bucket/key > file s3d put bucket/key < file s3d ls [bucket/prefix]

Show local changes not pushed to the hub

s3d diff [bucket/prefix]

Show bucket status and local store stats (objects, sizes, etc).

s3d status [bucket/prefix]

Push bucket changes (merge by last modified time).

s3d push [bucket/prefix]

Remove objects from local store based on age

s3d prune [bucket/prefix] ```