Async persistent background task processing for Rust applications with Tokio. Queue asynchronous tasks to be processed by workers. It's designed to be easy to use and horizontally scalable. It uses Postgres as a storage backend and can also be extended to support other types of storage.
High-level overview of how Backie works: - Client puts tasks on a queue - Server starts a multiple workers per queue - Worker pulls tasks off the queue and starts processing them - Tasks are processed concurrently by multiple workers
Backie started as a fork of fang crate, but quickly diverged significantly in its implementation.
Here are some of the Backie's key features:
Cargo.tomltoml
[dependencies]
backie = "0.1"
If you are not already using, you will also want to include the following dependencies for defining your tasks:
toml
[dependencies]
async-trait = "0.1"
anyhow = "1"
serde = { version = "1.0", features = ["derive"] }
diesel = { version = "2.0", features = ["postgres", "serde_json", "chrono", "uuid"] }
diesel-async = { version = "0.2", features = ["postgres", "bb8"] }
Those dependencies are required to use the #[async_trait] and #[derive(Serialize, Deserialize)] attributes
in your task definitions and to connect to the Postgres database.
Supports rustc 1.68+
backie_tasks table in the Postgres database. The migration can be found in the migrations directory.The [BackgroundTask] trait is used to define a task. You must implement this trait for all
tasks you want to execute.
One important thing to note is the use of the attribute [BackgroundTask::TASK_NAME] which must be unique for
the whole application. This attribute is critical for reconstructing the task back from the database.
The [BackgroundTask::AppData] can be used to argument the task with your application specific contextual information.
This is useful for example to pass a database connection pool to the task or other application configuration.
The [BackgroundTask::run] method is where you define the behaviour of your background task execution. This method
will be called by the task queue workers.
```rust use asynctrait::asynctrait; use backie::{BackgroundTask, CurrentTask}; use serde::{Deserialize, Serialize};
pub struct MyTask { info: String, }
impl BackgroundTask for MyTask { const TASKNAME: &'static str = "mytaskuniquename"; type AppData = ();
async fn run(&self, task: CurrentTask, context: Self::AppData) -> Result<(), anyhow::Error> {
// Do something
Ok(())
}
} ```
First, we need to create a [TaskStore] trait instance. This is the object responsible for storing and retrieving
tasks from a database. Backie currently only supports Postgres as a storage backend via the provided
[PgTaskStore]. You can implement other storage backends by implementing the [TaskStore] trait.
Then, we can use the task_store to start a worker pool using the [WorkerPool]. The [WorkerPool] is responsible
for starting the workers and managing their lifecycle.
A full example of starting a worker pool can be found in the examples directory.
After stating the workers, we get an instance of [Queue] which we can use to enqueue tasks. It is also possible
to directly create a [Queue] instance from with a [TaskStore] instance.
This will enqueue the task and whenever a worker is available it will start processing. Workers don't need to be started before enqueuing tasks. Workers don't need to be in the same process as the queue as long as the workers have access to the same underlying storage system. This enables horizontal scaling of the workers.
git checkout -b my-new-feature)git commit -am 'Add some feature')git push origin my-new-feature)I would like to thank the authors of the Fang and background_job crates which were the main inspiration for this project.