apalis

Simple, extensible multithreaded background job and messages processing library for Rust

Features

Simple and predictable job handling model.
Jobs handlers with a macro free API.
Take full advantage of the [tower] ecosystem of middleware, services, and utilities.
Runtime agnostic - Use tokio, smol etc.
Optional Web interface to help you manage your jobs.

apalis job processing is powered by [tower::Service] which means you have access to the [tower] middleware.

apalis has support for:

| Source | Crate | Example | | ----------- | ----------- | --------- | | Cron Jobs | | | Redis | | | Sqlite | | | Postgres | | | MySQL | | | Amqp | | | From Scratch | | |

Getting Started

To get started, just add to Cargo.toml

toml [dependencies] apalis = { version = "0.4", features = ["redis"] }

Usage

```rust use apalis::prelude::*; use apalis::redis::RedisStorage; use serde::{Deserialize, Serialize}; use anyhow::Result;

[derive(Debug, Deserialize, Serialize)]

struct Email { to: String, }

impl Job for Email { const NAME: &'static str = "apalis::Email"; }

/// A function that will be converted into a service. /// The following signatures are accepted /// rust /// async fn job(email: Email, ctx: JobContext) /// async fn job(email: Email, ctx: JobContext) -> anyhow::Result<anyhow::Error> /// async fn job(email: Email, ctx: JobContext) -> Result<(), my::Error> /// async fn job(email: Email, ctx: JobContext) -> primitive //eg str /// async fn job(email: Email, ctx: JobContext) -> impl IntoJobResponse /// async fn email_service(job: Email, ctx: JobContext) { info!("Do something"); }

[tokio::main]

async fn main() -> Result<()> { std::env::setvar("RUSTLOG", "debug"); envlogger::init(); let redis = std::env::var("REDISURL").expect("Missing env variable REDISURL"); let storage = RedisStorage::new(redis).await?; Monitor::new() .registerwithcount(2, move |index| { WorkerBuilder::new(format!("email-worker-{index}")) .withstorage(storage.clone()) .buildfn(emailservice) }) .run() .await }

```

Then

```rust //This can be in another part of the program or another application eg a http server async fn produceroutejobs(storage: &RedisStorage) -> Result<()> { let mut storage = storage.clone(); storage .push(Email { to: "test@example.com".to_string(), }) .await?; }

```

Feature flags

tracing (enabled by default) — Support Tracing 👀
redis — Include redis storage
postgres — Include Postgres storage
sqlite — Include SQlite storage
mysql — Include MySql storage
cron — Include cron job processing
sentry — Support for Sentry exception and performance monitoring
prometheus — Support Prometheus metrics
retry — Support direct retrying jobs
timeout — Support timeouts on jobs
limit — 💪 Limit the amount of jobs
filter — Support filtering jobs based on a predicate
extensions — Add a global extensions to jobs

Storage Comparison

Since we provide a few storage solutions, here is a table comparing them:

| Feature | Redis | Sqlite | Postgres | Sled | Mysql | Mongo | Cron | | :-------------- | :---: | :----: | :------: | :--: | :---: | :---: | :--: | | Scheduled jobs | ✓ | ✓ | ✓ | x | ✓ | x | ✓ | | Retry jobs | ✓ | ✓ | ✓ | x | ✓ | x | ✓ | | Persistence | ✓ | ✓ | ✓ | x | ✓ | x | BYO | | Rerun Dead jobs | ✓ | ✓ | ✓ | x | ✓ | x | x |

How apalis works

```mermaid sequenceDiagram participant App participant Worker participant Producer

App->>+Producer: Add job to queue
Producer-->>+Worker: Job data
Worker->>+Producer: Update job status to 'running' via Layer
Producer-->>-Worker: Confirmation
Worker->>+App: Notify job started via Layer
loop job execution
    Worker-->>-App: Report job progress via Layer
end
Worker->>+Producer: Update job status to 'completed' via Layer

```

Web UI

If you are running apalis Board, you can easily manage your jobs. See a working rest API example here

Thanks to

[tower] - Tower is a library of modular and reusable components for building robust networking clients and servers.
redis-rs - Redis library for rust
sqlx - The Rust SQL Toolkit

Roadmap

v 1.0 - [ ] Refactor the crates structure - [ ] Mocking utilities - [ ] Support for SurrealDB and Mongo - [ ] Lock free for Postgres - [ ] Add more utility layers - [ ] Use extractors in job fn structure - [ ] Polish up documentation - [ ] Improve and standardize apalis Board - [ ] Benchmarks

v 0.4

[x] Move from actor based to layer based processing
[x] Graceful Shutdown
[x] Allow other types of executors apart from Tokio
[x] Mock/Test Worker
[x] Improve monitoring
[x] Add job progress via layer
[x] Add more sources

v 0.3

[x] Standardize API (Storage, Worker, Data, Middleware, Context )
[x] Introduce SQL
[x] Implement layers for Sentry and Tracing.
[x] Improve documentation
[x] Organized modules and features.
[x] Basic Web API Interface
[x] Sql Examples
[x] Sqlx migrations

v 0.2

[x] Redis Example
[x] Actix Web Example

Resources

Background job processing with rust using actix and redis

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Authors

Njuguna Mureithi - Initial work - Njuguna Mureithi

See also the list of contributors who participated in this project.

License

This project is licensed under the MIT License - see the LICENSE.md file for details