Subliminal is a versatile task management API that enables seamless storage, retrieval, and execution of generic Task objects across a robust microservice architecture built on Google Cloud Platform (GCP). Developed primarily as a learning endeavour, this project showcases a spectrum of advanced technologies, including:
At its core, Subliminal functions as a comprehensive CRUD (Create, Read, Update, Delete) application, designed to facilitate various operations on Task objects stored within a database. Moreover, it introduces a twist by integrating the capability to dispatch these records to one or more executor services. These executors, whether hosted in the cloud or locally, execute tasks while asynchronously updating the API for seamless datastore reflection.
Subliminal's architecture is built upon the foundation of the subliminal crate, accessible via crates.io. This crate encompasses key utilities necessary for constructing each pivotal service within the broader Subliminal ecosystem:
The architecture's interaction is represented in the following diagram:
mermaid
flowchart LR
A[API] <--> B[Datastore]
A <--> C[Task Message Queue]
C --> D[Executor]
D --> E((Worker))
D --> F((Worker))
To enable efficient communication between the API and certain services, Subliminal employs gRPC communication. Base protocol buffers reside in the designated GitHub repository, triggering automated stub generation for Rust using prost through a GitHub action. The subliminal project then pulls the desired sublimina-protos-rust version as a git submodule, enabling seamless service definition implementation.
The API service interfaces synchronously with the datastore service, functioning similarly to a conventional CRUD application. It offers full CRUD functionality for task request and execution records, leveraging various gRPC definitions. Further insights into record types, database implementation specifics, and instance setup can be found in the subliminal-datstore repository.
Although the message queue is not a distinct service within Subliminal, it encapsulates Google's PubSub services to be used throughout the system. It is manifested as a MessageQueue, streamlining the enqueueing of task executions and updates asynchronously. Built using the MessageQueueBuilder, it is pivotal in decoupling the executor from other services.
A distinctive feature of Subliminal is its execution nodes, which distinguish it from traditional CRUD applications. These nodes are responsible for dequeuing tasks from the MessageQueue, deserializing them, and executing them based on their parameters. The process involves:
Task trait.Task trait for the defined struct, detailing the task executionExecutionNodeBuilder to construct an execution node with designated consumers.Consumers continually poll the MessageQueue asynchronously, deserializing data from task requests. The dispatcher distributes tasks to available workers, while workers execute tasks, updating the dispatcher for further processing.