Filespooler: CLI & Library for Sequential, Distributed, POSIX-style job queue processing

build docs

Quick Links

Introduction

Filespooler is a Unix-style tool that facilitates local or remote command execution, complete with stdin capture, with easy integration with various tools. I will decode what that means below. For now, here's a brief Filespooler feature list:

Filespooler consists of a command-line tool (fspl) for interacting with queues. It also consists of a Rust library that is used by fspl. main.rs for fspl is just a few lines long.

Use Cases

Imagine for a moment that you want to send incremental backups from one machine to your backup server. You might run something like this:

tar --incremental -cSpf - ... | ssh backupsvr tar -xvSpf - -C /backups

That will work when all is good. But when the network between the two machines drops, now what? Probably data loss. What we want is a way to reliably execute things, in order, with reordering in case of out-of-order data. This turns out to be useful in many situations: Git repository syncing, backups, etc.

Now, say you do something like this:

tar --incremental -cSpf - ... | fspl prepare -s ~/statefile -i - > ~/syncedpath/fspl-`uuid`.fspl

At this point, a tool like Syncthing or Dropbox will sync this syncedpath to the ~/queue/jobs/ directory under the queue on the backup server. Now you can run this (from cron, systemd, etc) on the backup serer:

fspl queue-process -q ~/queue tar -- -xvSpf - -C /backups

Boom. Done.

queue-process will (by default) delete jobs that finish successfully. It will keep track of which jobs have been completed and process them in order.

Copyright

Copyright (C) 2022 John Goerzen <jgoerzen@complete.org>

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.