Inspired by https://github.com/redboxllc/scuttle, https://github.com/joho/godotenv, and https://github.com/kubernetes/enhancements/issues/753, among others. This program is meant to be the entrypoint for the "main" container in a Pod that also contains sidecar containers. Proa is a wrapper around the main process in the main container. It waits for the sidecars to be ready before starting the main program, and it shuts down the sidecars when the main process exits so the whole Pod can exit gracefully, as in the case of a Job.
Briefly, it does this:
pkill would.If it encounters errors during shutdown, it logs each error, but it exits with the same exit code as the wrapped process.
If you like, just copy job.yaml and modify it for your use. The Job has a sidecar, simulated by a Python
script, that must be ready before the main process starts. We simulate a sidecar that starts slowly by sleeping for 30 seconds
before starting the Python HTTP server. Proa uses Kubernetes' knowledge about the readiness of the sidecar container. That means
the sidecars must each provide a readinessProbe, and the Pod's serviceAccount needs permission to read and watch the Pod it's
running in.
Or if you prefer, follow this step-by-step guide:
1. Build a container image that has both your main application and the proa executable. The easiest way to do this is probably
to use a multi-stage Dockerfile to compile proa and COPY it into your final image. See Dockerfile
for an example.
1. Create a ServiceAccount for your Job to use.
1. Create a Role and RoleBinding giving the service account permission to get, watch, and list the pods in its own
namespace.
1. Modify the Job spec.template.spec.serviceAccountName to refer to that service account.
1. Modify the Job and ensure that the spec.template.spec.containers entry for every sidecar has a readinessProbe. (It doesn't
matter if the main container has a readiness probe; proa will ignore it.)
1. Change the entrypoint (command and/or args) of the main container to call proa.
- Pass flags to tell proa how to shut down your sidecars. This will usually be --shutdown-http-get=URL or
--shutdown-http-post=URL. Those flags can be repeated multiple times.
- Pass the separator string --, followed by the path to the main program and all its arguments.
1. Optionally add a RUST_LOG environment variable to the main container to control proa's logging verbosity.
When it's time to shut down, proa can end the processes in your sidecars by sending SIGTERM, but it's probably not what you want. Most processes that receive SIGTERM will exit with status 143, or some other nonzero value. Kubernetes will interpret that as a container failure, and it will restart or recreate the Pod to try again.
If you're sure you want to use this, compile the program with feature kill, and also make sure your Pod meets these requirements:
- Need to shareProcessNamespace so proa can stop the sidecars, and either
- the main container with proa needs to run as UID 0 (not recommended)
- all containers need to run as the same UID.
- Don't use hostPID, or chaos will result as it tries to kill every process on the node.
It's a program to manage sidecars, but sidecar is a motorcycle metaphor, and Kubernetes is all about nautical memes. A proa is a sailboat with an outrigger, which is sort of like a sidecar on a motorcycle.
Requirements: - Use nix and direnv, or install the tools manually. See flake.nix for the list of tools you'll need. - Docker or equivalent.
kind create cluster to start a tiny Kubernetes cluster in your local Docker.skaffold dev to start the compile-build-deploy loop.Every time you save a file, skaffold will rebuild and redeploy, then show you output from the containers in the Pod.