playbook-rs v0.1.5

YAML driven Docker DevOps

Requirements

• Python3.x
• (Optional) nvidia-docker2 (Make sure you can nvidia-smi and docker images)

Installation

Run the following with sh cargo +nightly install playbook

Usage example

sh playbook say_hi.yml

Features

• Designed to work with nvidia-docker2
• Runs whitelisted steps in sequential manner (notice the whitelist and #[playbook(say_hi)])
• Context deduction: each step can run in a different docker image or on host
• Full access to host network services
• Simple step function siginature awesome_func(ctx) - easy to extend
• Minimal command line arguments: playbook some.yml
• Colorful logging for readability

To show you what I mean...

A task to be operationalized

```python

[playbook(say_hi)]

def say_hi(ctx): print("{whoami}: Hi!".format(**ctx)) ```

Description of the resources needed in a YAML file

```yml

whitelist: - src: sayhi.py whoami: Host steps: - name: Running in a container action: sayhi docker: image: aleozlx/playbook-hello dockeroverrides: whoami: Container - name: Running on host action: sayhi

```

It works!

Notice how the playbook driver spawns containers when necessary, and supply appropriate context variables to the task in its native data structure. ``` $ playbook sayhi.yml [2018-11-19 10:51:15] INFO Step 1: Running in a container [2018-11-19 10:51:15] INFO Entering Docker: aleozlx/playbook-hello [2018-11-19 10:51:15] INFO ["docker", "run", "--rm", "-t", "--net=host", "-v", "/home/alex/Code/playbook-rs/examples/hello:/home/alex/current-ro", "-w", "/home/alex/current-ro", "aleozlx/playbook-hello", "/usr/bin/env", "playbook", "--docker-step=0", "sayhi.yml"] [2018-11-19 16:51:15] INFO Step 1: Running in a container == Context ======================

# ctx(sayhi@sayhi.py) =

docker-step: "0" playbook: sayhi.yml whoami: Container name: Running in a container action: sayhi == EOF ========================== == Output ======================= Container: Hi! == EOF ========================== [2018-11-19 10:51:15] INFO Step 2: Running on host == Context ======================

# ctx(sayhi@sayhi.py) =

name: Running on host whoami: Host action: sayhi playbook: sayhi.yml == EOF ========================== == Output ======================= Host: Hi! == EOF ========================== ```

How to add steps?

1. Add a function def something(ctx). Current execution context is in ctx as dict. TODO ~~Keys are proxied to attributes to save a lot of brackets and quotes.~~

python def something(ctx): print(ctx['my_var'])

1. Whitelist your step function using #[playbook(...)] and add the source file path to whitelist variable.
2. Add an entry to your YAML file in steps where action is the step function name:

yml steps: - name: Some message here action: something my_var: goes_to_ctx

Context deduction rules

docker overrides > step context > global context

How to specify docker environment?

You may add a default docker environment. And use docker_overrides to change context variables when docker is in use. yml docker: image: aleozlx/tkstack2:latest runtime: nvidia gui: False ports: - 6006:6006 volumes: - /tmp:/workspace/build - /mnt/datasets:/workspace/datasets docker_overrides: storage: /workspace/datasets steps: - name: Some message here action: something storage: /mnt/datasets

Or override the docker environment completely per step yml docker: # ... steps: - name: Some message here action: something storage: /mnt/datasets docker: image: aleozlx/tkstack2:latest runtime: nvidia volumes: - /tmp:/workspace/build docker_overrides: storage: /workspace/datasets

Or use the host yml docker: # ... steps: - name: Some message here action: something storage: /mnt/datasets docker: null

Note: When a docker environment is present, the playbook starts docker accordingly and resumes itself inside docker to reuse many of the playbooks features, so that context deduction have a consistent behavior.

Security assumptions

Host file system: volumes specified in your playbook will be mounted read-only by default, which you may override. The current working directory is mounted read-only automatically. Playbook assumes that you use a docker image that uses non-root user whose uid:gid hopefully maps to you on host system.

Network: network services inside docker are not isolated from host in non-interactive mode to provide convenient access to host databases etc. Playbook assumes whatever you are operationalizing is trusted and that your host should have a proper set of INPUT rules, and that services inside docker should be protected by an independent firewall if necessary.

X11: the recommended docker image does intend to provide isolated X11 access by creating non-root user that presumably maps to you on host and your X authentication files are naturally mounted with proper permissions already in place.

Playbook itself: the playbook itself is obviously a very capable shell program. It is based on a simple whitelist to establish its symbol namespace and allow any actions to be executed. Predictable behaviors of the program relies on the playbook binaries share the same version both on the host and in the container.