c8-backupA CLI tool to automate backup and restore of Camunda Platform 8 deployments.
⚠️ Experimental ⚠️
Features: - [x] List backups - [x] Create backups - [x] Restore backups - [ ] Dry runs
Components: - [x] Zeebe - [x] Operate - [ ] Tasklist - [ ] Optimize
Deployments: - [x] Remote [Camunda Platform 8 Helm] installation (running locally, connecting through the current kubernetes context). - [ ] Local [Camunda Platform 8 Helm] installation (running as a pod inside the cluster).
Warning This tool is experimental and not safe for use yet! Running it can result in complete dataloss.
Download a pre-built binary from the latest release or build from sources with
shell
cargo install c8-backup
Currently, this tool is meant to run locally. It will connect to your current kubernetes context and tries to find C8 components there.
The list command lists recent backups by status and also shows which is the most recent, usable, backup.
This means that the backup is successfully completed by all components.
```
$ c8-backup list
c8backup::list::list{}
c8backup::list::Zeebe{}
0ms INFO c8_backup::list 2 backups Completed: 1683214620, 1683214072, ...
c8backup::list::Operate{} 0ms INFO c8backup::list 2 backups Completed: 1683214620, 1683214072, ...
1060ms INFO c8backup::list The most recent usable backup is 1683214620 1060ms INFO c8backup::list This backup was created 8 minutes ago at 2023-05-04 15:37:00 UTC ```
The create command starts without any additional confirmation and will take the following steps:
1. Take a backup of Operate
2. Pause Zeebe exporting
3. Take a backup of exported Zeebe records
4. Take a backup of Zeebe
5. Resume Zeebe exporting
Resuming exporting is crucial and this command tries to resume exporting if any error occurs while taking a backup but you should manually confirm that exporting resumed, for example by following the log output.
```shell $ c8-backup create c8backup::create::create{} c8backup::create::trybackup{backupid=1683214620} c8backup::create::backupoperate{backupid=1683214620} c8backup::operate::takebackup{backupid=1683214620}
267ms INFO c8_backup::create Started backup
521ms INFO c8_backup::create Checking again in 5 seconds, state is InProgress
5870ms INFO c8_backup::create Checking again in 5 seconds, state is InProgress
11233ms INFO c8_backup::create Checking again in 5 seconds, state is Incomplete
16598ms INFO c8_backup::create Backup completed
c8_backup::zeebe::pause_exporting{}
c8_backup::create::backup_zeebe_export{backup_id=1683214620}
c8_backup::elasticsearch::take_snapshot{req=SnapshotRequest { indices: "zeebe-record*", feature_states: ["none"] }, name="camunda_zeebe_records_1683214620"}
c8_backup::create::backup_zeebe{backup_id=1683214620}
c8_backup::zeebe::take_backup{backup_id=1683214620}
280ms INFO c8_backup::create Started backup
831ms INFO c8_backup::create Checking again in 5 seconds, state is InProgress
6180ms INFO c8_backup::create Backup completed
c8_backup::zeebe::resume_exporting{}```
The restore commands starts without any additional confirmation and will take the following steps:
1. Find the latest backup from Zeebe and Operate
2. Stop Zeebe and Operate
3. Delete all indices from Elasticsearch
4. Restore Elasticsearch indices based on backups
5. Delete all Zeebe data
6. Restore Zeebe data based on backup
7. Start Zeebe and Operate
```shell $ c8-backup restore c8backup::restore::restore{} c8backup::restore::findnewestbackup{} 1573ms INFO c8_backup::restore Using backup 1683214620
c8backup::restore::shutdownapps{} 331ms INFO c8backup::restore Shut down ccs23-dev-zeebe-gateway 380ms INFO c8backup::restore Shut down ccs23-dev-operate 431ms INFO c8_backup::restore Shut down ccs23-dev-zeebe
c8backup::restore::deleteindices{} c8backup::elasticsearch::getall_indices{}
434ms INFO c8_backup::elasticsearch Deleted index operate-flownode-instance-8.2.0_
695ms INFO c8_backup::elasticsearch Deleted index operate-decision-instance-8.2.0_
966ms INFO c8_backup::elasticsearch Deleted index operate-list-view-8.1.0_
1263ms INFO c8_backup::elasticsearch Deleted index operate-import-position-8.2.0_
1587ms INFO c8_backup::elasticsearch Deleted index operate-user-1.2.0_
1907ms INFO c8_backup::elasticsearch Deleted index zeebe-record_deployment-distribution_8.2.3_2023-05-04
2223ms INFO c8_backup::elasticsearch Deleted index operate-operation-8.2.0_
2509ms INFO c8_backup::elasticsearch Deleted index zeebe-record_deployment_8.2.3_2023-05-04
2809ms INFO c8_backup::elasticsearch Deleted index operate-batch-operation-1.0.0_
3077ms INFO c8_backup::elasticsearch Deleted index operate-process-8.1.8_
3352ms INFO c8_backup::elasticsearch Deleted index operate-web-session-1.1.0_
3634ms INFO c8_backup::elasticsearch Deleted index operate-incident-8.2.0_
3938ms INFO c8_backup::elasticsearch Deleted index zeebe-record_variable_8.2.3_2023-05-04
4216ms INFO c8_backup::elasticsearch Deleted index zeebe-record_job_8.2.3_2023-05-04
4472ms INFO c8_backup::elasticsearch Deleted index operate-variable-8.2.0_
4796ms INFO c8_backup::elasticsearch Deleted index operate-migration-steps-repository-1.1.0_
5069ms INFO c8_backup::elasticsearch Deleted index operate-decision-1.0.0_
5324ms INFO c8_backup::elasticsearch Deleted index operate-event-8.1.0_
5624ms INFO c8_backup::elasticsearch Deleted index zeebe-record_process-instance-creation_8.2.3_2023-05-04
5939ms INFO c8_backup::elasticsearch Deleted index zeebe-record_process_8.2.3_2023-05-04
6234ms INFO c8_backup::elasticsearch Deleted index operate-metric-1.0.0_
6526ms INFO c8_backup::elasticsearch Deleted index operate-sequence-flow-8.2.0_
6844ms INFO c8_backup::elasticsearch Deleted index zeebe-record_process-instance_8.2.3_2023-05-04
7164ms INFO c8_backup::elasticsearch Deleted index operate-decision-requirements-1.0.0_
c8backup::restore::restoreindices{backup=Backup { id: 1683214620, snapshots: ["camundazeeberecords1683214620", "camundaoperate16832146208.2.3part1of6", "camundaoperate16832146208.2.3part2of6", "camundaoperate16832146208.2.3part3of6", "camundaoperate16832146208.2.3part4of6", "camundaoperate16832146208.2.3part5of6", "camundaoperate16832146208.2.3part6of6"] }} 12203ms INFO c8backup::restore Restored snapshot camundazeeberecords1683214620 14050ms INFO c8backup::restore Restored snapshot camundaoperate16832146208.2.3part1of6 18322ms INFO c8backup::restore Restored snapshot camundaoperate16832146208.2.3part2of6 18925ms INFO c8backup::restore Restored snapshot camundaoperate16832146208.2.3part3of6 23336ms INFO c8backup::restore Restored snapshot camundaoperate16832146208.2.3part4of6 23991ms INFO c8backup::restore Restored snapshot camundaoperate16832146208.2.3part5of6 26099ms INFO c8backup::restore Restored snapshot camundaoperate16832146208.2.3part6of_6
c8backup::restore::deletezeebedata{backup=Backup { id: 1683214620, snapshots: ["camundazeeberecords1683214620", "camundaoperate16832146208.2.3part1of6", "camundaoperate16832146208.2.3part2of6", "camundaoperate16832146208.2.3part3of6", "camundaoperate16832146208.2.3part4of6", "camundaoperate16832146208.2.3part5of6", "camundaoperate16832146208.2.3part6of6"] }} 303ms INFO c8backup::restore Deleting data of data-ccs23-dev-zeebe-0 332ms INFO c8backup::restore Deleting data of data-ccs23-dev-zeebe-1 360ms INFO c8backup::restore Deleting data of data-ccs23-dev-zeebe-2 17284ms INFO c8backup::restore Deleted data of data-ccs23-dev-zeebe-0 17385ms INFO c8backup::restore Deleted data of data-ccs23-dev-zeebe-1 17482ms INFO c8backup::restore Deleted data of data-ccs23-dev-zeebe-2
c8backup::restore::restorezeebedata{backup=Backup { id: 1683214620, snapshots: ["camundazeeberecords1683214620", "camundaoperate16832146208.2.3part1of6", "camundaoperate16832146208.2.3part2of6", "camundaoperate16832146208.2.3part3of6", "camundaoperate16832146208.2.3part4of6", "camundaoperate16832146208.2.3part5of6", "camundaoperate16832146208.2.3part6of6"] }} 97ms INFO c8backup::restore Restoring data of data-ccs23-dev-zeebe-0 131ms INFO c8backup::restore Restoring data of data-ccs23-dev-zeebe-1 161ms INFO c8backup::restore Restoring data of data-ccs23-dev-zeebe-2 26025ms INFO c8backup::restore Restored data of data-ccs23-dev-zeebe-0 26117ms INFO c8backup::restore Restored data of data-ccs23-dev-zeebe-1 26210ms INFO c8backup::restore Restored data of data-ccs23-dev-zeebe-2
c8backup::restore::startapps{} 0ms INFO c8backup::restore Starting apps 29ms INFO c8backup::restore Started ccs23-dev-zeebe-gateway 59ms INFO c8backup::restore Started ccs23-dev-operate 90ms INFO c8backup::restore Started ccs23-dev-zeebe```