Gitlobster

A tool for full cloning all available repositories from a GitLab server.

EN | RU

Key features

• Clone all available repositories
• Clone all branches from each repository
• Upload all repositories to another GitLab server (or a group in the same GitLab)
• Download only updates (including all newly added repositories) after the first full cloning
• Preserve the group hierarchy
• Support filters (include regexp templates) for cloning only necessary repository

Install

Docker

shell docker run --rm -it lowitea/gitlobster:latest --help

Running pre-assembled binary files

1. Download an archive from the release page for your OS.
2. Unpack the archive.
3. Run the gitlobster file.

Cargo

shell cargo install gitlobster

Building from source

```shell

clone the repository

git clone https://github.com/lowitea/gitlobster.git

going to the downloaded directory

cd gitlobster

build

cargo build --release

run

./target/release/gitlobster --help ```

The option to run it in the developer mode without pre-build is also available.

```shell

in the project directory

cargo run -- --help ```

Usage

GitLab Token

In order for the tool to work, you need to generate a GitLab token with API read rights (read_api). If SSH copying is not used, then you will also need a permission to read repositories (read_repository).

If a second GitLab is used to copy repositories there, then a token is also required for it. Full API rights are required (api). If SSH upload is not used, then you will also need write permissions for repositories (write_repository).

You can generate tokens on the settings page.

SSH

If SSH copying is used, then ssh keys must be added in GitLab.

Help command

```text $ gitlobster --help

A tool for cloning all available repositories in a GitLab instance

Usage: gitlobster [OPTIONS] --fu --ft

Options: --fu The GitLab instance URL for fetch repositories (example: https://gitlab.local/) [env: GTLBSTRFETCHURL=] --ft Your personal GitLab token for fetch repositories [env: GTLBSTRFETCHTOKEN=] --bu The GitLab instance URL for backup repositories (example: https://backup-gitlab.local/) [env: GTLBSTRBACKUPURL=] --bt Your personal GitLab token for backup repositories [env: GTLBSTRBACKUPTOKEN=] --bg A target created group on backup GitLab for push repositories [env: GTLBSTRBACKUPGROUP=] -i, --include Include regexp patterns (cannot be used together with --exclude flag, may be repeated) [env: GTLBSTRINCLUDE=] -x, --exclude Comma separated exclude regexp patterns (cannot be used together with --include flag, may be repeated) [env: GTLBSTREXCLUDE=] -d, --dst A destination local folder for save downloaded repositories [env: GTLBSTRDST=] -v, --verbose... Verbose level (one or more, max four) --dry-run Show all projects to download --objects-per-page Low-level option, how many projects can fetch in one request [env: GTLBSTROBJECTSPERPAGE=] --limit Maximum projects to download [env: GTLBSTRLIMIT=] --concurrency-limit Limit concurrency download [env: GTLBSTRCONCURRENCYLIMIT=] [default: 21] --only-owned Download projects explicitly owned by user [env: GTLBSTRONLYOWNED=] --only-membership Download only user's projects [env: GTLBSTRONLYMEMBERSHIP=] --download-ssh Enable download by ssh instead of http. An authorized ssh key is required [env: GTLBSTRDOWNLOADSSH=] --upload-ssh Enable upload by ssh instead of http. An authorized ssh key is required [env: GTLBSTRUPLOADSSH=] --disable-hierarchy Disable saving the directory hierarchy [env: GTLBSTRDISABLEHIERARCHY=] --clear-dst Clear dst path before cloning [env: GTLBSTRCLEARDST=] --only-master Download only default branch [env: GTLBSTRONLY_MASTER=] -h, --help Print help information -V, --version Print version information ```

Copying all repositories to a second GitLab

shell gitlobster \ --ft=<FETCH_TOKEN> \ --fu=https://gitlab.com/ \ --bt=<UPLOAD_TOKEN> \ --bu=https://gitlab.com/ \ --bg=gitlobster_test/upload

Download all repositories to a local directory

shell gitlobster \ --ft=<FETCH_TOKEN> \ --fu=https://gitlab.com/ \ -d out_directory

Simultaneous saving repositories to a local directory and a second GitLab is supported.

Using filters and filtering flags

shell gitlobster \ --ft=<FETCH_TOKEN> \ --fu=https://gitlab.com/ \ --only-owned \ --include="^gitlobster_test/download/project_2" \ --include="^gitlobster_test/download/project_1" \ -d out_directory

It's also possible to use --exclude flag to load all repositories except repositories matching a necessary template.

Simultaneous use of both --exclude and --include flags isn't allowed.

Using with Docker

shell docker run --rm -it -v $(pwd)/out:/out gitlobster:dev \ --ft=<FETCH_TOKEN> \ --fu=https://gitlab.com/ \ --include='^gitlobster_test/example' \ -d /out

Analogues

• gitlabber
• ghorg