🎁 cargo-fetcher

Alternative to cargo fetch for use in CI or other "clean" environments that you want to quickly bootstrap with the necessary crates to compile/test etc your project(s).

Why?

• You run CI jobs inside of GCP and you want faster crates.io and git downloads so that your compute resources can be spent on the things that you actually care about.

Why not?

• You don't run CI inside of GCP. Currently cargo-fetcher only supports storing crates/git snapshots inside of GCS which means they can be located closer to the compute resources your CI is running on. PRs are of course welcome for adding additional storage backends though!
• cargo-fetcher should not be used in a typical user environment as it completely disregards various safety mechanisms that are built into cargo, such as file-based locking.

Features

gcs

The gcs feature enables the use of Google Cloud Storage as a backend.

• Must provide a url to the -u | --url parameter with the gsutil syntax gs://<bucket_name>(/<prefix>)?
• Must provide GCP service account credentials either with --credentials or via the GOOGLE_APPLICATION_CREDENTIALS environment variable

s3

The s3 feature enables the use of Amazon S3 as a backend.

• Must provide a url to the -u | --url parameter, it must of the form http(s)?://<bucket>.s3(-<region>).<host>(/<prefix>)?
• Must provide AWS credentials by the default mechanism(s) described here

fs

The fs feature enables use of a folder on a local disk to store crates to and fetch crates from.

• Must provide a url to the -u | --url parameter with the file: scheme

Examples

This is an example from our CI for an internal project.

Dependencies

• 424 crates.io crates: cached - 38MB, unpacked - 214MB
• 13 crates source from 10 git repositories: db - 27MB, checked out - 38MB

Scenario

The following CI jobs are run in parallel, each in a Kubernetes Job running on GKE. The container base is roughly the same as the official rust:1.39.0-slim image.

• Build modules for WASM
• Build modules for native
• Build host client for native

~ wait for all jobs to finish ~

• Run the tests for both the WASM and native modules from the host client

Before

All 3 build jobs take around 1m2s each to do cargo fetch --target x86_64-unknown-linux-gnu

After

All 3 build jobs take 3-4s each to do cargo fetcher --include-index mirror followed by 5-7s to do cargo fetch --target x86_64-unknown-linux-gnu.

Contributing

We welcome community contributions to this project.

Please read our Contributor Guide for more information on how to get started.

License

Licensed under either of

• Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.