fetch-hash

Fetch data files from a URL, if needed. Verify contents via SHA256.

Fetch-Hash makes creating example code easier. In this example, we download a genomics file from GitHub and print its size:

```rust use fetchhash::samplefile;

let path = sample_file("small.fam")?; println!("{}", std::fs::metadata(path)?.len()); // Prints 85

use fetch_hash::FetchHashError; // '#' needed for doctest

Ok::<(), FetchHashError>(())

```

Features

• Thread-safe -- allowing it to be used with Rust's multithreaded testing framework.
• Inspired by Python's popular Pooch and our PySnptools filecache module.
• Avoids run-times such a Tokio by using ureq to download files via blocking I/O.

Suggested Usage

You can use FetchHash many ways. Here are the steps for one way to use it, followed by sample code.

• Create a registry.txt file containing a whitespace-delimited list of files and their hashes. (This is the same format as Pooch). See Registry Creation below, for more information.

• As shown below, create a global static FetchHash instance that reads your registry.txt file. Tell it:

  • the URL root from which to download the files
  • an environment variable to set the local data directory where files are stored.

  • a qualifier, organization, and application to create a local data directory when the environment variable is not set. (See ProjectsDir for details.)

  • As shown below, define a public sample_file function that takes a file name and returns a Result containing the path to the downloaded file.

```rust use fetch_hash::{ctor, FetchHash, FetchHashError}; use std::path::{Path, PathBuf};

[ctor]

static STATICFETCHHASH: FetchHash = FetchHash::new( includestr!("../registry.txt"), "https://raw.githubusercontent.com/CarlKCarlK/fetch-hash/main/tests/data/", "BARAPPDATADIR", // env_var "com", // qualifier "Foo Corp", // organization "Bar App", // application );

/// Download a data file. pub fn samplefile>(path: P) -> Result { STATICFETCHHASH.fetchfile(path) }

```

You can now use your sample_file function to download your files as needed.

Registry Creation

You can create your registry.txt file many ways. Here are the steps for one way to create it, followed by sample code.

• Upload your data files to the Internet. (For example, Fetch-Hash puts its data files in tests/data, so they upload to this GitHub folder. In GitHub, by looking at the raw view of a data file, we see the root URL for these files.)
• As shown below, write code that
  • Creates a FetchHash instance without registry contents.
  • Lists the files in your data directory.
  • Calls gen_registry_contents on your list of files. It will download the files, compute their hashes, and create a string of file names and hashes.
• Print this string then manually put it into a file called registry.txt.

```rust use fetchhash::{FetchHash, dirtofilelist};

let fetchhash = FetchHash::new( "", // registrycontents ignored "https://raw.githubusercontent.com/CarlKCarlK/fetch-hash/main/tests/data/", "BARAPPDATADIR", // envvar "com", // qualifier "Foo Corp", // organization "Bar App", // application ); let filelist = dirtofilelist("tests/data")?; let registrycontents = fetchhash.genregistrycontents(filelist)?; println!("{registrycontents}");

use fetch_hash::FetchHashError; // '#' needed for doctest

Ok::<(), FetchHashError>(())

```

Notes

• Don't use our sample sample_file. Define your own sample_file that knows where to find your data files.
• You don't have to make your FetchHash instance global and static.
• You don't need to use a registry file or FetchHash instance. You can instead use utility functions such as fetch.
• Does binary downloads, so no line ending changes for Windows.

Project Links

• Installation
• Documentation
• Source code
• Discussion
• Bug Reports and Feature Requests