fetch-hash

This crate helps you retrieve data files from the Internet. Files are only downloaded when needed. The contents of the files are always verified via a SHA256 hash.

The ability to download data files makes creating sample code easier. For example, here we download a genomics file and print its size:

```rust use fetchhash::samplefile;

let path = sample_file("small.fam")?; println!("{}", std::fs::metadata(path)?.len()); // Prints 85

use fetch_hash::FetchHashError; // '#' needed for doctest

Ok::<(), FetchHashError>(())

```

Features

Thread-safe -- allowing it to be used with Rust's multithreaded testing framework.
Inspired by Python's popular Pooch and our PySnptools filecache module.
Avoids run-times by using ureq to download files via blocking I/O.

Suggested Usage (with Example Code)

Create a registry.txt file containing a whitespace-delimited list of files and their hashes. (This is the same format as Pooch). You can put this file anywhere in your project. I put it in tests/registry.txt and I put data files in tests/data. See Registry Creation for more information.
In your code, create a global static FetchHash instance that reads your registry.txt file. You should also give it:
- the URL root from which to download the files
- an environment variable that can set the local data directory where files are stored.
- a qualifier, organization, and application used to create a local data directory when the environment variable is not set.
- Define a public sample_file function that takes a file name and returns a Result containing the path to the file.

```rust use fetch_hash::{ctor, FetchHash, FetchHashError}; use std::path::{Path, PathBuf};

[ctor]

static STATICFETCHHASH: FetchHash = FetchHash::new( includestr!("../tests/registry.txt"), "https://raw.githubusercontent.com/CarlKCarlK/fetch-hash/main/tests/data/", "BARAPPDATADIR", "com", "Foo Corp", "Bar App", );

pub fn samplefile>(path: P) -> Result { STATICFETCHHASH.fetchfile(path) }

Ok::<(), FetchHashError>(())

```

You and your users can now use your sample_file function to download your files as needed.

Registry Creation (with Example Code)

Here is one suggested method for creating a registry.txt file.

Upload your data files to the Internet. For example, I put my data files in my project under tests/data, so they uploaded to this GitHub folder. In GitHub, by looking at the raw view of a data file, I see the root URL for these files.
Write code that
- Creates a FetchHash instance without registry contents
- Lists the files in your data directory.
- Calls gen_registry_contents on your list of files. It will download the files, compute their hashes, and create a string of file names and hashes.
Print this string and put it into a file called registry.txt.

```rust use fetchhash::{FetchHash, dirtofilelist};

let fetchhash = FetchHash::new( "", "https://raw.githubusercontent.com/CarlKCarlK/fetch-hash/main/tests/data/", "BARAPPDATADIR", "com", "Foo Corp", "Bar App", ); let filelist = dirtofilelist("tests/data")?; let registrycontents = fetchhash.genregistrycontents(filelist)?; println!("{registrycontents}");

use fetch_hash::FetchHashError; // '#' needed for doctest

Ok::<(), FetchHashError>(())

```

Notes

Don't use our sample_file. Define your own sample_file that knows about your data files.
You don't have to make your FetchHash instance global and static.
You don't need to use a registry file or FetchHash instance. You can instead use utility functions such as fetch.
Does binary downloads, so no line ending changes for Windows.

fetch-hash

use fetch_hash::FetchHashError; // '#' needed for doctest

Ok::<(), FetchHashError>(())

Features

Suggested Usage (with Example Code)

[ctor]

Ok::<(), FetchHashError>(())

Registry Creation (with Example Code)

use fetch_hash::FetchHashError; // '#' needed for doctest

Ok::<(), FetchHashError>(())

Notes

Project Links