gis_puller

This Rust crate aims to abstract and implement a general global Geographic Information System (GIS) data catalog and acquisition functionality.

Acquired catalog will follow the data provider derived structural separations as-is.

To build on top the below crates can be in future used to consume the acquired data and beyond:

• gis_loader will be available to load/transform these acquired dataset(s) by this crate into various data stores
• gis_api will be available to provide gRPC API to consume the loaded dataset(s)

Everything should be considered unstable including the interface definitions.

GIS Data Sources Supported

• Australia - Geoscape consumed via https://data.gov.au/

GIS DataSets Catalogued

• Suburb/Locality Boundaries: Australia (GDA94, GDA2020, GeoJSON)
• Address Forward/Reverse Lookup Data: Australia (GNAF)

Also see DIFFICULTIES.md

Environment Variables

• USE_MODE = Environment "default" gets merged with either "development" | "integration" | "production" runtime configuration
• GISCONFIGPATH = Filesystem path where the GIS datasource configuration files live
• TESTDATAPATH = Filesystem path where the mocks read data from including the "replay" recordings

Usage

Using Rust Cargo - requires cargo-edit

bash cargo add gis_puller

Or edit Cargo.toml by hand:

toml [dependencies] gis_puller = "0.1.0"

Create documentation bash env GIS_CONFIG_PATH=config RUN_MODE=development cargo doc --no-deps --open

Run tests (includes doctests and runs examples) with either replay or mocked datasources bash env GIS_CONFIG_PATH=config TEST_DATA_PATH=test_data RUN_MODE=development cargo test

Run clippy for additional lints bash cargo clippy --all-targets --all-features

Run tarpaulin for code/examples test coverage BROKEN: Waiting fix on httpmock/issues/#45 workaround manually with the Cargo env bash env GIS_CONFIG_PATH=config TEST_DATA_PATH=test_data RUN_MODE=development cargo +nightly tarpaulin --run-types Tests,Doctests,Benchmarks,Examples,Lib,Bins -v NOTE: Tarpaulin requires +nightly: rustup toolchain install nightly

@TODO binary crate -- To use in shell with gis_puller image Docker e.g. with Australia localities boundaries datasets obtained from data.gov.au: bash docker run --rm -t -i -e env GIS_PULLER_CONFIGPATH=/gis/etc/gis_puller GIS_PULLER_DATAPATH=/gis/data gis_puller gis_puller pull --all au locality/boundary

Or to use as a library: - TBD: Interfaces to appear in future gis_puller::Puller

Currently with AU locality/boundary dataset I doctest fetcher_matcher the below in development the below with recorded result replayed as a mock:

```rust mod mocks; use mocks::http_server; use httpmock::Method::GET; use tokio::task;

[tokio::main]

async fn main() -> Result<(), Box> { let auconfig = task::spawnblocking(move || { gis_puller::settings::Settings::load("au") }).await?.unwrap();

// @TODO: replace this mess with a macrorules! let mut aufetchermatcher = (*auconfig).fetcher_matcher.clone();

let mockserver = mocks::httpserver::serve().await?; let mockpath = format!("/replay/data.gov.au/api/v0/search/datasets"); let mockquery = format!("localities-boundaries");

let injectreplay = format!("testdata/replay/au/localityboundaries.json"); let injecturl = format!("http://localhost:{}{}", mockserver.port(), &mockpath);

aufetchermatcher.resturl = Some(injecturl); aufetchermatcher.query = Some(mock_query.clone());

let mock = mockserver.mock(|when, then| { when.method(GET)
.path(&mockpath) .queryparam("query", &mockquery); then.status(200) .header("Content-Type", "application/json") .bodyfromfile(&inject_replay); });

let catalog = gispuller::au::fetchermatcher(&aufetchermatcher).await?;

println!("Catalog = {:#?}", catalog);

Ok(()) }
```

Scratchpad for future interface: ```ignore

let pullerau = gispuller::Puller::load('au').await?;

let sourcesau = pullerau.sources('locality/boundary').await?;

// make sources_au Iterable for .iter()

sourcesau.pullall('/tmp').await?; sourcesau.pullone('nsw').await?;

```

The below features can be toggled via Cargo:

• 'xxx': Enables xxx

License

Licensed under either of:

• Apache License, Version 2.0, (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.