osm-pbf

Parse and write the PBF format for Open Street Map (OSM).

Compression

There is a feature for each supported compression algorithm:

Name|Default Feature|Decode (read)|Encode (write) ---|---|---|--- Zlib|✅|✅|✅ Zstd|❌|✅|✅ Lzma|❌|✅|✅ Bzip2|❌|❌|❌ Lz4|❌|❌|❌

Bzip2 has been deprecated for years so it is not supported, Lz4 support is not available yet.

There isn't any fine-grained control over encoding but feel free to file an issue if you are interested.

Execution

This crate is written with async I/O for use with tokio.

Parallelism

The code is serial in nature but it's possible to parallelize encoding/decoding since fileblocks are independent in PBF.

Read parallelization example:

1. Call get_osm_pbf_locations to get a stream of fileblock locations.
2. Chunk the fileblocks into relevant groups. Namely: header followed by any number of primitive fileblocks.
3. Call parse_osm_pbf_from_locations for each chunk independently (i.e. buffer_unordered).
4. Process blocks as desired.

Write parallelization example:

1. Split your blocks into chunks of size N. Note: each chunk must contain a Header block followed by any number of Primitive blocks.
2. Call write_osm_pbf for each chunk independently (i.e. buffer_unordered) with an in-memory vector as the writer.
3. As each call completes, write them to their final destination (i.e. a file).