OneIO is a Rust library that provides unified simple IO interface for reading and writing to and from data files from different sources and compressions.
Enable all compression algorithms, and handle remote files (default)
toml
oneio = "0.11"
Select from supported feature flags
toml
oneio = {version = "0.11", default-features=false, features = ["remote", "gz"]}
Supported feature flags:
- lib (default): ["gz", "bz", "lz", "remote", "json"]
- all: all flags (["gz", "bz", "lz", "remote", "json", "s3"]
- remote: allow reading from remote files
- gz: support gzip files
- bz: support bzip2 files
- lz: support lz4 files
- json: allow reading JSON content into structs directly
- s3: allow reading from AWS S3 compatible buckets
oneio commandline toolOneIO comes with a commandline tool, oneio, that opens and reads local/remote files
to terminal and handles decompression automatically. This can be useful if you want
read some compressed plain-text files from a local or remote source.
```text Mingwei Zhang mingwei@bgpkit.com OneIO is a Rust library that provides unified simple IO interface for reading and writing to and from data files from different sources and compressions.
USAGE:
oneio [OPTIONS]
ARGS:
OPTIONS:
--cache-dir wget
-h, --help Print help information
-o, --outfile
You can just specify a data file location after oneio. The following command
prints out the raw HTML file from https://bgpkit.com.
bash
oneio https://bgpkit.com
Here is another example of using oneio to read an remote compressed JSON file,
pipe it to jq and count the number of JSON objects in the array.
bash
$ oneio https://data.bgpkit.com/peer-stats/as2rel-latest.json.bz2 | jq '.|length'
802861
You can also directly download a file with the --download (or -d) flag.
```bash
$ oneio -d http://archive.routeviews.org/route-views.amsix/bgpdata/2022.11/RIBS/rib.20221107.0400.bz2
file successfully downloaded to rib.20221107.0400.bz2
$ ls -lh rib.20221107.0400.bz2 -rw-r--r-- 1 mingwei staff 122M Nov 7 16:17 rib.20221107.0400.bz2
$ monocle parse rib.20221107.0400.bz2 |head -n5 A|1667793600|185.1.167.24|3214|0.0.0.0/0|3214 1299|IGP|185.1.167.24|0|0|3214:3001|NAG|| A|1667793600|80.249.211.155|61955|0.0.0.0/0|61955 50629|IGP|80.249.211.155|0|0||NAG|| A|1667793600|80.249.213.223|267613|0.0.0.0/0|267613 1299|IGP|80.249.213.223|0|0|5469:6000|NAG|| A|1667793600|185.1.167.62|212483|1.0.0.0/24|212483 13335|IGP|152.89.170.244|0|0|13335:10028 13335:19000 13335:20050 13335:20500 13335:20530 lg:212483:1:104|NAG|13335|108.162.243.9 A|1667793600|80.249.210.28|39120|1.0.0.0/24|39120 13335|IGP|80.249.210.28|0|0|13335:10020 13335:19020 13335:20050 13335:20500 13335:20530|AG|13335|141.101.65.254 ```
The returned reader implements BufRead, and handles decompression from the following types:
- gzip: files ending with gz or gzip
- bzip2: files ending with bz or bz2
- lz4: files ending with lz4 or lz
It also handles reading from remote or local files transparently.
Read all into string: ```rust const TEST_TEXT: &str = "OneIO test file. This is a test."; fn main(){
let mut reader = oneio::get_reader("https://spaces.bgpkit.org/oneio/test_data.txt.gz").unwrap();
let mut text = "".to_string();
reader.read_to_string(&mut text).unwrap();
assert_eq!(text.as_str(), TEST_TEXT);
} ```
Read into lines: ```rust use std::io::BufRead; const TEST_TEXT: &str = "OneIO test file. This is a test.";
fn main() {
let lines = oneio::readlines("https://spaces.bgpkit.org/oneio/testdata.txt.gz").map(|line| line.unwrap()).collect::
assert_eq!(lines.len(), 2);
assert_eq!(lines[0].as_str(), "OneIO test file.");
assert_eq!(lines[1].as_str(), "This is a test.");
} ```
[get_writer] returns a generic writer that implements [Write], and handles decompression from the following types:
- gzip: files ending with gz or gzip
- bzip2: files ending with bz or bz2
Note: lz4 writer is not currently supported.
```rust fn main() { let toreadfile = "https://spaces.bgpkit.org/oneio/testdata.txt.gz"; let towritefile = "/tmp/testwrite.txt.bz2";
// read text from remote gzip file
let mut text = "".to_string();
oneio::get_reader(to_read_file).unwrap().read_to_string(&mut text).unwrap();
// write the same text to a local bz2 file
let mut writer = oneio::get_writer(to_write_file).unwrap();
writer.write_all(text.as_ref()).unwrap();
drop(writer);
// read from the newly-generated bz2 file
let mut new_text = "".to_string();
oneio::get_reader(to_write_file).unwrap().read_to_string(&mut new_text).unwrap();
// compare the decompressed content of the remote and local files
assert_eq!(text.as_str(), new_text.as_str());
std::fs::remove_file(to_write_file).unwrap();
} ```
rust
use std::collections::HashMap;
fn main() {
let mut reader = oneio::get_remote_reader(
"https://SOME_REMOTE_RESOURCE_PROTECTED_BY_ACCESS_TOKEN",
HashMap::from([("X-Custom-Auth-Key".to_string(), "TOKEN".to_string())])
).unwrap();
let mut text = "".to_string();
reader.read_to_string(&mut text).unwrap();
println!("{}", text);
}
rust
fn main() {
oneio::download(
"https://data.ris.ripe.net/rrc18/2022.11/updates.20221107.2325.gz",
"updates.gz",
None
).unwrap();
}
s3 feature flag)```rust fn main() { // upload to S3 s3_upload("oneio-test", "test/README.md", "README.md").unwrap();
// read directly from S3
let mut content = String::new();
s3_reader("oneio-test", "test/README.md")
.unwrap()
.read_to_string(&mut content)
.unwrap();
println!("{}", content);
// download from S3
s3_download("oneio-test", "test/README.md", "test/README-2.md").unwrap();
// get S3 file stats
let res = s3_stats("oneio-test", "test/README.md").unwrap();
dbg!(res);
// error if file does not exist
let res = s3_stats("oneio-test", "test/README___NON_EXISTS.md");
assert!(res.is_err());
// list S3 files
let res = s3_list("oneio-test", "test/", Some("/")).unwrap();
assert_eq!(
false,
s3_exists("oneio-test", "test/README___NON_EXISTS.md").unwrap()
);
assert_eq!(true, s3_exists("oneio-test", "test/README.md").unwrap());
} ```