This crate implements a sort algorithm for text files composed of lines or line records. For example CSV or TSV.
A data file composed of lines or line records, that is lines that are composed of fields separated by a delimiter, can be sorted using this crate. Example for such files are pgdump, CSV and GTFS data files. The motivation for writing this module was the need to sort pgdump files of the OpenStreetMap database containing billions of lines by the primary key of each table before converting the data to PBF format.
This implementation can be used to sort very large files, taking advantage of multiple CPU cores and providing memory usage control.
Issues are welcome and appreciated. Please submit to https://github.com/navigatorsguild/text-file-sort/issues
Benchmarks generated by benchmark-rs
```rust use std::path::PathBuf; use textfilesort::sort::Sort;
// optimized for use with Jemalloc use tikv_jemallocator::Jemalloc;
static GLOBAL: Jemalloc = Jemalloc;
// parallel record sort fn sortrecords(input: PathBuf, output: PathBuf, tmp: PathBuf) -> Result<(), anyhow::Error> { let mut textfile_sort = Sort::new(vec![input.clone()], output.clone());
// set number of CPU cores the sort will attempt to use. When given the number that exceeds
// the number of available CPU cores the work will be split among available cores with
// somewhat degraded performance. The default is to use all available cores.
text_file_sort.with_tasks(2);
// set the directory for intermediate results. The default is the system temp dir -
// std::env::temp_dir(), however, for large files it is recommended to provide a dedicated
// directory for intermediate files, preferably on the same file system as the output result.
text_file_sort.with_tmp_dir(tmp);
text_file_sort.sort()
} ```
License: MIT OR Apache-2.0