od-get

A Rust tool for recursively crawling & downloading data from open directories

• [x] Filtering (regex) support
  • [x] Exclude file patterns
  • [x] Include file patterns
  • [x] Exclude folder patterns
  • [x] Include folder patterns
• [x] Customizable output
  • [x] Target directory
  • [ ] Verbosity
  • [x] Metadata-JSON file generation
  • [ ] Log file/dynamic terminal output
• [ ] Customizable limits
  • [ ] recursion depth limit
  • [ ] file count limit
  • [ ] file count offset (skip n files)
• [x] Multi threaded (using rayon)
• [x] Resume on error (avoid re-downloading files)
• [ ] Multi-level recursion
• [x] Disable download (only crawl to JSON)

(work in progress, one layer of recursion works)

Licence & Copyright

Copyright (c) 2021 Bernd-L. All rights reserved.

od-get is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

od-get is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with od-get. If not, see https://www.gnu.org/licenses/.

This project (including its source code and its documentation) is released under the terms of the GNU Affero General Public License.