Product OS : Crawler

Product OS : Crawler is a powerful url crawler that leverages headless browsing and provides automation tooling to not only read content but take actions defined via an instruction language.

What is Product OS?

Product OS is a collection of packages that provide different tools and features that can work together to build products more easily for the Rust ecosystem.

Installation

Use the Rust crate package manager cargo to install Product OS : Crawler.

bash cargo install product-os-crawler

or add Product OS : Crawler to your cargo.toml [packages] section.

rust product-os-crawler = { version = "0.0.1", features = [], default-features = true, optional = false }

Features

Product OS Crawler supports a number of features leveraging existing Rust libraries to crawl and perform instructions including:

• Basic crawling capabilities with back-off rules
• Revisit logic and full configuration for tuning
• Complete configuration instruction set to perform actions on a page to create automation
• Scoring system fully configurable to determine value of content
• Ability to hand-off to indexing and storage services using a processor
• Currently supports Firefox as both visible and headless browser
• Supports insertion of a proxy such as Product OS : Proxy to allow for mitm manipulation and tunneling via VPN or Tor

rust // Feature samples TODO

Usage

rust // Examples TODO

Contributing

Contributions are not currently available but will be available on a public repository soon.

License

GNU AGPLv3