Product OS : Crawler

Product OS : Crawler is a browser based cralwer that utilises Product OS : Browser to perform advanced url crawling leveraging headless browsing and automation.

What is Product OS?

Product OS is a collection of packages that provide different tools and features that can work together to build products more easily for the Rust ecosystem.

Installation

Use the Rust crate package manager cargo to install Product OS : Crawler.

bash cargo add product-os-crawler

or add Product OS : Crawler to your cargo.toml [packages] section.

rust product-os-crawler = { version = "0.0.5", features = [], default-features = true, optional = false }

Features

Product OS Crawler supports a number of features leveraging existing Rust libraries to crawl and perform instructions including:

• Basic crawling capabilities with back-off rules
• Revisit logic and full configuration for tuning
• Scoring system fully configurable to determine value of content
• Ability to hand-off to indexing and storage services using a processor

rust // Feature samples TODO

Usage

rust // Examples TODO

Contributing

Contributions are not currently available but will be available on a public repository soon.

License

GNU AGPLv3