A robots.txt parser and applicability checker for Rust

Usage

Add it to your Cargo.toml:

toml [dependencies] robots-parser = "0.10"

Examples

Parse and check from URL

```rust use robots::RobotsParser; use url::Url;

fn main() { let parsed = RobotsParser::parseurl(Url::new("https://www.google.com/robots.txt"))?; assert!(parsed.canfetch("*", "https://www.google.com/search/about")); } ```

Parse and check from File

```rust use robots::RobotsParser;

fn main() { let parsed = RobotsParser::parsepath("~/test-robots.txt"))?; assert!(parsed.canfetch("*", "http://test.com/can_fetch")); } ```

Parse and check from &str

```rust use robots::RobotsParser;

fn main() { let parsed = RobotsParser::parsepath("Disallow: /test"))?; assert!(!parsed.canfetch("*", "http://test.com/test")); } ```

License

This work is released under Apache and MIT license. A copy of the licenses are provided in the LICENSE-APACHE and LICENSE-MIT files.