robotparser-rs

robots.txt parser for Rust.

Installation

Add it to your Cargo.toml:

toml [dependencies] robotparser = "0.11"

Examples

```rust use robotparser::http::RobotsTxtClient; use robotparser::service::RobotsTxtService; use reqwest::Client; use url::Url;

fn main() { let client = Client::new(); let robotstxturl = Url::parse("https://www.python.org/robots.txt").unwrap(); let robotstxt = client.fetchrobotstxt(robotstxturl.origin()).unwrap().getresult(); let fetchurl = Url::parse("https://www.python.org/robots.txt").unwrap(); assert!(robotstxt.canfetch("*", &fetchurl)); } ```

License

This work is released under the MIT license. A copy of the license is provided in the LICENSE file.