robotparser-rs

robots.txt parser for Rust.

Installation

Add it to your Cargo.toml:

toml [dependencies] robotparser = "0.4"

Add extern crate robotparser to your crate root and your're good to go!

Examples

```rust extern crate robotparser;

use robotparser::RobotFileParser;

fn main() { let parser = RobotFileParser::new("http://www.python.org/robots.txt"); parser.read(); assert!(parser.can_fetch("*", "http://www.python.org/robots.txt")); } ```

License

This work is released under the MIT license. A copy of the license is provided in the LICENSE file.