A Rust library the crate help you Write a function similar to that of Python's request repository ([Python] BS4 library).
Each new requests initializes a cache instance which stores the parsed data in key value pairs
When you get an instance of connect, you can call the parser method to parse the data in the form of closure
```rust let data = Cache::new(); let client = Requests::new(&data); let mut rq = client.connect("https://www.qq.com/", Headers::Default);
rq.parser(|p| { p.findall("a", |x| { x.attr("href").mapor(false, |v| v.starts_with("http")) }, "href") }, "href"); // data.print() ```
Use data.print you can view the value stored as the [href] key. It is a value enumeration type that contains most data types.
rust
Key -- "href" Value -- LIST(["https://qzone.qq.com", "https://qzone.qq.com", "https://mail.qq.com", "https://mail.qq.com/cgi-bin/loginpage", "http://news.qq.com/", "https://v.qq.com/?isoldly=1", "http://gongyi.qq.com/",...]
Headers defines three types of request headers. The default Header::default has only one user agent, or it can be without any Headers::None.also use JSON string to make a request header containing useragent and host, this code:
rust
let headers = r#"{"user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.69 Safari/537.36", "host": "www.qq.com"}"#;
let store = Cache::new();
let client = Requests::new(&store);
let mut p = client.connect("https://www.qq.com", Headers::JSON(headers));
If you need more request header fields, you need to use add corresponding fields in headers.rs
When you use the connect method to connect to a URL, you can use the parser method to write the work of parsing HTML, the current parser has find and find_ all method
the parser must return Value type
If you use find find_all the code:
rust
rq.parser(|p| {
p.find_all("a", |x| {
x.attr("href").map_or(false, |v| v.starts_with("http"))
}, "href")
}, "href")
The first parameter of parser is obtained by closure automatically saves the value to value:: list
In general, you may need to handle the parsing manually and customize the returned Value, please use p.select function , this example code:
```rust let data = Cache::new(); let client = Requests::new(&data); let mut parser = client.connect("https://www.qq.com", Headers::Default); parser.parser(|p| { let mut result = HashMap::new();
let navs = p.select("li.nav-item", |nodes| {
let navs = nodes.into_iter().map(|n| {
let mut item = HashMap::new();
n.find(Name("a")).next().map_or(HashMap::from([("".to_string(), Value::NULL)]), |a| {
let nav_name = a.text();
let nav_href = a.attr("href").map_or(String::from(""), |x| x.to_string());
item.insert("nav_name".to_string(), Value::STR(nav_name));
item.insert("nav_href".to_string(), Value::STR(nav_href));
item
})
}).collect::<Vec<HashMap<String, Value>>>();
Value::VECMAP(navs)
});
let news = p.select("ul.yw-list", |nodes| {
let mut news = Vec::new();
for node in nodes {
for n in node.find(Class("news-top")) {
for a in n.find(Name("a")) {
let title = a.text();
news.push(title);
}
}
}
Value::LIST(news)
});
result.insert("titles".to_owned(), news);
result.insert("nav".to_owned(), navs);
Value::MAP(result)
}, "index");
data.print();
```
```rust
pub enum Value {
/// 字符串类型
STR(String),
/// 字符串列表
LIST(Vec
/// map类型的列表
VECMAP(Vec<HashMap<String, Value>>),
/// map类型
MAP(HashMap<String, Value>)
} ``` Add the data type you need in the Value.rs
use rayon library test the concurrency, this have a simple code:
```rust
let data = Cache::new();
let client = Requests::new(&data);
let urls = ["https://www.baidu.com", "https://www.qq.com", "https://www.163.com"];
let _ = urls.pariter().map(|url| {
let mut p = client.connect(url, Headers::Default);
p.parser(|p| {
p.findall("a", |f| f.attr("href").mapor(false, |v| v.startswith("http://")), "href")
}, format!("{}link", url).asstr());
p.parser(|p| {
p.find("title", |f| f.text() != "", "text")
}, format!("{}_title", url).as_str());
})
.map(|_| String::from("")).collect::<String>();
match data.get("https://www.qq.com_title") {
Value::STR(i) => assert_eq!(i, "腾讯首页"),
_ => panic!("")
};
if let Value::STR(i) = data.get("https://www.163.com_title") {
assert_eq!(i, "网易");
}
if let Value::STR(i) = data.get("https://www.baidu.com_title") {
assert_eq!(i, "百度一下，你就知道");
}
```