Spider Worker

crate version

A spider worker to decentralize the crawl lifting.

Dependencies

This project depends on the spider crate.

Usage

cargo run

WIP

  1. ~~request handling io load~~
  2. parse links per domain