Crawls websites to gather all possible pages really fast and uses gRPC.
Make sure to have Rust installed or use Docker. This project requires that you start up another gRPC server on port 50051 following proto spec - gRPC node example. This crawler is optimized for reduced latency and performance as it can handle over 10,000 pages within seconds.
cargo run or docker compose upYou can use the program as a docker image.
You can use the crate to setup a gRPC server to run on the machine.
In order to use the crawler atm you need to add the grpc client based in the proto location called website.proto. Streams support is in the making to remove the extra need for the client.
Check the license file in the root of the project.