Crawls websites to gather all possible pages really fast and uses gRPC.
Make sure to have Rust installed or use Docker. This project requires that you start up another gRPC server on port 50051 following proto spec. We are working on adding streams instead of a post hook approach. The server sends all request into another thread and follows up on extracting the links via gRPC callbacks.
cargo run or docker compose upYou can use the program as a docker image.
You can use the crate to setup a gRPC server to run on the machine.
In order to use the crawler atm you need to add the grpc client based in the proto location called website.proto. Streams support is in the making to remove the extra need for the client.
Check the license file in the root of the project.