webreg (web regex)

CLI tool for testing regexes against web pages.

Test if a list of websites match a given regex

Installation

bash cargo install webreg

Usage

bash webreg [OPTIONS] <REGEX>

``` Arguments: A regular expression to match against the site content

Options: -u, --urls Comma separated list of urls -i, --file A file containing a list of urls -c, --case-insensitive Case insensitive search -f, --fix-urls Fix urls that don't start with http:// or https:// -r, --retry Retry failed urls -s, --save Saves the output to the results folder (./results/) -h, --help Print help ```

Examples

Basic usage

bash webreg -u "https://example.com" "Hello World"

This will check if the string "Hello World" is present in the content of https://example.com. If it is, it will print the url to stdout.

Multiple urls

bash webreg -u "https://example.com,https://example.org" "Hello World"

Domains

bash webreg -u -f "example.com,example.org" "Hello World"

The -f flag will fix urls that don't start with http:// or https://

Case insensitive

bash webreg -u -c "https://example.com" "hello world"

The -c flag will make the search case insensitive.

File input

bash webreg -i urls.txt "Hello World"

urls.txt:

bash https://example.com https://example.org

The -i flag will read the urls from a file. The file should contain one url per line. Empty lines will be ignored and whitespace will be trimmed.

Pipe input

bash cat urls.txt | webreg -i "Hello World"

urls.txt:

bash https://example.com https://example.org

Save the output

bash webreg -u -s "https://example.com" "Hello World"

The -s flag will save the output to the results folder (./results/<regex>). This will also output lists urls that couldn't be fetched and urls that didn't match the regex.