Prometheus Web Exporter

Prometheus Web Exporter is a prometheus exporter that collects information about web pages. You can collect 2 different type of information from url. Network health info (response time and response body size) and content based info (Number of elements in a page that matches with given css query.)

Web exporter uses same technology with firefox browser engine (servo) to parse and run css queries. So it is quite fast and supports wide veriety of css queries. Unfortunately it does not run javascript so any dom manipulation done with javascript will not be caught by web exporter. (no SPA support.)

Configuration

Web configuration is done with a file named webexporter.yaml that is located in the same directory with the executable. Here is an example for webexporterl.yaml file:

``` yaml

ip address server will listen. default: 0.0.0.0

ip_address: "0.0.0.0"

#

port that servier will listen. default: 3030

port: 3030

#

metrics path. default: /metrics

metrics_path: "metrics"

#

targets to crawl for each request.

targets: # 200 response with queries - url: "https://www.rust-lang.org/" queries: - "#language-values div.flex-none section" - "header h1" - "footer div.attribution" extralabels: name: homepage # 404 response with queries - url: "https://www.rust-lang.org/invalid-page-with-404-response" headers: Referer: "https://www.rust-lang.org/" queries: - "div.flex" - "div" extralabels: name: 404 page # Network error. (Queries will not return any value since they will not be running.) - url: "https://www.page-does-not-exist.io/" queries: - "div" extralabels: name: nonexistentpage # Invalid query (return value will be 0 and css query parse error will be logged.) - url: "https://www.rust-lang.org/invalid-css-query" queries: - "XX" extralabels: name: querywithinvalidcss # 200 page without any query (only response time and size will be returned.) - url: "https://www.rust-lang.org/no-css-query" # google search test example with queryparameters and extra headers. - url: "https://www.google.com/search" queryparameters: q: rust headers: referer: "https://www.google.com/" user-agent: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10154) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.61 Safari/537.36" queries: - "div.g" extralabels: name: googlesearch ```

Configuration above will generate metrics like following:

txt web_exporter_response_duration_milliseconds{url="https://www.rust-lang.org/", method="GET", error="false", name="homepage", status="200" } 787 web_exporter_response_response_size_bytes{url="https://www.rust-lang.org/", method="GET", error="false", name="homepage", status="200" } 19220 web_exporter_query_count{url="https://www.rust-lang.org/", method="GET", error="false", name="homepage", status="200", query="#language-values div.flex-none section" } 3 web_exporter_query_count{url="https://www.rust-lang.org/", method="GET", error="false", name="homepage", status="200", query="header h1" } 1 web_exporter_query_count{url="https://www.rust-lang.org/", method="GET", error="false", name="homepage", status="200", query="footer div.attribution" } 1 web_exporter_response_duration_milliseconds{error="false", url="https://www.rust-lang.org/invalid-page-with-404-response", status="404", name="404 page", method="GET" } 142 web_exporter_response_response_size_bytes{error="false", url="https://www.rust-lang.org/invalid-page-with-404-response", status="404", name="404 page", method="GET" } 8244 web_exporter_query_count{error="false", url="https://www.rust-lang.org/invalid-page-with-404-response", status="404", name="404 page", method="GET", query="div.flex" } 6 web_exporter_query_count{error="false", url="https://www.rust-lang.org/invalid-page-with-404-response", status="404", name="404 page", method="GET", query="div" } 14 web_exporter_response_duration_milliseconds{name="nonexistent_page", status="0", url="https://www.page-does-not-exist.io/", method="GET", error="true" } 83 web_exporter_response_response_size_bytes{name="nonexistent_page", status="0", url="https://www.page-does-not-exist.io/", method="GET", error="true" } 0 web_exporter_response_duration_milliseconds{error="false", name="query_with_invalid_css", method="GET", status="404", url="https://www.rust-lang.org/invalid-css-query" } 110 web_exporter_response_response_size_bytes{error="false", name="query_with_invalid_css", method="GET", status="404", url="https://www.rust-lang.org/invalid-css-query" } 8244 web_exporter_query_count{error="false", name="query_with_invalid_css", method="GET", status="404", url="https://www.rust-lang.org/invalid-css-query", query="**XX**" } 0 web_exporter_response_duration_milliseconds{error="false", status="404", url="https://www.rust-lang.org/no-css-query", method="GET" } 127 web_exporter_response_response_size_bytes{error="false", status="404", url="https://www.rust-lang.org/no-css-query", method="GET" } 8244 web_exporter_response_duration_milliseconds{name="google_search", url="https://www.google.com/search", status="200", method="GET", error="false" } 964 web_exporter_response_response_size_bytes{name="google_search", url="https://www.google.com/search", status="200", method="GET", error="false" } 406579 web_exporter_query_count{name="google_search", url="https://www.google.com/search", status="200", method="GET", error="false", query="div.g" } 11 web_exporter_scrape_duration_milliseconds 972

How to get it

With docker.

``` bash

Download sample configuration from github

wget https://raw.githubusercontent.com/huseyinyilmaz/webexporter/master/sampleweb_exporter.yaml

Rename it to web_exporter.yaml

mv samplewebexporter.yaml web_exporter.yaml

Run Docker and mount configuration file to container

$ docker run \ --rm -ti -d \ -p 3030:3030 \ --name webexporter \ -v $(pwd)/webexporter.yaml:/usr/local/prometheuswebexporter/webexporter.yaml \ huseyinyilmaz/webexporter:v1.0.5

endpoint should be on http://localhost:3030/metrics

```

With docker-compose

First download sample configuration from repository and put it in the same directory as docker-compose.yaml file. Then you can add following config to your docker-compose file. webexporter: image: huseyinyilmaz/web_exporter:v1.0.5 volumes: - ./web_exporter.yaml:/usr/local/prometheus_web_exporter/web_exporter.yaml environment: WEB_EXPORTER_LOG_LEVEL: warn ports: - 3030:3030 expose: - 3030

With Cargo

``` bash

First install web_exporter with cargo.

cargo install prometheuswebexporter

After installation cargo will show where it is installed it is likely on

~/.cargo/bin. Go to installation folder and download sample config next to executable.

cd ~/.cargo/bin

Download sample configuration from github

wget https://raw.githubusercontent.com/huseyinyilmaz/webexporter/master/sampleweb_exporter.yaml

Rename it to web_exporter.yaml

mv samplewebexporter.yaml web_exporter.yaml

run web_exporter

./prometheuswebexporter

endpoint should be on http://localhost:3030/metrics

```

With prebuilt binary.

You can download the binary for your os from releases section in github. After getting the binary, just put the configuration next to binary you are good to go.

Logging:

Logging configuration can be provided through environment variables. To run the project with info logging level you can run it like this

bash $ WEB_EXPORTER_LOG_LEVEL=info ./prometheus_web_exporter