Prometheus Exporter for Solana

This is a Prometheus exporter for Solana originally based on the Golang original by CertusOne but now providing additional functionality. It is the basis for Grafana dashboards and status alerts.

Metrics

The tool exports the following metrics:

Build

Setup

By default solana-exporter takes input config from ~/.solana-exporter/config.toml. If it doesn't exist it can be generated with template values. Here is a demo example of a config file that works with the public mainnet RPC server and doesn't require running a validator or opening a MaxMind account: rpc = 'https://api.mainnet-beta.solana.com/' target = '127.0.0.1:9179' pubkey_whitelist = []

Here is a production config template for monitoring the entire network via the local validator RPC port and getting server location data from MaxMind: ``` rpc = 'http://localhost:8899/' target = '127.0.0.1:9179' pubkey_whitelist = []

[maxmind] username = 'username' password = 'password' ```

Run

Run this as a systemd service by a non-root user with a script like this one: ``` [Unit] Description=Solana Exporter After=solana.service Requires=solana.service

[Service] User=solana Restart=always RestartSec=20 ExecStart=/home/solana/.cargo/bin/solana-exporter

[Install] WantedBy=multi-user.target ```

Run it as a docker container. shell docker run -d -v /path/to/your/config.toml:/etc/solana-exporter/config.toml -v solana-exporter-data:/exporter solana-exporter

Prometheus and Grafana Setup

If querying a public RPC port, solana-exporter can be run from anywhere, not necessarily from the validator machine. If querying a private RPC port, install Prometheus on the validator machine. Add the following snippet to the scrape_configs section of the prometheus.yml config file:

- job_name: solana static_configs: - targets: ['localhost:9179']

Restart Prometheus. Now the solana-exporter metrics should be available to view at http://localhost:9179/metrics. If running on the validator machine, it is highly advisable to only open the Prometheus datasource port to the Grafana machine. This can be achieved with nft. Here is an example /etc/nftables.conf:

```

!/usr/sbin/nft -f

flush ruleset

table inet filter { chain input { type filter hook input priority 0; # allow connection to Prometheus datasource only locally and from Grafana ip saddr { 127.0.0.1, } tcp dport 9090 accept tcp dport 9090 drop # allow connection to Prometheus exporter endpoints only internally ip saddr != 127.0.0.1 tcp dport 9100 drop ip saddr != 127.0.0.1 tcp dport 9179 drop } chain forward { type filter hook forward priority 0; } chain output { type filter hook output priority 0; } } ```

Note the order of commands. An accept clause should appear before the corresponding drop clause.

When solana-exporter is used on a mainnet validator node, Grafana must always run on a different machine to circumvent potential DDoS attacks on the validator. In the Grafana dashboard, add the Prometheus data source http://<Validator IP>:9090. Then import the default dashboard using that data source.

To display pie charts we use a Grafana pie chart plugin. Prior to Grafana v8 it needed to be installed in order for the pie charts to be displayed. Starting from v8 pie charts are included in the core distribution.

Examples

Dashboard

A starting point can be our default dashboard.

Sample output to the Prometheus target endpoint

Some repetitive lines are ellipsed for brevity in the example below. ```

HELP solanaactivevalidators Total number of active validators

TYPE solanaactivevalidators gauge

solanaactivevalidators{status="current"} 561 solanaactivevalidators{status="delinquent"} 51

HELP solanaactivevalidatorsdcstake Datacenter of active validators grouped by stake

TYPE solanaactivevalidatorsdcstake gauge

solanaactivevalidatorsdcstake{dcidentifier="11427-US-Austin"} 9672542164238 solanaactivevalidatorsdcstake{dcidentifier="11524-US-Portland"} 35172007429 solanaactivevalidatorsdcstake{dc_identifier="12212-CA-Toronto"} 407342367161475 ...

HELP solanaactivevalidatorsispcount ISP of active validators

TYPE solanaactivevalidatorsispcount gauge

solanaactivevalidatorsispcount{ispname="Advanced Solutions LLC"} 1 solanaactivevalidatorsispcount{ispname="Amazon.com"} 48 solanaactivevalidatorsispcount{isp_name="CAIW Internet"} 1 ...

HELP solanaactivevalidatorsispstake ISP of active validators grouped by stake

TYPE solanaactivevalidatorsispstake gauge

solanaactivevalidatorsispstake{ispname="Advanced Solutions LLC"} 230372233054571 solanaactivevalidatorsispstake{ispname="Amazon.com"} 97432578281165840 solanaactivevalidatorsispstake{isp_name="CAIW Internet"} 352505951070073 ...

HELP solanacurrentepoch Current epoch

TYPE solanacurrentepoch gauge

solanacurrentepoch 186

HELP solanacurrentepochfirstslot Current epoch's first slot

TYPE solanacurrentepochfirstslot gauge

solanacurrentepochfirstslot 80665865

HELP solanacurrentepochlastslot Current epoch's last slot

TYPE solanacurrentepochlastslot gauge

solanacurrentepochlastslot 81097865

HELP solanaleaderslots Validated and skipped leader slots per validator

TYPE solanaleaderslots counter

solanaleaderslots{pubkey="12CUDzb3oe8RBQ4tYGqsuPsCbsVE4KWfktXRihXf8Ggq",status="skipped"} 54 solanaleaderslots{pubkey="12CUDzb3oe8RBQ4tYGqsuPsCbsVE4KWfktXRihXf8Ggq",status="validated"} 146 solanaleaderslots{pubkey="12oRmi8YDbqpkn326MdjwFeZ1bh3t7zVw8Nra2QK2SnR",status="skipped"} 35 solanaleaderslots{pubkey="12oRmi8YDbqpkn326MdjwFeZ1bh3t7zVw8Nra2QK2SnR",status="validated"} 217 ... ```