Elasticsearch exporter

Prometheus Elasticsearch exporter capable of working with large clusters. Caution you may overload Prometheus server by enabling all metrics, exporter is capable to export over near 1 million metrics. To avoid overloading Prometheus server run multiple Elasticsearch exporters that target just few specific metrics.

bash $ curl -s http://127.0.0.1:9222/metrics | wc 940272 1887011 153668390

Try it out with docker

bash $ docker run --network=host -it vinted/elasticsearch_exporter --elasticsearch_url=http://IP:PORT

Features

Metric collection is decoupled from serving /metrics page
Skips zero/empty metrics (controlled with flag exporter_allow_zero_metrics)
Elasticsearch "millis" converted to seconds
Elasticsearch "kilobytes" converted to bytes
All time based metrics are converted as f64 seconds, keywords millis replaced with seconds
Added _bytes and _seconds postfix
Preserves metrics tree namespace up to last leaf
- elasticsearchcatindicespriwarmertotaltimesecondsbucket
- elasticsearchcathealth_unassign
- elasticsearchnodesinfojvmmemheapmaxinbytes
Custom namespace labels vin_cluster_version for convenient comparison of metrics between cluster versions
Automatic cluster metadata updates every 5 minutes
/_nodes/* API additional labels injection by mapping node ID to fetched cluster metadata
- name (map from node ID to name) namespaced -> name
- version (Elasticsearch node version) namespaced to vin_cluster_version
- IP namespaced -> ip
Automatic metrics deletion based on lifetime settings, by default metric by value will be deleted after 600s since last occurrence.
Metric names are normalized to snake case, colon is replaced with underscore, brackets are replaced with colon (:)
- "transportactionscluster:monitor/nodes/info[n]requestscount" -> "transportactionsclustermonitornodesinfo:n:requests_count"
- "transportactionsinternal:cluster/coordination/join/pingrequestscount" -> "transportactionsinternalclustercoordinationjoinpingrequestscount"

Options

Configurable labels "skip" and/or "include" (flags: exporter_include_labels, exporter_skip_labels)
Configurable skip metrics (controlled with (flag exporter_skip_metrics)
Configurable global timeout (flag elasticsearch_global_timeout)
Configurable global polling interval (flag exporter_poll_default_interval)
Configurable per metric polling interval (flag exporter_poll_intervals)
Configurable metrics collection (flag exporter_metrics_enabled)
Configurable metadata collection (flag exporter_metadata_refresh_interval)

Usage cheat sheet

Scraping /_nodes/stats subsystem thread_pool path metric

$ docker run --network=host -it vinted/elasticsearch_exporter --elasticsearch_url=http://IP:PORT --exporter_metrics_enabled="nodes_stats=true" --elasticsearch_path_parameters="nodes_stats=thread_pool"

Scraping /_nodes/stats subsystem thread_pool + fs paths metric

```shell $ curl -s http://127.0.0.1:9222 Vinted Elasticsearch exporter

Available /cat subsystems: - catallocation - catshards - catindices - catsegments - catnodes - catrecovery - cathealth - catpendingtasks - cataliases - catthreadpool - catplugins - catfielddata - catnodeattrs - catrepositories - cattemplates - cattransforms Available /cluster subsystems: - clusterhealth Available /nodes subsystems: - nodesusage - nodesstats - nodesinfo Available /stats subsystems: - stats

Exporter settings: elasticsearchurl: http://127.0.0.1:9200 elasticsearchglobaltimeout: 30s elasticsearchqueryfields: elasticsearchsubsystemtimeouts: - nodesstats: 15s elasticsearchpathparameters: - nodesinfo: http,jvm,threadpool - nodesstats: breaker,indices,jvm,os,process,transport,threadpool exporterskiplabels: - catallocation: health,status - catfielddata: id - catindices: health,status - catnodeattrs: id - catnodes: health,status,pid - catplugins: id,description - catsegments: health,status,checkpoint,prirep - catshards: health,status,checkpoint,prirep - cattemplates: composedof - catthreadpool: nodeid,ephemeralnodeid,pid - cattransforms: health,status - clusterstats: segment,patterns exporterincludelabels: - cataliases: index,alias - catallocation: node - catfielddata: node,field - cathealth: shards - catindices: index - catnodeattrs: node,attr - catnodes: ip,name,noderole - catpendingtasks: index - catplugins: name - catrecovery: index,shard,stage,type - catrepositories: index - catsegments: index,shard - catshards: index,node,shard - cattemplates: name,indexpatterns - catthreadpool: nodename,name,type - cattransforms: index - clusterhealth: status - nodesinfo: name - nodesstats: name - nodesusage: name - stats: index exporterskipmetrics: - cataliases: filter,routingindex,routingsearch,iswriteindex - catnodeattrs: pid - catrecovery: starttime,starttimemillis,stoptime,stoptimemillis - cattemplates: order - nodesusage: _nodestotal,nodessuccessful,since exporterpolldefaultinterval: 15s exporterpollintervals: - clusterhealth: 5s exporterskipzerometrics: true exportermetricsenabled: - cathealth: true - catindices: true - nodesinfo: true - nodesstats: true exportermetadatarefreshinterval: 180s exportermetricslifetimedefaultinterval: 15s exportermetricslifetimeinterval: - catindices: 180s - catnodes: 60s - catrecovery: 60s ```

Self exporter metrics

```

HELP elasticsearchsubsystemrequestdurationseconds The Elasticsearch subsystem request latencies in seconds.

TYPE elasticsearchsubsystemrequestdurationseconds histogram

elasticsearchsubsystemrequestdurationsecondsbucket{cluster="devnull",subsystem="/nodes/os",le="0.005"} 0 elasticsearchsubsystemrequestdurationsecondssum{cluster="devnull",subsystem="/nodesstats"} 0.130069193 elasticsearchsubsystemrequestdurationsecondscount{cluster="devnull",subsystem="/nodesstats"} 1

HELP httprequestduration_seconds The HTTP request latencies in seconds.

TYPE httprequestduration_seconds histogram

httprequestdurationsecondsbucket{handler="/metrics",le="0.005"} 1 httprequestdurationsecondssum{handler="/metrics"} 0.004372555 httprequestdurationsecondscount{handler="/metrics"} 1

HELP processcpuseconds_total Total user and system CPU time spent in seconds.

TYPE processcpuseconds_total counter

processcpuseconds_total 0.24

HELP processmaxfds Maximum number of open file descriptors.

TYPE processmaxfds gauge

processmaxfds 1024

HELP processopenfds Number of open file descriptors.

TYPE processopenfds gauge

processopenfds 16

HELP processresidentmemory_bytes Resident memory size in bytes.

TYPE processresidentmemory_bytes gauge

processresidentmemory_bytes 25006080

HELP processstarttime_seconds Start time of the process since unix epoch in seconds.

TYPE processstarttime_seconds gauge

processstarttime_seconds 1605894185.46

HELP processvirtualmemory_bytes Virtual memory size in bytes.

TYPE processvirtualmemory_bytes gauge

processvirtualmemory_bytes 1345773568 ```

Debug

Levels: info,warn,error,debug,trace

To debug HTTP requests

export RUST_LOG=info,reqwest=debug

To trace everything

export RUST_LOG=trace

Development

To start:

shell cargo run --bin elasticsearch_exporter

To test:

shell cargo test

License

MIT