Elasticsearch exporter

Prometheus Elasticsearch exporter capable of working with large clusters. Caution you may overload Prometheus server by enabling all metrics, exporter is capable to export over near 1 million metrics. To avoid overloading Prometheus server run multiple Elasticsearch exporters that target just few specific metrics.

bash $ curl -s http://127.0.0.1:9222/metrics | wc 940272 1887011 153668390

Try it out with docker

bash $ docker run --network=host -it vinted/elasticsearch_exporter --elasticsearch_url=http://IP:PORT

Features

• Metric collection is decoupled from serving /metrics page
• Skips zero/empty metrics (controlled with flag exporter_allow_zero_metrics)
• Elasticsearch "millis" converted to seconds
• Elasticsearch "kilobytes" converted to bytes
• All time based metrics are converted as f64 seconds, keywords millis replaced with seconds
• Added _bytes and _seconds postfix
• Preserves metrics tree namespace up to last leaf
  • elasticsearchcatindicespriwarmertotaltimesecondsbucket
  • elasticsearchcathealth_unassign
  • elasticsearchnodesinfojvmmemheapmaxinbytes
• Custom namespace labels vin_cluster_version for convenient comparison of metrics between cluster versions
• Automatic cluster metadata updates every 5 minutes
• /_nodes/* API additional labels injection by mapping node ID to fetched cluster metadata
  • name (map from node ID to name) namespaced -> name
  • version (Elasticsearch node version) namespaced to vin_cluster_version
  • IP namespaced -> ip
• Automatic metrics deletion based on lifetime settings, by default metric by value will be deleted after 600s since last occurrence.
• Metric names are normalized to snake case, colon is replaced with underscore, brackets are replaced with colon (:)
  • "transportactionscluster:monitor/nodes/info[n]requestscount" -> "transportactionsclustermonitornodesinfo:n:requests_count"
  • "transportactionsinternal:cluster/coordination/join/pingrequestscount" -> "transportactionsinternalclustercoordinationjoinpingrequestscount"

Options

• Configurable labels "skip" and/or "include" (flags: exporter_include_labels, exporter_skip_labels)
• Configurable skip metrics (controlled with (flag exporter_skip_metrics)
• Configurable global timeout (flag elasticsearch_global_timeout)
• Configurable global polling interval (flag exporter_poll_default_interval)
• Configurable per metric polling interval (flag exporter_poll_intervals)
• Configurable metrics collection (flag exporter_metrics_enabled)
• Configurable metadata collection (flag exporter_metadata_refresh_interval)

Usage cheat sheet

Scraping /_nodes/stats subsystem thread_pool path metric

$ docker run --network=host -it vinted/elasticsearch_exporter --elasticsearch_url=http://IP:PORT --exporter_metrics_enabled="nodes_stats=true" --elasticsearch_path_parameters="nodes_stats=thread_pool"

Scraping /_nodes/stats subsystem thread_pool + fs paths metric

$ docker run --network=host -it vinted/elasticsearch_exporter --elasticsearch_url=http://IP:PORT --exporter_metrics_enabled="nodes_stats=true" --elasticsearch_path_parameters="nodes_stats=thread_pool,fs"

Scraping /stats for total.indexing and total.search metrics only

$ docker run --network=host -it vinted/elasticsearch_exporter --elasticsearch_url=http://IP:PORT --exporter_metrics_enabled="stats=true" --elasticsearch_query_filter_path="stats=indices.*.total.indexing,indices.*.total.search"

Scraping /_cat/shards for search.fetch* metrics only. In this case elasticsearch_query_filter_path must always include index,shard, and dotted format is not supported. Example:

$ docker run --network=host -it vinted/elasticsearch_exporter --elasticsearch_url=http://IP:PORT --exporter_metrics_enabled="cat_shards=true" --elasticsearch_query_filter_path="cat_shards=index,shard,search*fetch*"

```shell $ curl -s http://127.0.0.1:9222 Vinted Elasticsearch exporter

Available /cat subsystems: - catallocation - catshards - catindices - catsegments - catnodes - catrecovery - cathealth - catpendingtasks - cataliases - catthreadpool - catplugins - catfielddata - catnodeattrs - catrepositories - cattemplates - cattransforms Available /cluster subsystems: - clusterhealth Available /nodes subsystems: - nodesusage - nodesstats - nodesinfo Available /stats subsystems: - stats

Exporter settings: elasticsearchurl: http://127.0.0.1:9200 elasticsearchglobaltimeout: 30s elasticsearchqueryfields: elasticsearchsubsystemtimeouts: - nodesstats: 15s elasticsearchpathparameters: - nodesinfo: http,jvm,threadpool - nodesstats: breaker,indices,jvm,os,process,transport,threadpool exporterskiplabels: - catallocation: health,status - catfielddata: id - catindices: health,status - catnodeattrs: id - catnodes: health,status,pid - catplugins: id,description - catsegments: health,status,checkpoint,prirep - catshards: health,status,checkpoint,prirep - cattemplates: composedof - catthreadpool: nodeid,ephemeralnodeid,pid - cattransforms: health,status - clusterstats: segment,patterns exporterincludelabels: - cataliases: index,alias - catallocation: node - catfielddata: node,field - cathealth: shards - catindices: index - catnodeattrs: node,attr - catnodes: ip,name,noderole - catpendingtasks: index - catplugins: name - catrecovery: index,shard,stage,type - catrepositories: index - catsegments: index,shard - catshards: index,node,shard - cattemplates: name,indexpatterns - catthreadpool: nodename,name,type - cattransforms: index - clusterhealth: status - nodesinfo: name - nodesstats: name - nodesusage: name - stats: index exporterskipmetrics: - cataliases: filter,routingindex,routingsearch,iswriteindex - catnodeattrs: pid - catrecovery: starttime,starttimemillis,stoptime,stoptimemillis - cattemplates: order - nodesusage: _nodestotal,nodessuccessful,since exporterpolldefaultinterval: 15s exporterpollintervals: - clusterhealth: 5s exporterskipzerometrics: true exportermetricsenabled: - cathealth: true - catindices: true - nodesinfo: true - nodesstats: true exportermetadatarefreshinterval: 180s exportermetricslifetimedefaultinterval: 15s exportermetricslifetimeinterval: - catindices: 180s - catnodes: 60s - catrecovery: 60s ```

Self exporter metrics

```

HELP elasticsearchsubsystemrequestdurationseconds The Elasticsearch subsystem request latencies in seconds.

TYPE elasticsearchsubsystemrequestdurationseconds histogram

elasticsearchsubsystemrequestdurationsecondsbucket{cluster="devnull",subsystem="/nodes/os",le="0.005"} 0 elasticsearchsubsystemrequestdurationsecondssum{cluster="devnull",subsystem="/nodesstats"} 0.130069193 elasticsearchsubsystemrequestdurationsecondscount{cluster="devnull",subsystem="/nodesstats"} 1

HELP httprequestduration_seconds The HTTP request latencies in seconds.

TYPE httprequestduration_seconds histogram

httprequestdurationsecondsbucket{handler="/metrics",le="0.005"} 1 httprequestdurationsecondssum{handler="/metrics"} 0.004372555 httprequestdurationsecondscount{handler="/metrics"} 1

HELP processcpuseconds_total Total user and system CPU time spent in seconds.

TYPE processcpuseconds_total counter

processcpuseconds_total 0.24

HELP processmaxfds Maximum number of open file descriptors.

TYPE processmaxfds gauge

processmaxfds 1024

HELP processopenfds Number of open file descriptors.

TYPE processopenfds gauge

processopenfds 16

HELP processresidentmemory_bytes Resident memory size in bytes.

TYPE processresidentmemory_bytes gauge

processresidentmemory_bytes 25006080

HELP processstarttime_seconds Start time of the process since unix epoch in seconds.

TYPE processstarttime_seconds gauge

processstarttime_seconds 1605894185.46

HELP processvirtualmemory_bytes Virtual memory size in bytes.

TYPE processvirtualmemory_bytes gauge

processvirtualmemory_bytes 1345773568 ```

Debug

Levels: info,warn,error,debug,trace

To debug HTTP requests

export RUST_LOG=info,reqwest=debug

To trace everything

export RUST_LOG=trace

Development

To start:

shell cargo run --bin elasticsearch_exporter

To test:

shell cargo test

License

MIT