Kommitted - Measure Kafka consumers lag

Measure Kafka Consumer **Offset Lag** _and_ **Time Lag**

Kommitted is a service to measure the Lag (i.e. Latency) of Kafka consumers. It works with all consumers that commit their offsets into Kafka (i.e. the standard way), as it consumes the internal __consumer_offsets topic.

Metrics are exported following the Prometheus Exposition formats.

Please see DESIGN.md for details about the overall architecture, dependencies and other details.

Features

[x] Track Offset for all consumers
[x] Track Offset Lag for all consumers
[x] Track Time Lag for all consumers
[x] Offset and Lag metrics are tracked with all contextual information to identify exact topic partition assignments
[ ] Exposes additional metrics to track status of Kafka cluster (topics, members, brokers, partitions)
[ ] Exposes Kafka-polling metrics, to assess its own performance
[x] Metrics exposed in Prometheus format, at /metrics endpoint
[ ] REST API to build further automation on top of it (e.g. auto-scaling logics that depend on Consumer Group lag)

All of this comes based on:

A fast and efficient Rust implementation, built on Tokio
The widely used librdkafka, the de-facto standard for Kafka Clients (outside of Java)

Please see the complete list of exposed Metrics, for further details.

Getting started

WIP installation

In Docker

WIP

Usage

Kommitted supports compact (-h) and extended (--help) usage instructions. Use the former for a quick look up; use the latter to better understand what each argument can do.

Compact: kommitted -h

```shell Usage: kommitted [OPTIONS] --brokers

Options:
-b, --brokers <BOOTSTRAP_BROKERS>     Initial Kafka Brokers to connect to (format: 'HOST:PORT,...')
--client-id <CLIENT_ID>           Client identifier used by the internal Kafka (Admin) Client [default: kommitted]
--kafka-conf <CONF_KEY:CONF_VAL>  Additional configuration used by the internal Kafka (Admin) Client (format: 'CONF_KEY:CONF_VAL').
--cluster-id <CLUSTER_ID>         Override identifier of the monitored Kafka Cluster
--history <SIZE>                  For each Topic Partition, how much history of offsets to track in memory. [default: 3600]
-l, --listen-on <HOST:PORT>           Address to listen on (i.e. bind to), to receive HTTP requests. [default: localhost:9090]
-v, --verbose...                      Verbose logging.
-q, --quiet...                        Quiet logging.
-h, --help                            Print help (see more with '--help')
-V, --version                         Print version

```

Extended: kommitted --help

```shell Usage: kommitted [OPTIONS] --brokers

Options: -b, --brokers Initial Kafka Brokers to connect to (format: 'HOST:PORT,...').

        Equivalent to '--config=bootstrap.servers:host:port,...'.

    --client-id <CLIENT_ID>
        Client identifier used by the internal Kafka (Admin) Client.

        Equivalent to '--config=client.id:my-client-id'.

        [default: kommitted]

    --kafka-conf <CONF_KEY:CONF_VAL>
        Additional configuration used by the internal Kafka (Admin) Client (format: 'CONF_KEY:CONF_VAL').

        To set multiple configurations keys, use this argument multiple times.
        See: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md.

    --cluster-id <CLUSTER_ID>
        Override identifier of the monitored Kafka Cluster.

        If set, it replaces the value `cluster.id` from the Brokers' configuration. This can be useful when `cluster.id` is not actually set.

    --history <SIZE>
        For each Topic Partition, how much history of offsets to track in memory.

        Offsets data points are collected every 500ms, on average: so, on average,
        30 minutes of data points is 3600 offsets, assuming partition offsets are
        regularly produced to.

        Once this limit is reached, the oldest data points are discarded, realising
        a "moving window" of offsets history.

        [default: 3600]

-l, --listen-on <HOST:PORT>
        Address to listen on (i.e. bind to), to receive HTTP requests.

        In addition to the canonical 'HOST:PORT' format, it also allows for:

        * ':PORT' / 'PORT' (assumes default 'HOST')
        * 'HOST:' / 'HOST' (assumes default 'PORT')
        * ':'              (fallback on default)

        [default: localhost:9090]

-v, --verbose...
        Verbose logging.

        * none    = 'WARN'
        * '-v'    = 'INFO'
        * '-vv'   = 'DEBUG'
        * '-vvv'  = 'TRACE'

        Alternatively, set environment variable 'KOMMITTED_LOG=(ERROR|WARN|INFO|DEBUG|TRACE|OFF)'.

-q, --quiet...
        Quiet logging.

        * none    = 'WARN'
        * '-q'    = 'ERROR'
        * '-qq'   = 'OFF'

        Alternatively, set environment variable 'KOMMITTED_LOG=(ERROR|WARN|INFO|DEBUG|TRACE|OFF)'.

-h, --help
        Print help (see a summary with '-h')

-V, --version
        Print version

```

Connect to Kafka cluster requiring `SASL_SSL`

shell $ kommitted \ --brokers {{ BOOTSTRAP_BROKERS or BROKER_ENDPOINT }} \ --config security.protocol:SASL_SSL \ --config sasl.mechanisms=PLAIN \ --config sasl.username:{{ USERNAME or API_KEY }} \ --config sasl.password:{{ PASSWORD or API_SECRET }} \ ...

Log verbosity

Ksunami follows the long tradition of -v/-q to control the verbosity of it's logging:

| Arguments | Log verbosity level | Default | |----------:|:--------------------|:-------:| | -qq... | OFF | | | -q | ERROR | | | none | WARN | x | | -v | INFO | | | -vv | DEBUG | | | -vvv... | TRACE | |

It uses log and envlogger, and so logging can be configured and fine-tuned using the Environment Variable KOMMITTED_LOG. Please take a look at envlogger doc for more details.

License

Licensed under either of

Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.