Kafka replicator

Kafka Replicator is an easy to use tool for copying data between two Apache Kafka clusters with configurable re-partitionning strategy.

Data will be read from topics in the origin cluster and written to a topic/topics in the destination cluster according config rules.

Features

Lets start with an overview of features that exist in kafka-replicator:

• [x] Data replication: Real-time event streaming between Kafka clusters and data centers;
• [ ] Schema replication: Copy schema from source cluster to destination;
• [x] Flexible topic selection: Select topics with configurable config;
• [ ] Auto-create topics: Destination topics are automatically created for strict_p2p strategy;
• [x] Stats: The tool shows replication status;
• [ ] Monitoring: Kafka replicator exports stats via prometheus.
• [ ] Cycle detection

Use cases

• Replicate data between Kafka clusters;
• Aggregate record from several topics and put them into one;
• Extend bandwidth for exist topic via repartitioning strategy.

Installation

System dependencies

shell libsasl2-dev libssl-dev

Install from crates.io

If you have the Rust toolchain already installed on your local system.

shell rustup update stable cargo install kafka-replicator

Compile and run it from sources

Clone the repository and change it to your working directory.

```shell git clone https://github.com/lispython/kafka-replicator.git cd kafka-replicator

rustup override set stable rustup update stable cargo install ```

Usage

shell RUST_LOG=info kafka-replicator /path/to/config.yml

Run it using Docker

shell sudo docker run -it -v /replication/:/replication/ -e RUST_LOG=info lispython/kafka_replicator:latest kafka-replicator /replication/config.yml

Example config

``` yaml clusters: - name: cluster1 hosts: - replicator-kafka-1:9092 - replicator-kafka-1:9092 - name: cluster2 hosts: - replicator-kafka-2:9092

clients: - client: cl1client1 cluster: cluster1 config: # optional message.timeout.ms: 5000 auto.offset.reset: earliest - client: cl2client1 cluster: cluster2

routes: - upstreamclient: cl1client1 downstreamclient: cl1client1 upstreamtopics: - 'topic1' downstreamtopic: 'topic2' repartitioningstrategy: random # strictp2p | random upstreamgroupid: group22 progresseverysecs: 10 limits: messagespersec: 10000 numberof_messages:

• upstreamclient: cl1client1 downstreamclient: cl2client1 upstream_topics:

  • 'topic2' downstreamtopic: 'topic2' repartitioningstrategy: strictp2p upstreamgroupid: group22 progresseverysecs: 10

• upstreamclient: cl2client1 downstreamclient: cl1client1 upstream_topics:

  • 'topic2' downstreamtopic: 'topic3' repartitioningstrategy: strictp2p # strictp2p | random defaultbeginoffset: earliest # optional upstreamgroupid: group2 progressevery_secs: 10

observers: - client: cl1client1 name: "my name" topics: - 'topic1' - 'topic2' fetchtimeoutsecs: 5 showprogresseverysecs: 10

• client: cl2client_1 topic: 'topic3' topics:

  • 'topic2' showprogressevery_secs: 5

• client: cl1client_1 topic: 'topic1' topics: [] ```

Options describing

Root config options: - clusters - are a list of Kafka Clusters - clients - are a list of configurations for consumers - routes - are a list of replication rules - observers - are a list of observers

Contributing

Any suggestion, feedback or contributing is highly appreciated. Thank you for your support!