Kafka Connect Sandbox

This repository refers to the code mentioned in the article available at https://estevaosaleme.medium.com/a-sandbox-for-kafka-connect-connectors-experimentation-2e41f99aae15

What does this sandbox do?

Spin up Apache Kafka.
Build and spin up Kafka Connect.
Spin up Kafkdrop - a UI for Apache Kafka.
Spin up Confluent Kafka REST Proxy.
Spin up a Prometheus server.
Health check on Apache Kafka.
Health check on Kafka Connect.
Read data from a file and ingest data to Kafka (File source test).
Read data from kafka and output it to a local file (File sink test).
Read data from kafka and output it a database (Postgres sink test).
Check the status of the source and sink connectors using Kafka Connect's API.

Getting started

First, build the Kafka Connect image using docker-compose:

docker-compose build

Then, you should be able to run the environment:

docker-compose up

Accessible endpoints

Kafka Connect connectors

http://localhost:8083/connector-plugins

Confluent REST Proxy

http://localhost:8082

Kafdrop - Kafka UI

http://localhost:9020

Prometheus UI endpoint

http://localhost:9090

References

Apache Kafka 3.7 documentation: https://kafka.apache.org/37/documentation.html
Kafka Connect FileStream Connectors: https://docs.confluent.io/platform/current/connect/filestream_connector.html
JDBC Sink Connector for Confluent Platform: https://docs.confluent.io/kafka-connectors/jdbc/current/sink-connector/overview.html
Confluent REST Proxy for Apache Kafka: https://docs.confluent.io/platform/current/kafka-rest/index.html