Skip to content

CDC-to-Kafka 4.0.0
Compare
Choose a tag to compare
@woodlee woodlee released this 08 Aug 21:01
· 16 commits to master since this release
v4.0.0
cc559bb
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

The primary focus of this new version was to adopt Kafka transactions to improve the reliability of exactly-once producing, and also to ensure that messages produced to "unified" topics have the same guarantees as those produced to the corresponding single-table topics. Some breaking changes, noted below, led to the major-version bump:

  1. In the config options, "whitelist" / "blacklist" language has been removed, in favor of "include" / "exclude". So e.g., the option TABLE_WHITELIST_REGEX is now TABLE_INCLUDE_REGEX.
  2. To support the use of Kafka transactions, a new required config parameter KAFKA_TRANSACTIONAL_ID has been added. You should set this to something that will remain stable across restarts of a particularly-configured instance of this process.
  3. If you are currently using the KafkaReporter as one of the metrics reporters configured in METRICS_REPORTERS, be aware that the Avro schema of the messages it produces has changed in a breaking way in this release. You may wish to switch to a new metrics topic, or to temporarily relax schema compatibility checking in the schema registry for your existing metrics topic.

Other changes:

  1. Improved type checking in code; now passes mypy analysis.
  2. Upgraded to Python 3.11 for the Docker image
  3. Upgraded 3rd-party dependencies
  4. Faster startup time
  5. Improvements in validation/testing scripts
Assets 2
Loading