Restructure kafka client into backup #87
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
About this change - What it does
This PR restructures some of the components in Guardian so that the dependencies are minimized. This is mainly a result of working on #83, as it turns there are configurations/components which are mandatory that shouldn't be, or are in the wrong module.
Regarding the cli feature, without this PR we run into the problem where we are forced to write configurations which dont make sense (i.e.
compaction-bucket
for backup) or in other cases we there is no configuration for options we do need to provide (i.e.group-id
)Why this way
The PR is split into 3 commits, each complaining the reasoning behind the change
Move KafkaClient into core-backup
:KafkaClient
is a KafkaConsumer that listens to a Kafka cluster, strictly speaking this functionality is only needed for backing up of data hence why it was moved tocore-backup
instead of justcore
. One of the reasons behind this change was the fact that certain Kafka configurations such asbootstrap-servers
are global for every component but others such asgroup-id
(aka consumer group) only exist for Kafka consumers and not producers (i.e. restore).Add kafka-group-id config into Backup config
: This addsgroup-id
configuration as a core config into thebackup-core
Backup
configuration. Interestingly Kafka configuration with alpakka is all over the place, some configurations such asbootstrap-servers
have a default config inreference.conf
that can be overridden, others such asgroup-id
have no Typesafe configuration at all. Since there is no non-programmatic way to configuregroup-id
we added it into our own config.Remove compaction-bucket config from core-s3
: This configuration was placed into the wrong project, it should rather belong incompaction-s3
/restore-s3
since thats the only place where this bucket will be used. Note that only theS3
case class that is serialized from the typesafe config had the configuration option removed, the path for thecompaction-bucket
will still be contained within thes3-config
path inreference.conf
its just that the config will be placed in thereference.conf
of other projects (and this will work fine since typesafe config naturally supports merging of configs).