Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v2.5 #59

Merged
merged 14 commits into from
Jan 17, 2025
Merged

v2.5 #59

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
66d909b
use PrepareConfigForLogs for RegistryClientCreateError
IharYakimush Dec 6, 2024
c037379
24 consume and produce in one transaction (#48)
IharYakimush Dec 10, 2024
ac4c590
Added produce and consume sample
IharYakimush Dec 20, 2024
990cc49
update readme
IharYakimush Dec 20, 2024
7a929dd
Consume and produce in one transaction #24
IharYakimush Jan 10, 2025
d983394
Update dependencies to Confluent v2.8 #50 (#52)
IharYakimush Jan 10, 2025
f0c2884
Validation for consumer groups usage #51 (#53)
IharYakimush Jan 13, 2025
da50776
Validation for consumer groups usage #51
IharYakimush Jan 15, 2025
7ac066b
Mock broker #56 (#57)
IharYakimush Jan 16, 2025
8ba6498
54 add samples and update docs (#58)
IharYakimush Jan 16, 2025
7410843
publication samples
IharYakimush Jan 16, 2025
07a63e4
add samples
IharYakimush Jan 16, 2025
333b397
update samples
IharYakimush Jan 16, 2025
4f30fc4
update sample
IharYakimush Jan 16, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .editorconfig
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,7 @@ dotnet_style_qualification_for_property = true:suggestion
dotnet_style_qualification_for_method = true:suggestion
dotnet_style_qualification_for_event = true:suggestion
csharp_style_unused_value_expression_statement_preference = discard_variable:none
dotnet_diagnostic.IDE0045.severity = none
dotnet_diagnostic.IDE0055.severity = none
dotnet_diagnostic.IDE0057.severity = none
dotnet_diagnostic.CA1051.severity = warning
Expand Down
8 changes: 6 additions & 2 deletions .github/workflows/dotnet.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,13 @@ jobs:
with:
dotnet-version: 9.0.x
- name: Restore
run: dotnet restore
run: |
dotnet restore
dotnet restore sample
- name: Build
run: dotnet build -c Debug --no-restore
run: |
dotnet build -c Debug --no-restore
dotnet build sample -c Debug --no-restore

- name: Start containers
if: steps.filter.outputs.tst == 'true'
Expand Down
9 changes: 0 additions & 9 deletions Epam.Kafka.sln
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Src", "Src", "{E0525F24-5B2
src\Directory.Build.props = src\Directory.Build.props
EndProjectSection
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Sample", "Sample", "{07C5B87A-BBE0-4E3E-83CB-8A95B2D6855E}"
EndProject
Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Tests", "Tests", "{837928B6-19AD-471C-A875-912C7F1F97C7}"
ProjectSection(SolutionItems) = preProject
tests\Directory.Build.props = tests\Directory.Build.props
Expand All @@ -36,8 +34,6 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Epam.Kafka.PubSub.Tests", "
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Epam.Kafka.PubSub.EntityFrameworkCore.Tests", "tests\Epam.Kafka.PubSub.EntityFrameworkCore.Tests\Epam.Kafka.PubSub.EntityFrameworkCore.Tests.csproj", "{B89145BF-B42C-4674-BDD5-61A9BFEB4AA3}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Epam.Kafka.Sample", "sample\Epam.Kafka.Sample\Epam.Kafka.Sample.csproj", "{46C55727-E63E-4571-BA2B-5C5A5A5D6EC9}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Epam.Kafka.PubSub.IntegrationTests", "tests\Epam.Kafka.PubSub.IntegrationTests\Epam.Kafka.PubSub.IntegrationTests.csproj", "{2E0E5146-9022-41A4-A8F6-7460F69639B1}"
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Epam.Kafka.PubSub.EntityFramework6", "src\Epam.Kafka.PubSub.EntityFramework6\Epam.Kafka.PubSub.EntityFramework6.csproj", "{191052C8-A238-40F2-9322-A8B98CECA156}"
Expand Down Expand Up @@ -103,10 +99,6 @@ Global
{B89145BF-B42C-4674-BDD5-61A9BFEB4AA3}.Debug|Any CPU.Build.0 = Debug|Any CPU
{B89145BF-B42C-4674-BDD5-61A9BFEB4AA3}.Release|Any CPU.ActiveCfg = Release|Any CPU
{B89145BF-B42C-4674-BDD5-61A9BFEB4AA3}.Release|Any CPU.Build.0 = Release|Any CPU
{46C55727-E63E-4571-BA2B-5C5A5A5D6EC9}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{46C55727-E63E-4571-BA2B-5C5A5A5D6EC9}.Debug|Any CPU.Build.0 = Debug|Any CPU
{46C55727-E63E-4571-BA2B-5C5A5A5D6EC9}.Release|Any CPU.ActiveCfg = Release|Any CPU
{46C55727-E63E-4571-BA2B-5C5A5A5D6EC9}.Release|Any CPU.Build.0 = Release|Any CPU
{2E0E5146-9022-41A4-A8F6-7460F69639B1}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{2E0E5146-9022-41A4-A8F6-7460F69639B1}.Debug|Any CPU.Build.0 = Debug|Any CPU
{2E0E5146-9022-41A4-A8F6-7460F69639B1}.Release|Any CPU.ActiveCfg = Release|Any CPU
Expand Down Expand Up @@ -138,7 +130,6 @@ Global
{C74D6423-77F4-4778-8C25-9B4EA5512841} = {E0525F24-5B2C-4D26-A93F-6817B1540C87}
{1C278B5D-4685-4677-8895-90244028BEB3} = {837928B6-19AD-471C-A875-912C7F1F97C7}
{B89145BF-B42C-4674-BDD5-61A9BFEB4AA3} = {837928B6-19AD-471C-A875-912C7F1F97C7}
{46C55727-E63E-4571-BA2B-5C5A5A5D6EC9} = {07C5B87A-BBE0-4E3E-83CB-8A95B2D6855E}
{2E0E5146-9022-41A4-A8F6-7460F69639B1} = {222BCFF6-0433-4BE2-B19F-800790CC9261}
{191052C8-A238-40F2-9322-A8B98CECA156} = {E0525F24-5B2C-4D26-A93F-6817B1540C87}
{9BF7230E-5647-44F5-A7E5-CAA5C80A5A94} = {837928B6-19AD-471C-A875-912C7F1F97C7}
Expand Down
12 changes: 11 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,21 @@ Also it is a framework for building pub/sub batch processing applications.
* Assign and external offsets: `IExternalOffsetsStorage` interface implementation for offsets storage. Partition assigmnent based on configuration.
* Default implementation for `IExternalOffsetsStorage` using EntityFramework (Core and EF6).
* Default implementation for subscription that store data using EntityFramework (Core and EF6). Possibility to commit data and offsets in same database transaction.
* Default implementation for subscription that consume data from kafka, then process it, and finally publish result to kafka. In case of kafka internal offsets storage and publishing to same cluster from which data was consumed it is possible to update offsets and publish result in same transaction.

### Publication specific

* Default implementation to publish data from EntityFramework (Core and EF6) context.
* Support for transactional producers.
* Support for transactional producers.

## Samples

* Dependency injection for Confluent.Kafka consumer, producer, client with named configs and options pattern. https://github.com/epam/epam-kafka/tree/develop/sample/ProduceAndConsume
* Read data from Kafka, proccess it using custom handler, store processed offsets in Kafka, use consumer group for rebalance. https://github.com/epam/epam-kafka/tree/develop/sample/Subscribe
* Read data from Kafka, proccess it using custom handler, store processed offsets in DB using Entity Framework Core, optionally use consumer group for rebalance. https://github.com/epam/epam-kafka/tree/develop/sample/SubscribeEfCoreOffsets
* Read data from Kafka, proccess it and save in DB using Entity Framework Core, store processed offsets in DB using Entity Framework Core (optionally commit offsets and processed data in same DB transaction), optionally use consumer group for rebalance. https://github.com/epam/epam-kafka/tree/develop/sample/SubscribeEfCore
* Read data from DB using Entity framework core, convert to message, publish message to kafka, and finally update row state in database. https://github.com/epam/epam-kafka/tree/develop/sample/PublishEfCore
* Read data from DB using Entity framework core, convert to multiple messages, publish messages to kafka in single transaction, and finally update row state in database. https://github.com/epam/epam-kafka/tree/develop/sample/PublishEfCoreTransactional

## Packages

Expand Down
2 changes: 2 additions & 0 deletions build.cmd
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
dotnet restore
dotnet build -c Debug --no-restore
dotnet restore sample
dotnet build sample -c Debug --no-restore
dotnet test -c Debug --no-build -f net8.0 --verbosity normal --settings .runsettings
dotnet test -c Debug --no-build -f net6.0 --verbosity normal --settings .runsettings
dotnet test -c Debug --no-build -f net462 --verbosity normal --settings .runsettings
Expand Down
31 changes: 0 additions & 31 deletions sample/Epam.Kafka.Sample.Net462/App.config
View file
Open in desktop

This file was deleted.

11 changes: 0 additions & 11 deletions sample/Epam.Kafka.Sample.Net462/Data/KafkaEntity.cs
View file
Open in desktop

This file was deleted.

28 changes: 0 additions & 28 deletions sample/Epam.Kafka.Sample.Net462/Data/SampleDbContext.cs
View file
Open in desktop

This file was deleted.

19 changes: 0 additions & 19 deletions sample/Epam.Kafka.Sample.Net462/Data/SamplePublicationEntity.cs
View file
Open in desktop

This file was deleted.

15 changes: 0 additions & 15 deletions sample/Epam.Kafka.Sample.Net462/Data/SampleSubscriptionEntity.cs
View file
Open in desktop

This file was deleted.

Loading
Loading