Feat[Storagetool]: List queue appid numrecords #534

bbpetukhov · 2024-11-29T17:54:57Z

NB: This PR depends on #531 and will be ready for review as soon as #531 is merged

Describe your changes
Added new functional to bmqstoragetool to list number of all types of records per queue.
Supported resolution of Queue Key and App Key.
Added extra cmd line arg --min-records-per-queue <threshold> to restrict displaying queues with low number of records (less than given threshold)

Testing performed

Manual testing with a few real journal file and corresponding CSL files.
with/without CSL file to make sure that Queue Keys resolution working.
Added a new Unit Test to check storage tool output

Additional context

Usage example:
./bmqstoragetool.tsk --summary --min-records-per-queue 1012012 --journal-file file.bmq_journal --csl-file file.bmq_csl

Sample Output (section related to this PR):

...
Number of records per Queue:
    Queue                 : 1234567890
    Total Records         : 333334
    Num Queue Op Records  : 1
    Num Message Records   : 111111
    Num Confirm Records   : 111111
    Num Delete Records    : 111111
    Queue                 : 1234567890A
    Total Records         : 1111481
    Num Queue Op Records  : 881
    Num Message Records   : 100
    Num Confirm Records   : 670
    Num Records Per App   : 1234567890B=80 1234567890C=90 1234567890D=100 1234567890E=100 1234567890F=100 12345678910=100 12345678911=100 
    Num Delete Records    : 100

Sample Output showcasing the resolution of Queue Key and App Key:

Run without CSL file:

Number of records per Queue:
    Queue Key             : E12EF21974
    Total Records         : 5
    Num Queue Op Records  : 1
    Num Message Records   : 1
    Num Confirm Records   : 2
    Num Records Per App   : 793E7A6FD3=1 D6B7D942A8=1
    Num Delete Records    : 1

Run with CSL file (i.e. --csl-file ./bmq_csl_20240930_090323_CF83E8E13A.bmq_csl):

Number of records per Queue:
    Queue Key             : E12EF21974
    Queue URI             : bmq://bmq.test.persistent.fanout/dima
    Total Records         : 5
    Num Queue Op Records  : 1
    Num Message Records   : 1
    Num Confirm Records   : 2
    Num Records Per App   : baz=1 foo=1
    Num Delete Records    : 1

Signed-off-by: Dmitrii Petukhov <[email protected]>

Signed-off-by: Aleksandr Ivanov <[email protected]>

Signed-off-by: Dmitrii Petukhov <[email protected]>

src/applications/bmqstoragetool/bmqstoragetool.m.cpp

src/applications/bmqstoragetool/m_bmqstoragetool_journalfileprocessor.cpp

src/applications/bmqstoragetool/m_bmqstoragetool_parameters.h

src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.h

src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.cpp

Co-authored-by: Alexander Ivanov <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

Signed-off-by: Dmitrii Petukhov <[email protected]>

src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.h

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov · 2025-01-13T10:23:24Z

Merged with upstream main. Ready for review.

src/applications/bmqstoragetool/m_bmqstoragetool_journalfileprocessor.t.cpp

src/applications/bmqstoragetool/m_bmqstoragetool_parameters.h

src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.cpp

src/groups/bmq/bmqu/bmqu_operationchain.h

Signed-off-by: Dmitrii Petukhov <[email protected]>

src/applications/bmqstoragetool/bmqstoragetool.m.cpp

src/applications/bmqstoragetool/m_bmqstoragetool_journalfileprocessor.t.cpp

src/applications/bmqstoragetool/m_bmqstoragetool_parameters.cpp

src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.cpp

src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.h

Co-authored-by: Evgeny Malygin <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

…ocessor.t.cpp Co-authored-by: Evgeny Malygin <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

Signed-off-by: Dmitrii Petukhov <[email protected]>

Co-authored-by: Evgeny Malygin <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov · 2025-01-16T14:43:55Z

@678098 I addressed all the review feedback. Made necessary changes.
All pipeline checks passed.

678098

Looks good

678098

Sanitizer failures

src/applications/bmqstoragetool/m_bmqstoragetool_parameters.h

Signed-off-by: Dmitrii Petukhov <[email protected]>

678098

Let's check sanitizer now

bbpetukhov · 2025-01-17T13:40:31Z

Let's check sanitizer now

I checked in a separate branch: https://github.com/bbpetukhov/blazingmq/actions/runs/12828238944/job/35771862423
The warning is not reproduced anymore.

bbpetukhov and others added 12 commits November 28, 2024 12:17

wip

4a3ab78

Signed-off-by: Dmitrii Petukhov <[email protected]>

Add support for Queue URI

dc981b7

Signed-off-by: Dmitrii Petukhov <[email protected]>

Support printer formatting

8111259

Signed-off-by: Dmitrii Petukhov <[email protected]>

Can calc records by type per queue

c9a7c23

Signed-off-by: Dmitrii Petukhov <[email protected]>

Calculate per records counts app key

4d091fe

Signed-off-by: Dmitrii Petukhov <[email protected]>

Add ""min records per queue" cmd line parameter

90ab10a

Signed-off-by: Dmitrii Petukhov <[email protected]>

Move printRecord methods to RecordPrinter namespace

fd373de

Signed-off-by: Aleksandr Ivanov <[email protected]>

Resolve AppKey to AppId

0757a2a

Signed-off-by: Dmitrii Petukhov <[email protected]>

Update README.md

5b99185

Signed-off-by: Dmitrii Petukhov <[email protected]>

Sort app ids ny num records in asc order

059aacd

Signed-off-by: Dmitrii Petukhov <[email protected]>

Add comments

621ca50

Signed-off-by: Dmitrii Petukhov <[email protected]>

Run clang-format

185194d

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov force-pushed the list-queue-appid-numrecords branch from 608ae0c to 185194d Compare November 29, 2024 18:16

alexander-e1off reviewed Dec 2, 2024

View reviewed changes

bbpetukhov and others added 6 commits December 2, 2024 12:42

Update src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.cpp

7f3fa10

Co-authored-by: Alexander Ivanov <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

Update src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.cpp

5734e06

Co-authored-by: Alexander Ivanov <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

Update src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.cpp

12dfb48

Co-authored-by: Alexander Ivanov <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

Always print Queue Key in detailed queue information

55fdc36

Signed-off-by: Dmitrii Petukhov <[email protected]>

Refactor method to calc total records count

079d26c

Signed-off-by: Dmitrii Petukhov <[email protected]>

Prefer const_iterator

b0de035

Signed-off-by: Dmitrii Petukhov <[email protected]>

alexander-e1off reviewed Dec 2, 2024

View reviewed changes

src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.h Outdated Show resolved Hide resolved

Make min-records-per-queue parameter unsigned

7295527

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov changed the title ~~List queue appid numrecords~~ Feat[Storagetool]: List queue appid numrecords Dec 2, 2024

Run clang-format

959f269

Signed-off-by: Dmitrii Petukhov <[email protected]>

pniedzielski self-assigned this Dec 2, 2024

Remove redundant Parameters default ctor

1ef7d9b

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov force-pushed the list-queue-appid-numrecords branch from 1d1fc97 to 1ef7d9b Compare December 2, 2024 16:50

Add unit test for queue detailed output

ff069a3

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov force-pushed the list-queue-appid-numrecords branch from 9d70563 to ff069a3 Compare December 3, 2024 13:26

bbpetukhov marked this pull request as ready for review December 3, 2024 17:46

Refactor Parameters class - test creator helper function

e28074b

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov force-pushed the list-queue-appid-numrecords branch from 5b31e2b to e28074b Compare January 13, 2025 09:40

678098 reviewed Jan 13, 2025

View reviewed changes

src/applications/bmqstoragetool/m_bmqstoragetool_journalfileprocessor.t.cpp Show resolved Hide resolved

alexander-e1off reviewed Jan 13, 2025

View reviewed changes

src/applications/bmqstoragetool/m_bmqstoragetool_parameters.h Outdated Show resolved Hide resolved

alexander-e1off reviewed Jan 13, 2025

View reviewed changes

src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.cpp Outdated Show resolved Hide resolved

alexander-e1off reviewed Jan 13, 2025

View reviewed changes

src/groups/bmq/bmqu/bmqu_operationchain.h Outdated Show resolved Hide resolved

bbpetukhov added 4 commits January 15, 2025 13:17

Revert bmqu files

ff44dd3

Signed-off-by: Dmitrii Petukhov <[email protected]>

Remove redundant comments

5dae9e5

Signed-off-by: Dmitrii Petukhov <[email protected]>

Remove redundant virtual methods

4fa70e0

Signed-off-by: Dmitrii Petukhov <[email protected]>

Merge branch 'main' into list-queue-appid-numrecords

0cc6060

678098 requested changes Jan 15, 2025

View reviewed changes

bbpetukhov and others added 3 commits January 16, 2025 11:04

Update src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.h

9ef1aec

Co-authored-by: Evgeny Malygin <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

Update src/applications/bmqstoragetool/m_bmqstoragetool_journalfilepr…

dd8ff03

…ocessor.t.cpp Co-authored-by: Evgeny Malygin <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

Optimize queueInfo variable declaration

79017f2

Signed-off-by: Dmitrii Petukhov <[email protected]>

bbpetukhov force-pushed the list-queue-appid-numrecords branch from fd309e9 to 79017f2 Compare January 16, 2025 10:09

bbpetukhov and others added 6 commits January 16, 2025 12:09

Update src/applications/bmqstoragetool/m_bmqstoragetool_searchresult.h

22f0050

Co-authored-by: Evgeny Malygin <[email protected]> Signed-off-by: Dmitrii Petukhov <[email protected]>

Extend logic of createTestParameters helper function

c6c87dd

Signed-off-by: Dmitrii Petukhov <[email protected]>

Remove redundant function

0c68133

Signed-off-by: Dmitrii Petukhov <[email protected]>

Disable default alloc check in unit tests for SunOS

21f2298

Signed-off-by: Dmitrii Petukhov <[email protected]>

Use zero as default value for records limit

86a267c

Signed-off-by: Dmitrii Petukhov <[email protected]>

Merge branch 'main' into list-queue-appid-numrecords

2b6c8fd

678098 approved these changes Jan 16, 2025

View reviewed changes

678098 requested changes Jan 16, 2025

View reviewed changes

src/applications/bmqstoragetool/m_bmqstoragetool_parameters.h Show resolved Hide resolved

src/applications/bmqstoragetool/m_bmqstoragetool_parameters.h Show resolved Hide resolved

bbpetukhov added 2 commits January 17, 2025 09:27

Merge branch 'main' into list-queue-appid-numrecords

f452ebc

Initialize Command Line arguments fields to fix msan warning

01172e6

Signed-off-by: Dmitrii Petukhov <[email protected]>

678098 approved these changes Jan 17, 2025

View reviewed changes

678098 merged commit 005a33d into bloomberg:main Jan 17, 2025
33 of 40 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat[Storagetool]: List queue appid numrecords #534

Feat[Storagetool]: List queue appid numrecords #534

bbpetukhov commented Nov 29, 2024 •

edited

Loading

bbpetukhov commented Jan 13, 2025

bbpetukhov commented Jan 16, 2025 •

edited

Loading

678098 left a comment

678098 left a comment

678098 left a comment

bbpetukhov commented Jan 17, 2025

Feat[Storagetool]: List queue appid numrecords #534

Feat[Storagetool]: List queue appid numrecords #534

Conversation

bbpetukhov commented Nov 29, 2024 • edited Loading

bbpetukhov commented Jan 13, 2025

bbpetukhov commented Jan 16, 2025 • edited Loading

678098 left a comment

Choose a reason for hiding this comment

678098 left a comment

Choose a reason for hiding this comment

678098 left a comment

Choose a reason for hiding this comment

bbpetukhov commented Jan 17, 2025

bbpetukhov commented Nov 29, 2024 •

edited

Loading

bbpetukhov commented Jan 16, 2025 •

edited

Loading