issue-1444: Use kernel delay accounting to calculate cpu wait #1630

antonmyagkov · 2024-07-17T15:57:37Z

Add stats fetcher type to configuration
Rename ICGroupStatsFetcher to IStatsFetcher
Implement TTaskStatsFetcher

taskstats doc:
https://www.kernel.org/doc/Documentation/accounting/taskstats.txt
https://www.kernel.org/doc/Documentation/accounting/delay-accounting.rst

github-actions · 2024-07-17T22:55:09Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit d491e70.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
5458	5454	0	1	1	2

github-actions · 2024-07-23T16:37:31Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 487a76d.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
5492	5491	0	1	0	0

github-actions · 2024-08-27T17:05:41Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 3bc2042.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
5687	5686	0	1	0	0

github-actions · 2024-11-01T22:05:57Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 52f2bd0.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
6061	6060	0	1	0	0

github-actions · 2024-11-02T00:37:28Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 497d987.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
6077	6077	0	0	0	0

github-actions · 2024-11-02T23:15:05Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 8001b7a.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
6077	6077	0	0	0	0

github-actions · 2024-11-03T23:41:03Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 68aa05f.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
6077	6077	0	0	0	0

github-actions · 2024-11-04T13:37:51Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 39c756e.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
6077	6077	0	0	0	0

github-actions · 2024-11-09T20:44:13Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit a516eec.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
6098	6098	0	0	0	0

cloud/storage/core/libs/diagnostics/cgroup_stats_fetcher.cpp

cloud/storage/core/libs/diagnostics/stats_fetcher.cpp

github-actions · 2024-11-20T00:09:42Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 57a7799.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
6119	6119	0	0	0	0

cloud/storage/core/libs/diagnostics/stats_fetcher.h

github-actions · 2024-12-03T23:13:41Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit 5d3fedc.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
6158	6157	0	1	0	0

github-actions · 2024-12-04T14:20:18Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 6ae006d.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
6167	6167	0	0	0	0

antonmyagkov · 2024-12-06T09:48:03Z

cloud/storage/core/libs/diagnostics/stats_fetcher.cpp

+
+////////////////////////////////////////////////////////////////////////////////
+
+struct TTaskStatsFetcher final: public IStatsFetcher


TTaskStatsFetcher - основное изменение в этом пул реквесте

github-actions · 2024-12-24T14:49:59Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit 6ae006d.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
6314	6314	0	0	0	0

ya-ksgamora · 2024-12-28T11:58:13Z

cloud/storage/core/libs/diagnostics/stats_fetcher.cpp

+
+    TResultOrError<TDuration> GetCpuWait() override
+    {
+        try {


может, использовать function try блок, чтобы не было двойной табуляции?

Прикольно. Я не знал про такую фичу.

Я бы оставил как есть для единообразия кода(По крайней мере я не нашел примеров использования function try в нашем коде).

drbasic · 2025-01-16T06:26:43Z

cloud/storage/core/libs/diagnostics/stats_fetcher.cpp

+            auto cpuDelay = NThreading::NewPromise<TResultOrError<TDuration>>();
+            netlinkSocket.SetCallback(
+                NL_CB_VALID,
+                [this, &cpuDelay](nl_msg* nlmsg) {


чем гарантируется что this не протухнет к моменту вызова колбэка?

drbasic · 2025-01-16T06:28:13Z

cloud/storage/core/libs/diagnostics/stats_fetcher.cpp

+                    return NL_OK;
+                });
+            netlinkSocket.Send(message);
+            auto cpuWait = cpuDelay.GetFuture().GetValue(NetlinkSocketTimeout);


аа, получается мы тут повиснем на ожидании коллбэка

мы блокируемся в Send на самом деле. Там синхронная реализация https://github.com/ydb-platform/nbs/blob/main/cloud/storage/core/libs/netlink/socket.cpp#L45

в целом ничего не мешает сделать его асинхронным, но тогда нужно будет аккуратно проставить error колбэки и пошарить контекст, например, через shared_from_this как сделано в nbd

cloud/storage/core/libs/diagnostics/stats_fetcher.cpp

drbasic · 2025-01-16T06:33:12Z

cloud/storage/core/libs/diagnostics/stats_fetcher.cpp

+
+////////////////////////////////////////////////////////////////////////////////
+
+struct TCgroupStatsFetcher final


лучше в разных файлах оставить разные реализации, тогда будет видно что код просто перенесли.

cloud/storage/core/libs/diagnostics/task_stats_fetcher_ut.cpp

drbasic · 2025-01-16T06:41:11Z

cloud/storage/core/libs/diagnostics/stats_fetcher.h


 ////////////////////////////////////////////////////////////////////////////////

-ICgroupStatsFetcherPtr CreateCgroupStatsFetcher(
+IStatsFetcherPtr CreateCgroupStatsFetcher(


а чем CreateCgroupStatsFetcher/CreateTaskStatsFetcher отличается от BuildStatsFetcher не достаточно одной функции?

Теперь точно нужны, так как я разнес реализации CGroupStatsFetcher и TaskStatsFetcher по разным cpp файлам

drbasic · 2025-01-16T06:43:30Z

cloud/storage/core/libs/diagnostics/ya.make

    cloud/storage/core/protos

    library/cpp/lwtrace/mon

+    contrib/restricted/libnl/lib/nl-3


от contrib/restricted/libnl начинает зависеть еще один ya.make нам снова прописывать исключения (((

Есть re-implementation of libnl-2 с лицензий apache 2.0 https://android.googlesource.com/platform/system/core/+/29d238d/libnl_2/README

@tpashkin Может мы можем на нее перейти?

@tpashkin Может мы можем на нее перейти?

Почему бы и нет, нам нужен довольно базовый набор функций

Я создал отдельный тикет: #2869
Попробуем перейти на другую реализацию

от contrib/restricted/libnl начинает зависеть еще один ya.make нам снова прописывать исключения (((

@drbasic а напомни, плиз, что за исключения. вы сейчас не линкуетесь с libnl в аркадии?

github-actions · 2025-01-16T18:05:46Z

Note

This is an automated comment that will be appended during run.

🔴 linux-x86_64-relwithdebinfo: some tests FAILED for commit c3a8efa.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
6335	6332	0	3	0	0

github-actions · 2025-01-17T00:01:55Z

Note

This is an automated comment that will be appended during run.

🟢 linux-x86_64-relwithdebinfo: all tests PASSED for commit c39eff4.

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
6334	6334	0	0	0	0

antonmyagkov force-pushed the users/myagkov/issue-1444-use-delayacct branch from 9416d9c to d491e70 Compare July 17, 2024 21:37

antonmyagkov force-pushed the users/myagkov/issue-1444-use-delayacct branch 5 times, most recently from 2044242 to 487a76d Compare July 23, 2024 15:19

antonmyagkov force-pushed the users/myagkov/issue-1444-use-delayacct branch from 487a76d to 3bc2042 Compare August 27, 2024 15:41

antonmyagkov force-pushed the users/myagkov/issue-1444-use-delayacct branch 3 times, most recently from 4701cd8 to 52f2bd0 Compare November 1, 2024 20:38

antonmyagkov force-pushed the users/myagkov/issue-1444-use-delayacct branch 2 times, most recently from a08f79b to 497d987 Compare November 1, 2024 23:06

antonmyagkov force-pushed the users/myagkov/issue-1444-use-delayacct branch from f5b8b26 to 68aa05f Compare November 3, 2024 22:09

antonmyagkov force-pushed the users/myagkov/issue-1444-use-delayacct branch from b8b085c to be4fec9 Compare November 8, 2024 22:30

tpashkin reviewed Nov 12, 2024

View reviewed changes

cloud/storage/core/libs/diagnostics/cgroup_stats_fetcher.cpp Outdated Show resolved Hide resolved

tpashkin reviewed Nov 12, 2024

View reviewed changes

cloud/storage/core/libs/diagnostics/cgroup_stats_fetcher.cpp Outdated Show resolved Hide resolved

antonmyagkov commented Nov 13, 2024

View reviewed changes

cloud/storage/core/libs/diagnostics/cgroup_stats_fetcher.cpp Outdated Show resolved Hide resolved

antonmyagkov force-pushed the users/myagkov/issue-1444-use-delayacct branch 4 times, most recently from f58ff78 to 50f14dd Compare November 15, 2024 12:25

tpashkin previously approved these changes Nov 19, 2024

View reviewed changes

antonmyagkov commented Nov 19, 2024

View reviewed changes

cloud/storage/core/libs/diagnostics/stats_fetcher.cpp Outdated Show resolved Hide resolved

antonmyagkov dismissed tpashkin’s stale review via 56ab434 November 19, 2024 18:11

tpashkin reviewed Nov 20, 2024

View reviewed changes

cloud/storage/core/libs/diagnostics/stats_fetcher.h Outdated Show resolved Hide resolved

antonmyagkov mentioned this pull request Nov 21, 2024

issue-1444: create CpuWaitFailure counter in filestore/blockstore actors #2541

Merged

antonmyagkov force-pushed the users/myagkov/issue-1444-use-delayacct branch from 57a7799 to 3dd7d98 Compare December 3, 2024 12:21

antonmyagkov added 2 commits December 4, 2024 12:45

issue-1444: Use kernel delay accounting to calculate cpu wait

09f97ec

refactor: remove namespace

459971a

antonmyagkov force-pushed the users/myagkov/issue-1444-use-delayacct branch from 8cbd414 to 7c3c9b6 Compare December 4, 2024 12:46

fix: build

6ae006d

antonmyagkov force-pushed the users/myagkov/issue-1444-use-delayacct branch from 7c3c9b6 to 6ae006d Compare December 4, 2024 12:48

antonmyagkov requested a review from tpashkin December 4, 2024 17:23

antonmyagkov commented Dec 6, 2024

View reviewed changes

antonmyagkov added the rebase Add this label if you want to rebase your PR for test run label Dec 24, 2024

tpashkin previously approved these changes Dec 27, 2024

View reviewed changes

ya-ksgamora reviewed Dec 28, 2024

View reviewed changes

drbasic reviewed Jan 16, 2025

View reviewed changes

antonmyagkov dismissed tpashkin’s stale review via 3b4deb3 January 16, 2025 16:26

refactor: move stats fetcher implementation to different files

c3a8efa

antonmyagkov force-pushed the users/myagkov/issue-1444-use-delayacct branch from 3b4deb3 to c3a8efa Compare January 16, 2025 16:48

remove semicolon

c39eff4

antonmyagkov requested review from tpashkin, drbasic and ya-ksgamora January 17, 2025 10:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

issue-1444: Use kernel delay accounting to calculate cpu wait #1630

issue-1444: Use kernel delay accounting to calculate cpu wait #1630

antonmyagkov commented Jul 17, 2024 •

edited

Loading

github-actions bot commented Jul 17, 2024

github-actions bot commented Jul 23, 2024

github-actions bot commented Aug 27, 2024

github-actions bot commented Nov 1, 2024

github-actions bot commented Nov 2, 2024

github-actions bot commented Nov 2, 2024

github-actions bot commented Nov 3, 2024

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 9, 2024

github-actions bot commented Nov 20, 2024

github-actions bot commented Dec 3, 2024

github-actions bot commented Dec 4, 2024

antonmyagkov Dec 6, 2024

github-actions bot commented Dec 24, 2024

ya-ksgamora Dec 28, 2024

antonmyagkov Dec 28, 2024

drbasic Jan 16, 2025

drbasic Jan 16, 2025

antonmyagkov Jan 17, 2025

tpashkin Jan 17, 2025 •

edited

Loading

drbasic Jan 16, 2025

drbasic Jan 16, 2025

antonmyagkov Jan 16, 2025 •

edited

Loading

drbasic Jan 16, 2025

antonmyagkov Jan 16, 2025

tpashkin Jan 17, 2025

antonmyagkov Jan 17, 2025 •

edited

Loading

tpashkin Jan 17, 2025 •

edited

Loading

github-actions bot commented Jan 16, 2025

github-actions bot commented Jan 17, 2025


		////////////////////////////////////////////////////////////////////////////////

		struct TTaskStatsFetcher final: public IStatsFetcher


		////////////////////////////////////////////////////////////////////////////////

		struct TCgroupStatsFetcher final

issue-1444: Use kernel delay accounting to calculate cpu wait #1630

Are you sure you want to change the base?

issue-1444: Use kernel delay accounting to calculate cpu wait #1630

Conversation

antonmyagkov commented Jul 17, 2024 • edited Loading

github-actions bot commented Jul 17, 2024

github-actions bot commented Jul 23, 2024

github-actions bot commented Aug 27, 2024

github-actions bot commented Nov 1, 2024

github-actions bot commented Nov 2, 2024

github-actions bot commented Nov 2, 2024

github-actions bot commented Nov 3, 2024

github-actions bot commented Nov 4, 2024

github-actions bot commented Nov 9, 2024

github-actions bot commented Nov 20, 2024

github-actions bot commented Dec 3, 2024

github-actions bot commented Dec 4, 2024

Choose a reason for hiding this comment

github-actions bot commented Dec 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tpashkin Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antonmyagkov Jan 16, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

antonmyagkov Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

tpashkin Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Jan 16, 2025

github-actions bot commented Jan 17, 2025

antonmyagkov commented Jul 17, 2024 •

edited

Loading

tpashkin Jan 17, 2025 •

edited

Loading

antonmyagkov Jan 16, 2025 •

edited

Loading

antonmyagkov Jan 17, 2025 •

edited

Loading

tpashkin Jan 17, 2025 •

edited

Loading