Draft: Update core assignment algorithm in benchexec/resources.py #892
+3,001
−331
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
CGall42
added
the
resource allocation
PhilippWendler
requested changes
Jan 26, 2023
There was a problem hiding this comment.
This is a preliminary review with some hints, mostly on code style and documentation. Please fix these issues and provide documentation, such that a full review becomes possible and makes sense.
As part of this (as first step actually), please format the source code with the formatter black and make sure to use it in the future for each commit. Consistent code formatting is a big help for readability.
And please also have a look at all the other CI failures. After each commit these checks will run automatically, so always check whether CI is green for your most recent commit. The check check-format is fixed automatically if you use the code formatter, and reuse only complains about the missing copyright header. But flake8 and pytype provide hints about potential errors in your code. And of course the unit-tests checks just execute our tests.
This method adds a "root" hierarchy level, if the system topology doesnt have one. Necessary for iterating through the whole topology.
now chooses the right starting core for the next thread
The added assertion is required to ensure the new definitions (which allow arbitrary CPU layouts¹) are actually identical to the existing ones. It will be removed together with the old definitions after the refactoring Note that the definition is based on the physical CPU layout, with the number of virtual cores per core being the second value of the (<cpu_layout>, <virtual_cores_per_core>) tupel. ¹) with the only restriction that we require CPUs in multi CPU systems to be identical, i.e. the layout needs to be symmetric
As noted in the previous commit, those definitions are based on the physical layout of the CPUs, with the number of hyperthreading / virtual cores per core being a seperate argument. This has the upside that the machine definitions do not need to be internally consistent, i.e. we do not need to take care if the total number of cores is actually possible with the number of virtual cores.
As the assertions show that they are equal for the existing tests, we can replace the existing logic to translate the old definitions and directly use the new ones. Also removes debug print statements and replaces the use of num_of_hyperthreading_siblings, so we can use the one in the machine definition (which we later can easily use with pytest parameterization).
Additionally, we temporarily as a shim set self.num_of_cores based on the physical cores times the number of virtual cores per core
As we now only use the single list-based machine definition, the specific layers are redundant and unused
The pytest-based assertion will replace the use of the assertInvalid method
As we use the equivalent pytest-based assertion, the old assertion is redundant
The _test_nCoresPerRun method is a remainder of prior refactoring - as it only contains a single call to another function, we can remove and inline its content safely
As with the other decorators, this centralization of the logic allows us to use pytest parameterization later on
This commit, again, still has the old calls, to have both in one test run
All calls to assertValid (from within the actual test classes) are now also performed via decorator
The existing logic was quite hard to read - to make future maintenance easier, the logic is broken up a bit to use explicit variable names, bundled in a single auxillary function
Identical functionality to the existing function can be achieved via set comparision, resulting in significantly simplified logic.
Using pythonic methods, especially the code to remove superfluous hyperthreading cores can be made more readable, while keeping identical functionality.
blocked_cores is completely unused and serves no purpose even after excessive search
While it does not do much for readability in the while loop itself, this increases readability of the if branch conditions inside it
This prior use of a temporary variable before assigning a shallow copy to the actual variable does not serve any purpose
To reduce the complexity of core_allocation_algorithm, it seems like a good idea to decouple the high level parts of the algorithm itself from the actual crawling of lists and dicts. Note that this commit only duplicates the code, to be able to use an assertion to make sure the new auxillery function returns equivalent results. The existing code block (and assertion) is removed in the next commit.
(and assertion)
Using enumerate is the more pythonic way, and this code should be more readable in general
From the old name it was quite unclear what this function actually does
Instead of having a any() construction with an auxillery variable, we can use a negated all() construct
…num_of_values To be able to remove check_asymmetric_num_of_values, we move the actual code check_symmetric_num_of_values, removing the final negation from the all() statement
As check_asymmetric_num_of_values is only a negation of another function, and as "not is_symmetric_hierachy" is quite clear, we can remove this function
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Referring to issue #748, the core assignment can now handle additional hierarchy layers (such as a shared L3 cache).
The addition of further layers can be implemented without knowing the exact topology of a machine - the hierarchy of the layers (CPUs, NUMA nodes, L3 caches, hyperthreading, etc) is determined by the algorithm.
Fixes #748
Fixes #850
Kernel documentation: