Load balancing across GPUs #1
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
profiling is done for each task execution and the required information about the task executed is written to the trace. The execution time is not explicitly calculated as it can be found from each event begin and end in the trace.
profiling is done for each task execution and the required information about the task executed is written to the trace. The execution time is not explicitly calculated as it can be found from each event begin and end in the trace.
same-name cache variable and breaks findHWLOC from PaRSECConfig.cmake Signed-off-by: Aurelien Bouteiller <[email protected]>
find_data_size() was using number of parameters to find the total data the task was operating on. This was corrected, now we use number of flows to calculate the total data.
First arena is queried for the task data size. If arena is NULL original is queried for the data size.
…ar device queue. A device queue is created for each gpu device to hold the migrated tasks (migrated_task_list). The parsec_cuda_kernel_schedule() function schedules the migrated task to the correct queue.
…. parsec_cuda_kernel_dequeue() schedules a migrated task to the gpu device. This function will be called in __parsec_context_wait() just before parsec_current_scheduler->module.select(). This will ensure that the migrated tasks will get priority over new tasks. When a compute thread calls this function, it is forced to try to be a manager of the device. If the device already has a manager, the compute thread passes the control of the task to the manager. If not the compute thread will become the manager.
…e a thread to be a manager thread, if there are any tasks migrated to a particular device. This will also ensure that a migrated task gets priority in execution when compared to a new task. Using migrate_if_starving(), the manager checks if there are starving devices. This checking is done before a new task is selected for execution. If there are any starving devices, the manager migrate tasks to the starving device, if there are available tasks to migrate.
Decrement the gpu task count when a task is executed by the gpu. Decrement the task at the dealer gpu when task is migrated. Increment the task at the straving gpu when the migrated task is recieved.
if cuda is available, the cuda nvml library will be available along with it.
…ad of parsec_cuda_set_device_task()
…ex, while the correct index was dealer_device->super.device_index-2
This policy makes sure that only tasks with an affinity to the starving node is migrated.
should be carried out by the GPU manager thread or some other worker thread.
Some other minor changes.
perc of evictions wrt stage in.
The second thread that offloads a task to the device is trasnistioned to a manager that handles only task migration.
Only to be used for testing purposes.
…eams in the device. 2. Same starvation condition used on both dealer and starving device.
2. Functions and variables renamed. 3. More statistics added.
There is no need to track task count at every stage.
before the parsec_complete() was called. This was correted.
mapped for iterative applications.
parsec_cuda_iterative = 1 maps only migrated task while parsec_cuda_iterative = 2 maps al tasks
2. Add zone_free data to gpu_mem_lru when migrating
2. Documentation updated.
2. Code cleanup 3. documenattion updated
Memory leak problem addressed by using a temporary list.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The manager of a GPU device identifies starvation in other GPUs within a node and migrates tasks to the starving node.