Cherrypicks for v1.8.x-aws #354
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The idpool is freed when refcnt goes to zero, so it should be initialized again when refcnt increases. Signed-off-by: Eric Raut <[email protected]> (cherry picked from commit 213172a)
During the 1.8.0-aws release, we realized the distribution generation had been broken for a while. Adding quick sanity checks as a Github Action to prevent this from happening in the future. Runs to completion in 2 minutes. Signed-off-by: Raghu Raja <[email protected]> (cherry picked from commit e2f795d)
When progressing completions, we could get a completion error for a request before NCCL gets to calling test() explicitly for that request. Since NCCL tests for completions in order, this can lead to hangs when there are non-recoverable failures in the network and NCCL never receives a successful completion for the earliest request. With this change, completion errors are always passed up the stack so NCCL can abort the job and fail gracefully where possible. This logic can further be enhanced based on provider-specific information from completion error entry to distinguish between fatal errors vs recoverable user errors, but that would not be portable. Fixes aws#346 Signed-off-by: Raghu Raja <[email protected]> (cherry picked from commit 5aac4dc)
rauteric
approved these changes
Feb 23, 2024
bwbarrett
approved these changes
Feb 23, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.