Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: add event sync in fsdp tutorial #3166

Merged
merged 4 commits into from
Jan 8, 2025

Conversation

zzb610
Copy link
Contributor

@zzb610 zzb610 commented Nov 20, 2024

Fixes #ISSUE_NUMBER

Description

The event should be synchronized before calling elapsed_time to ensure that the corresponding event has completed. This change is necessary to avoid runtime errors related to incomplete events.

For more details, refer to the relevant code in the PyTorch repository:

Example Code

import torch
def case():
    torch.cuda.set_device(0)

    init_start_event = torch.cuda.Event(enable_timing=True)
    init_end_event = torch.cuda.Event(enable_timing=True)

    init_start_event.record()
    for i in range(100):
        a = torch.randn(size=(1024, 1024), device="cuda")
        b = torch.randn(size=(1024, 1024), device="cuda")
        out = torch.add(a, b)
        out = torch.matmul(a, b)
        out = torch.matmul(a, b)
        out = torch.matmul(a, b)
        out = torch.matmul(a, b)
    init_end_event.record()

    # print("init_end_event.query(): ", init_end_event.query()) # False
    # # RuntimeError: CUDA error: device not ready
    # print(
    #     f"CUDA event elapsed time: {init_start_event.elapsed_time(init_end_event) / 1000}sec"
    # )

    # OK
    init_end_event.synchronize()
    print("init_end_event.query(): ", init_end_event.query())
    print(
        f"CUDA event elapsed time: {init_start_event.elapsed_time(init_end_event) / 1000}sec"
    )

cc @wconstab @osalpekar @H-Huang @kwen2501

Copy link

pytorch-bot bot commented Nov 20, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/tutorials/3166

Note: Links to docs will display an error until the docs builds have been completed.

⏳ No Failures, 14 Pending

As of commit 9fee938 with merge base 355f281 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@zzb610
Copy link
Contributor Author

zzb610 commented Nov 25, 2024

@HamidShojanazeri Could you please take a look at this PR when you have a moment? Thanks a lot!

@zzb610
Copy link
Contributor Author

zzb610 commented Jan 6, 2025

@svekars Hi, Could you please take a look at this PR when you have a moment? Thanks a lot!

@svekars svekars added the skip-link-check Will allow you to skip linkcheck on a PR. Should only should be used when a link can't be fixed. label Jan 8, 2025
@svekars svekars merged commit 33a52a5 into pytorch:main Jan 8, 2025
19 checks passed
titaiwangms added a commit to titaiwangms/tutorials that referenced this pull request Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla signed distributed skip-link-check Will allow you to skip linkcheck on a PR. Should only should be used when a link can't be fixed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants