python train_auto.py --model fno --data_name cavity_prop_bc_geo 内存溢出

[PlusOrMinus]

具体报错如下
Traceback (most recent call last):
File "/public/yzk/D/DeepLearning/CFDBench/src/train_auto.py", line 381, in
main()
File "/public/yzk/D/DeepLearning/CFDBench/src/train_auto.py", line 351, in main
train(
File "/public/yzk/D/DeepLearning/CFDBench/src/train_auto.py", line 233, in train
outputs: dict = model(**batch)
File "/public/yzk/D/Anaconda3/envs/CFDBench/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/public/yzk/D/Anaconda3/envs/CFDBench/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
return forward_call(*args, **kwargs)
File "/public/yzk/D/DeepLearning/CFDBench/src/models/fno/fno2d.py", line 208, in forward
props = props.repeat(1, 1, height, width) # (B, p, H, W)
RuntimeError: CUDA error: out of memory
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

[chen-yingfa]

请问你是用什么硬件跑的？

[PlusOrMinus]

您好，显卡是32G的V100

…

------------------ 原始邮件 ------------------ 发件人: "luo-yining/CFDBench" ***@***.***>; 发送时间: 2025年1月9日(星期四) 凌晨0:21 ***@***.***>; ***@***.******@***.***>; 主题: Re: [luo-yining/CFDBench] python train_auto.py --model fno --data_name cavity_prop_bc_geo 内存溢出 (Issue #10) 请问你是用什么硬件跑的？ — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[PlusOrMinus]

您好，显卡是32G的V100。python train_auto.py --model  unet  --data_name cavity_prop_bc_geo的话，可以正常运行。

…

------------------ 原始邮件 ------------------ 发件人: "luo-yining/CFDBench" ***@***.***>; 发送时间: 2025年1月9日(星期四) 凌晨0:21 ***@***.***>; ***@***.******@***.***>; 主题: Re: [luo-yining/CFDBench] python train_auto.py --model fno --data_name cavity_prop_bc_geo 内存溢出 (Issue #10) 请问你是用什么硬件跑的？ — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[chen-yingfa]

可能显存太小，建议尝试降低batch size或者使用别的常见的降低显存占用的手段

[PlusOrMinus]

好的，谢谢

…

------------------ 原始邮件 ------------------ 发件人: "luo-yining/CFDBench" ***@***.***>; 发送时间: 2025年1月10日(星期五) 中午11:21 ***@***.***>; ***@***.******@***.***>; 主题: Re: [luo-yining/CFDBench] python train_auto.py --model fno --data_name cavity_prop_bc_geo 内存溢出 (Issue #10) 可能显存太小，建议尝试降低batch size或者使用别的常见的降低显存占用的手段 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>