When I use GPU to train the model, I will have problem 'CUDA out of memory...'

[RobinLu1209]

Is there someone who also has this problem?

[Cestbo]

me too

[RobinLu1209]

me too

I have solved this problem. I remember that there are some code written like to be [to(device='cpu')]
You can change these code to be 'cuda', and have a try again. Maybe you will solve this problem.

[Cestbo]

Is there someone who also has this problem?

me too

I have solved this problem. I remember that there are some code written like to be [to(device='cpu')]
You can change these code to be 'cuda', and have a try again. Maybe you will solve this problem.

thank you ,i will try. i always think my gpu is weak

[Noahprog]

Nah, the problem is in validation method! In training 'batch_size' is used to prevent memory overload, but when validating it passes in the whole validation set. This means memory gets blown up.

[vsfh]

Nah, the problem is in validation method! In training 'batch_size' is used to prevent memory overload, but when validating it passes in the whole validation set. This means memory gets blown up.

do you know how to fix it, thx

[Noahprog]

Yes, but I don’t have time to do it for you. The solution is to rewrite the validation to also work with batches.

The training is already done this way, so that is a nice example for this.

[LiaoLW]

Nah, the problem is in validation method! In training 'batch_size' is used to prevent memory overload, but when validating it passes in the whole validation set. This means memory gets blown up.

do you know how to fix it, thx

Change the split line into about 0.9/0.95, then my problem solved

[Coolgiserz]

I change the dataset split (0.8/0.1/0.1) and solve this problem.