Skipping ERROR caught in nll = model.optimize_parameters(current_step): svd_cuda: the updating process of SBDSDC did not converge (error: 23)

[flybiubiu]

Thx author!I train x4 is ok!
But when I train x8:
Skipping ERROR caught in nll = model.optimize_parameters(current_step):
svd_cuda: the updating process of SBDSDC did not converge (error: 23)

Python 3.8.5 (default, Sep 4 2020, 07:30:14)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.

import torch
print(torch.version)
1.7.1+cu110
print(torch.version.cuda)
11.0

print(torch.backends.cudnn.version())
8005

············································································································
My GPU is 3090.I run setup code and find the cuda version is not compare.After that I reinstall with (pip install torch==1.7.1+cu110 torchvision==0.8.2+cu110 torchaudio===0.7.2

About iters is 10000.

[RedRAINXXXX]

I encountered the same problem as you. When this error occurs, subsequent data will have this error

[JingzheLyp]

Hi, I encountered the same problem as you. Have you solved the problem? @flybiubiu, @RedRAINXXXX

[RedRAINXXXX]

Hi, I encountered the same problem as you. Have you solved the problem? @flybiubiu, @RedRAINXXXX

Perhaps because the learning rate is too high, you can try warm up or lower the learning rate directly