experience on SVHN outputs anormally high confidence on adversarial accuracy

[feather0011]

Hello. Thank you for opening your code and experience log.

While running your code to train SVHN, I found that training SVHN gives strangely high adversarial accuracy on test set.

Including your paper, SVHN usually shows adversarial accuracy near 55~60%.

However, when I run your code for 4 times with different seeds(0~3), 3 of them gives accuracy near 90%.

The only change I made on the original code is to add 3 lines at the begining of the code to assign GPU.

import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]="0"   #can be "1", "2", "3"

I share the logs of that are trained so far.

(The trainings are not finished yet, but as you describe the "Best" performance also in the paper, strange best performance can be an issue)

output1.log
output2.log
output3.log
output0.log

I never saw any paper that claims their adversarial accuracy on SVHN is near 90%. So I presume this is result of a bug but I am not certain.

[theFool32]

Maybe you should set pgd_alpha to 1 according to the log in this repo.

[mnzhao]

Same issue here.