What is the difference between noise types?

[matifali]

What is the difference between these 2 output noise types,

InferenceRPUConfig.forward.w_noise = 0.02  # Some short-term w-noise.
InferenceRPUConfig.forward.out_noise = 0.02  # Some output noise.

They are both output noises.

[maljoras]

They are both output referred noise, but the out_noise is independent additive Gaussian applied to the outputs, whereas the w_noise is the strength of the noise and depends on the w_noise_type. It is noise that scales with the weight values. Take a look at the API documentation

[matifali]

from the documentation, I get that, if w_noise_type is ADDITIVE_CONSTANT then it does not scale with the weights and becomes independent. The only difference is then it is being added to the weights while out_noise is added to the output.

[maljoras]

Additive constant in context of the w_noise_type means that it is additive added to the current weights. So the resulting output noise strength scales with weight value. Whereas the out_noise is simply additive constant to outputs thus does never vary with weights. You could make a simple linear layer and plot the ideal versus analog output, to see the effect of these parameters.

[matifali]

When you say it scales with the weights, do you mean that variance w_noise is scaled with weights or it means, y= (W + w_noise*rand(n,1))*x = Wx + w_noise*rand(n,1)*x, the input is being scaled with a noise of w_noise and added to actual output?

[maljoras]

Right, sorry for my confusing comment. In case of ADDITIVE_CONSTANT w_noise is actually not scaled with the weight but with the input as you correctly write, it is then y= (W + w_noise*rand(n,m))*x. We have some other weight noise types where the actual weight values matter.

[matifali]

Thanks a lot for clearing up the confusion.

[matifali]

Also, is out_noise for every analog layer output?

[maljoras]

Yes, to each output line random noise is added drawn anew for each mat-vec. So something like y = W x + out_noise * randn(n,1)

[matifali]

Thanks