This document summarizes a research paper that proposes a technique called Stable Rank Normalization to improve generalization in neural networks and GANs. The technique aims to reduce the Lipschitz constant of neural networks by normalizing the stable rank of the weight matrices. The stable rank is a measure of how many effective dimensions a matrix has. Normalizing it makes networks less sensitive to certain parameter settings. The paper shows experimentally that stable rank normalization improves generalization on image recognition and GAN tasks without affecting performance.
19. 実験
分類
ResNet-110, WideResNet-28-10, DenseNet-100, VGG100,
AlexNet on CIFAR100
GAN
Compare with SN-GAN, WGAN-GP, Ortho-GAN on CIFAR100,
CIFAR10, CelebA
Inception Score, FID, Neural Divergence score
stable rank for with fixed
hyperparameter . ( SN)
r = c min(m, n) W ∈ Rm×n
0 < c ≤ 1 c = 1 ⇒
19