Information in the Representations and Information in the Weights of Deep Learning
Date:
This is a review talk. Effectiveness of deep learning is often ascribed to the ability of deep networks to learn representations. Using established principles from Statistics and Information Theory, we introduce the desirable properties of representations and the relationships between them. It is shown that invariance to nuisance factors in a deep neural network is equivalent to information minimality of the learned representation. On the other hand, we want to find parameters (weights) that yield good generalization, so we focus on the information in the weights which can be controlled by implicit or explicit regularization. With some additional assumptions, we get a connection between information in the representations and information in the weights.