Learned Structures – Non_Interactive – Software & ML

🌈 Abstract

The article discusses the author's fascination with neural network architectures, particularly the advancements in the field from 2019-2021. It explores the types of architectural tweaks that can meaningfully impact model performance, categorizing them into two groups: modifications that improve numerical stability during training, and modifications that enhance the expressiveness of a model in learnable ways.

🙋 Q&A

[01] Architectural Tweaks for Improved Performance

1. What are the two main categories of architectural tweaks that can meaningfully impact model performance?

• Modifications that improve numerical stability during training
• Modifications that enhance the expressiveness of a model in learnable ways

2. What are some examples of modifications that improve numerical stability?

• Where and how to normalize activations
• Weight initialization
• Smoothed non-linearities

3. What is the core idea behind modifications that enhance learnable expressiveness? The core idea is to build structured representations of the data and allow those structures to interact in learnable ways. This includes:

• MLPs allowing all elements of a vector to interact with each other through the weights
• Attention layers allowing a set of vectors to interact with each other
• Mixture of Experts dynamically selecting weights based on the values within the vector

4. Why is it important for neural networks to learn in stages? The author suggests that training neural networks by optimizing the entire parameter space from the beginning leads to the parameters fighting to optimize simple patterns in the data distribution. The learnable structures, such as attention and Mixture of Experts, are not meaningful in the early training regime but become valuable as the network learns more complex layers of the data distribution.

[02] Mixture of StyleGANs

1. What is the author's idea for improving the performance and fidelity of image generation models like StyleGAN? The author suggests training separate StyleGAN-like models for different modal components of an image (e.g., a person's face, a hand, a tree in the background) and then using a "mixture of StyleGANs" to compose the final image, rather than trying to fit the entire data distribution into a single StyleGAN model.

2. What is an existing application of this general idea that the author finds interesting? The author mentions using StyleGAN to fix faces in StableDiffusion, which they consider an early application of the "mixture of StyleGANs" concept.