CLIP, LLaVA, and the Brain

🌈 Abstract

The article compares recent multimodal transformer networks like CLIP and LLaVA to the mammalian brain, focusing on the similarities and differences in their attention mechanisms and processing.

🙋 Q&A

[01] Similarities and Differences between Transformer Networks and the Brain

1. Questions related to the content of the section:

• What are the similarities between the attention in transformer networks like CLIP and LLaVA, and the attention mechanisms in the brain?
  • The article notes that vision transformers, CLIP, and LLaVA perform a type of processing analogous to pre-attentive visual processing in the brain, which is done in the initial feedforward visual responses before recurrence.
  • However, the brain has much richer interactions between areas, with higher-level areas exerting influence over lower-level areas through feedback connections. This allows for conscious top-down attention and automatic unconscious feedback.
• How do the feedforward processing in transformer networks differ from the bidirectional processing in the brain?
  • In contrast to the brain's bidirectional processing, most current deep learning architectures, including transformers, propagate activity in a single feedforward direction.
  • The article suggests that this feedforward processing in transformers is analogous to pre-attentive visual processing in the brain, which has limitations in dealing with complex or ambiguous stimuli.

[02] CLIP and LLaVA Architectures

1. Questions related to the content of the section:

• What are the key features of the CLIP architecture?
  • CLIP takes image-caption pairs from the internet and trains an image encoder and text encoder to bring the encodings of matching pairs closer together.
  • This allows CLIP to be used for zero-shot classification, but it does not generate text descriptions from images.
  • The image encoder and text encoder in CLIP are independent, meaning the image encoder must encode everything potentially relevant to the task.
• What are the key features of the LLaVA architecture?
  • LLaVA extends CLIP by adding the ability to describe and answer questions about images.
  • LLaVA uses the ViT-L/14 vision transformer model from CLIP for image encoding, and generates language responses one token at a time.
  • LLaVA also uses techniques like expanding captions to form instructions and using bounding box information.

[03] Limitations of Feedforward Processing in Transformer Networks

1. Questions related to the content of the section:

• What are the limitations of the feedforward processing in transformer networks compared to the brain's bidirectional processing?
  • The article suggests that the lack of bidirectional processing and internal states in transformer networks, including CLIP and LLaVA, constrains their processing capabilities.
  • This is especially true for image processing, as the image encoding is done independently of the text instructions, unlike the brain's dynamic allocation of resources based on task-driven attention.
  • The article conjectures that most convolutional, vision transformer, and multimodal transformer networks are restricted to processing analogous to pre-attentive feedforward visual processing in the brain, which has limitations in dealing with complex or cluttered scenes.
• What are some alternative architectures that go beyond the limitations of feedforward processing?
  • The article mentions long-short term memory (LSTM) models, the Mamba architecture, and extended LSTM models as examples of architectures that are not limited to pre-attentive feedforward processing.
  • Diffusion models are also noted as having a limited type of recurrence that uses the image as the state between iterations.