Subobject-level Image Tokenization

🌈 Abstract

The paper introduces the concept of "subobject"-level image tokenization, which lies between objects and pixels, as an alternative to the standard patch-level tokenization used in vision transformer models. The key ideas are:

• Subobjects are semantically meaningful visual entities (e.g., parts of objects) obtained through image segmentation models like Segment Anything Models (SAM).
• The authors propose a Direct Segment Anything Model (DirectSAM) that can efficiently produce comprehensive subobject segmentations.
• They also introduce a Sequence-to-sequence AutoEncoder (SeqAE) to embed the irregular-shaped subobjects into compact latent vectors.
• Finally, they incorporate the subobject tokens into a Large Vision Language Model (LVLM) by treating them as textual subword tokens, with additional positional embeddings.

Empirical results show that subobject-level tokenization significantly accelerates vision-language learning and improves accuracy in counting objects and recognizing visual attributes compared to standard patch-level tokenization.

🙋 Q&A

[01] Subobject Segmentation with DirectSAM

1. What are the key advantages of using subobject boundaries obtained from Segment Anything Models (SAM) compared to other segmentation methods?

• Subobject boundaries from SAM are semantically meaningful, open-vocabulary, and comprehensive, satisfying the key requirements for image tokenization.
• Other alternatives like superpixel segmentation or semantic/instance/panoptic segmentation do not fully meet these requirements.

2. What is the main limitation of the standard "segment everything" approach using SAM, and how does DirectSAM address it?

• The standard "segment everything" approach with SAM is time-consuming, as it requires prompting the model with a grid of points and running the mask decoder many times.
• DirectSAM addresses this limitation by directly predicting the subobject boundaries in a single pass, reducing the time complexity from O(n) to O(1), where n is the number of subobjects.

3. How is DirectSAM trained, and what are the key architectural choices?

• DirectSAM is trained on the SA-1B dataset, with the mask annotations converted to boundaries via contour detection.
• The model uses multi-scale augmentation to improve the quality of the segmentations.
• The target output is a binary map indicating the subobject boundaries, which is trained using a mean squared error loss.

[02] Subobject Embedding with SeqAE

1. What are the limitations of using square perception windows (e.g., in Transformer encoders) to encode subobject segments with irregular sizes and shapes?

• Square perception windows can only losslessly encode subobject segments within a certain aspect ratio range, leading to inefficient encoding for segments with extreme aspect ratios.

2. How does the Sequence-to-sequence AutoEncoder (SeqAE) address this issue?

• SeqAE flattens the raw subobject pixels and masks into data sequences, allowing it to make full use of the available context length without the need for downsampling.
• It uses learnable query tokens and a bottleneck projector to extract compact latent representations of the subobject segments.
• The autoencoding objective with mean squared error loss encourages the model to learn efficient compression of the subobject information.

3. What are the key architectural choices in the SeqAE model?

• The encoder and decoder both have 16 Transformer layers, with 768-dimensional outputs and 16 learnable query tokens.
• The bottleneck projector reduces the dimension from the encoder output to 256, which is then used to reconstruct the query tokens.
• The model is trained on the large-scale SA-1B dataset with a context length of 1024 tokens.

[03] LVLM based on Subobject-level Image Tokenization

1. How does the authors' methodology of incorporating subobject tokens into a Large Language Model (LLM) work?

• The authors treat the subobject tokens as textual subword tokens in a new "language", and insert them into the LLM alongside the textual subword tokens.
• They use special tokens and to mark the start and end of subobject tokens from a single image.

2. What are the two key technical modifications made to accommodate the unique nature of images compared to natural languages?

• Additional positional embedding for subobject tokens: The authors introduce 2D positional embeddings based on the bounding box coordinates of the subobject segments, in addition to the 1D positional embeddings in the original LLM.
• No autoregressive prediction for subobject tokens: Since subobjects do not have a causal one-dimensional structure like natural language, the authors skip the autoregressive prediction loss for subobject tokens during training.

3. What dataset and model are used for the LVLM implementation, and what are the key training details?

• The authors use the Phi-2 model, a 2.7B parameter LLM, as the base for their LVLM.
• They create a synthetic image captioning dataset from CLEVR, converting the scene graph annotations into textual descriptions of object counts, size, material, and shape.
• The LVLM is trained for 10 epochs with a cosine learning rate scheduler and an effective batch size of 32.