TTT models might be the next frontier in generative AI | TechCrunch

🌈 Abstract

The article discusses the search for new AI architectures to replace the dominant transformer model, which is facing technical roadblocks related to computational efficiency. It highlights a promising new architecture called test-time training (TTT) that claims to be more scalable and efficient than transformers. The article also mentions other alternative architectures, such as state space models (SSMs), that are being explored by various AI companies and researchers.

🙋 Q&A

[01] The search for new AI architectures

1. What are the key issues with the current transformer architecture?

• Transformers are not especially efficient at processing and analyzing vast amounts of data, at least when running on off-the-shelf hardware.
• This is leading to steep and perhaps unsustainable increases in power demand as companies build and expand infrastructure to accommodate transformers' requirements.

2. What is the key idea behind the test-time training (TTT) architecture?

• TTT models replace the "hidden state" (a long list of data) in transformers with a machine learning model, which encodes the data it processes into representative variables called weights.
• This allows TTT models to efficiently process far more data than transformers without consuming as much compute power.

3. How do TTT models differ from transformers in terms of scalability?

• Unlike a transformer's lookup table, the size of the internal model in a TTT model does not grow as it processes additional data.
• This allows TTT models to potentially process billions of pieces of data, from words to images to audio recordings to videos, which is far beyond the capabilities of today's transformer models.

[02] Comparison to other alternative architectures

1. What are some other alternative architectures being explored?

• State space models (SSMs), like TTT models, appear to be more computationally efficient than transformers and can scale up to larger amounts of data.
• AI startups like Mistral, AI21 Labs, and Cartesia are exploring the use of SSMs as alternatives to transformers.

2. How do these alternative architectures compare to transformers?

• The article suggests that if these alternative efforts, such as TTT and SSMs, succeed, it could make generative AI even more accessible and widespread than it is now, for better or worse.
• However, it's still too early to say definitively whether these alternatives will supersede transformers, as they are not yet direct drop-in replacements.