TPU transformation: A look back at 10 years of our AI-specialized chips | Google Cloud Blog

🌈 Abstract

The article discusses the development of Google's Tensor Processing Units (TPUs), custom-designed chips for accelerating AI and machine learning workloads. It covers the history of how Google recognized the need for specialized hardware to support the growing demand for AI, the evolution of TPU generations, and how TPUs have become the backbone for AI across Google's products and services.

🙋 Q&A

[01] The Discovery of the Need for Specialized Hardware

1. What prompted Google to realize the need for a new kind of chip?

• Google's research teams began thinking seriously about launching speech recognition features at a global scale.
• The team did some back-of-the-napkin math and realized that the compute power required to handle hundreds of millions of people talking to Google for just 3 minutes a day would take up basically all the compute power Google had deployed at the time.
• They realized they would need to double the number of computers in Google data centers to support these new features, and concluded that "there must be a better way."

2. What were the limitations of the existing hardware options on the market?

• The team looked at different approaches that existed on the market, but ultimately realized they were not able to meet the sheer demand of even those basic machine learning workloads their products were operating, let alone what might follow in the years to come.

3. What led Google to decide to develop a custom chip?

• Google's leaders realized they were going to need a whole new kind of chip, so a team that had already been exploring custom silicon designs enlisted Googlers from other machine-learning teams and laid down the framework for what would ultimately be their first Tensor Processing Unit (TPU).

[02] The Development of TPUs

1. How do TPUs differ from CPUs and GPUs?

• Where CPUs are designed as the jack-of-all-trades general-purpose "brains" for a computer, and GPUs were specialized chips designed to work in tandem with a CPU to accelerate complex tasks in graphics, video rendering, and simulations, TPUs were purpose-built specifically for AI.
• TPUs are an application-specific integrated circuit (ASIC), a chip designed for a single, specific purpose: running the unique matrix and vector-based mathematics that's needed for building and running AI models.

2. What was the initial scale and impact of the first TPU v1 chip?

• The first TPU v1 chip was deployed internally in 2015 and was instantly a hit across different parts of Google.
• The team had thought they'd maybe build under 10,000 of them, but ended up building over 100,000 to support all kinds of great stuff including Ads, Search, speech projects, AlphaGo, and even some self-driving car stuff.

3. How have TPUs evolved over the years?

• In the decade since, TPUs have advanced in performance and efficiency across generations and spread to serve as the backbone for AI across nearly all of Google's products.
• The latest generation, Trillium, offers more power and efficiency to help train Google's next generation of cutting-edge AI models.

[03] The Expansion of TPUs to Google Cloud

1. What challenges did early AI teams face in accessing the compute power they needed?

• One of the co-founders of an ML startup in 2012 had to buy used gaming GPUs online and build servers on their coffee table, running the GPUs and then turning on the microwave, causing the power to go out.

2. How did Google make TPUs available to its cloud customers?

• By early 2018, a small team launched the first generation of Cloud TPUs to help Google Cloud customers accelerate their own training and inference workloads.
• Today, Anthropic, Midjourney, Salesforce, and other well-known AI teams use Cloud TPUs intensively, with more than 60% of funded generative AI startups and nearly 90% of gen AI unicorns using Google Cloud's AI infrastructure, including Cloud TPUs.

[04] The Future of TPUs

1. What are the plans for the future evolution of TPUs?

• The solution that's lined up today is very different from the solution tomorrow, as Google is changing its data center designs to match the needs even better.
• The future is "full stack customization all the way, from silicon to concrete," with Google building a global network of data centers filled with TPUs.