Google unveils two distinct TPU chips for agentic AI

Targets growing divide in AI workloads

Google has announced a major redesign of its AI hardware strategy, unveiling two distinct chips aimed at meeting the rapidly changing demands of advanced AI systems.

Speaking at its annual Google Cloud Next conference on Wednesday, the company introduced TPU 8t and TPU 8i, the first time in a decade that Google has split its Tensor Processing Unit (TPU) line into separate architectures for training and inference.

For years, Google relied on a single TPU design per generation to handle both the training of AI models and the process of running them. But the company now says those workloads have diverged too far to be efficiently handled by one chip.

"The rise of agentic AI requires infrastructure that can handle long context windows and complex sequential logic," Google said in a blog post.

"The eighth-generation TPUs (TPU 8t and TPU 8i) are our answer to these challenges, ensuring that every workload, from the first token of training to the final step of a multi-turn reasoning chain, is running on the most efficient path possible."

Agentic AI refers to systems capable of breaking tasks into smaller goals, simulating outcomes, and iterating decisions, often across multiple steps and specialised sub-agents.

Two chips, two roles

TPU 8t is designed for large-scale training, including trillion-parameter models. It retains Google's established 3D torus network, optimised for predictable, high-throughput computation across thousands of chips.

TPU 8i, by contrast, is built for inference, the stage where trained models are deployed. It focuses on reducing latency and handling memory-intensive workloads required by long reasoning chains.

A key feature of TPU 8i is its ability to keep entire "key-value caches" – critical data structures used in AI reasoning – directly on the chip. Google says this eliminates delays caused by accessing external memory, a growing bottleneck in modern AI systems.

Rethinking AI infrastructure

The move comes amid broader changes in how AI systems are built and used.

Earlier chatbot-style systems typically processed a single prompt and response. Newer agentic models operate in loops, delegating subtasks and maintaining context over extended interactions. This places heavier demands on memory bandwidth, communication speed, and system coordination.

Google says TPU 8i includes a new "Collectives Acceleration Engine" that reduces latency in coordination-heavy tasks by up to five times. It also introduces a new "Boardfly" network design, reducing communication steps between chips in large clusters.

TPU 8t offers around 2.7 times better price-performance than its predecessor for training, according to the company, while TPU 8i delivers 80% improvements for inference.

Independent benchmarks are yet to confirm these figures.

For the first time, Google will offer bare-metal access to its TPUs, allowing customers to bypass virtual machines and directly control hardware, an option long favoured by developers using Nvidia GPUs.

The company also previewed TorchTPU, bringing native support for the popular PyTorch framework, an area where TPUs have historically lagged.

Both TPU 8t and TPU 8i are expected to ship later this year through Google's AI Hypercomputer platform, though exact availability dates have not been confirmed.