Google Cloud finds Gemma 3 12B outscales 27B on TPU

Google Cloud has published benchmark results comparing Gemma 3 12B and Gemma 3 27B on TPU v6e infrastructure. The tests found clear differences between classification and generation workloads.

The larger 27B model hit a throughput ceiling in generation tasks once concurrency rose above 64 users, while the smaller 12B model continued to scale more strongly at 128 users. In classification tasks, by contrast, both models showed broadly similar scaling at the highest concurrency tested.

The benchmarking focused on two types of large language model inference. One was a classification workload with long inputs of about 4,000 tokens and very short outputs of roughly 10 tokens. The other was a generation workload with shorter prompts of about 500 tokens and much longer responses of around 1,000 tokens.

Google Cloud ran the tests on a Google Kubernetes Engine Autopilot cluster connected to a single-host TPU v6e node pool with a 2x2 chip topology. It used vllm-project/tpu-inference as the serving framework and applied the same global serving configuration across all runs to keep the comparison consistent.

Generation gap

For generation workloads, the two models tracked each other closely at lower concurrency before separating sharply at the top end. At 16 concurrent users, throughput for Gemma 3 12B was the baseline at 1.00x, while Gemma 3 27B reached 1.05x. At 32 users, the figures were 1.98x for the 12B model and 1.97x for the 27B model.

The divergence appeared under heavier load. At 64 users, throughput reached 2.96x for Gemma 3 12B and 4.00x for Gemma 3 27B. At 128 users, however, the 12B model climbed to 8.19x while the 27B model rose only slightly to 4.12x, indicating a plateau.

That suggests model size can become a constraint for decode-heavy tasks, where the system spends much of its time producing long responses. Google Cloud said the larger model appeared to hit memory or compute limits earlier under those conditions.

Classification parity

In the classification tests, the pattern was different. At 16 users, Gemma 3 12B measured 1.00x and Gemma 3 27B measured 0.76x. At 32 users, those figures were 1.18x and 1.53x, respectively.

At 64 concurrent users, throughput rose to 2.04x for the 12B model and 3.15x for the 27B model. At 128 users, the two models converged, with Gemma 3 12B at 6.37x and Gemma 3 27B at 6.04x.

The figures indicate that in prefill-heavy workloads, where the model processes large prompts and produces short answers, the difference in parameter count does not translate into a major throughput penalty at scale. That could affect how companies choose between smaller and larger models for tasks such as compliance review, summarisation, and other structured text analysis.

Latency signals

The tests also tracked end-to-end latency, which Google Cloud highlighted as a better guide for scaling decisions than standard CPU or memory thresholds. Hardware saturation can show up as latency spikes and request dropouts rather than obvious infrastructure alarms, it said.

For Gemma 3 12B, generation latency moved from a 1.00x baseline at 16 users to 1.13x at 32 users, 1.40x at 64 users, and 1.70x at 128 users. Classification latency for the same model stayed almost flat at 32 users, at 0.99x, before rising to 1.79x at 64 users and 2.90x at 128 users.

Gemma 3 27B showed steeper latency growth in generation, rising from 1.20x at 16 users to 1.68x at 32 users, 2.93x at 64 users, and 3.33x at 128 users. In classification, latency was 1.20x at 16 users, 1.95x at 32 users, unchanged at 1.95x at 64 users, and then jumped to 3.88x at 128 users.

These thresholds matter because they point to different breakpoints depending on workload and model size. In practical terms, a deployment serving long-form generation requests may need stricter limits on concurrent requests per replica than one handling short classification outputs.

Google Cloud advised operators to keep settings such as max-num-seqs and max-model-len aligned with average traffic and token counts to avoid dropped requests. It also pointed to TPU-specific bucket padding adjustments in vllm as an important way to reduce memory waste.

The benchmark underlines a broader issue in commercial AI deployment: infrastructure choices and request patterns can shape costs and performance as much as model selection. Based on these results, organisations running decode-heavy generation at high concurrency would see stronger scaling from Gemma 3 12B on TPU v6e, while those handling prefill-heavy classification could use Gemma 3 27B without a significant throughput penalty.

At 128 concurrent users, the sharpest contrast in the data remained the split between generation throughput of 8.19x for Gemma 3 12B and 4.12x for Gemma 3 27B.