Google’s Gemma 4 open AI models use “speculative decoding” to get up to 3x faster - Ars Technica

Google has released Gemma 4, a new version of its open-weight AI model family, which incorporates a technique called speculative decoding to improve inference speed. The company claims the models can achieve up to three times faster generation compared to standard approaches. The release includes Multi-Token Prediction drafters, which are designed to predict multiple future tokens in parallel.

The speculative decoding method allows the models to run on consumer-grade GPUs, potentially increasing accessibility for developers and hobbyists. Google has positioned Gemma 4 as part of its open AI strategy, offering the model weights and code under a permissive license. The speed improvements come from a draft model that proposes tokens, which the main model then verifies, reducing latency.

The release follows a trend among major AI labs to offer more efficient models that can operate on local hardware, reducing reliance on cloud infrastructure. The company has not disclosed specific benchmark comparisons for accuracy versus speed trade-offs.