Why Are We Renting H100 Clusters When UC Berkeley Just Fitted a 753B Model on an 8GB GPU?

I keep coming back to the quiet assumption that running frontier open models requires renting a specialized cluster. We talk about open weights as a victory for democratization, but the hardware bill usually tells a different story. A 35B model at 16-bit precision demands about seventy gigabytes just for its weights, pushing local deployment out of reach for standard workstations.

That equation shifted when UC Berkeley open-sourced FreeToken, detailed in their recent arXiv paper. The engine delivers two to four times the inference speed of Ollama by taking a fresh look at Mixture-of-Experts architectures.

Most inference engines treat hardware memory as a static wall. FreeToken profiles the bandwidth of your specific machine at startup, then splits expert retrieval dynamically on every single token. If a requested expert lives in system RAM, the engine dynamically balances whether to pull it across PCIe into the GPU or compute it directly on the CPU, matching the exact bottlenecks of your hardware instead of guessing.

The benchmark numbers challenge conventional scaling logic. We are looking at DeepSeek-V4-Flash running on a thirty-two gigabyte GPU, and GLM-5.2 running on ninety-six gigabytes, all while serving OpenAI and Anthropic compatible APIs out of the box.

For agent architectures, the bigger breakthrough is checkpoint management. Coding agents rewrite their execution history constantly, which normally triggers massive prefill recomputation. FreeToken aligns its caching boundaries directly with agent framework steps, keeping first-token latency grounded.

Releasing model weights determines who holds the intellectual property. But engines like this determine who can actually execute them without a cloud budget.