Free Your Compute. 753B Models Fit Where You Thought Nothing Could.

I genuinely don't know how to feel about this.

You've been told the opposite for years. Bigger models need bigger GPUs. Want to run a 753B parameter model? Rent an H100 cluster and join the queue. It's the dogma. Every vendor pitch and capex plan rests on it.

Then UC Berkeley dropped FreeToken.

An open-source inference engine claiming 2 to 4 times speedups over Ollama. It runs GLM-5.2 at 753 billion parameters on a single 96GB GPU. DeepSeek-V4-Flash, 284B parameters, fits on 32GB by offloading experts between GPU and CPU RAM in real time. No distributed training tricks. Just smarter routing of what already exists.

Qwen3.8-Flash-Next, a 125B MoE model, hits 21 tokens per second decode and 364 tokens per second prefill with 250K context on a consumer RTX 4090. Experts route to system RAM instead of hogging full VRAM. On an older RTX 3090, the Cafe-llama.cpp stack bumped throughput from 11 to nearly 28 tokens per second.

That's real decode throughput you could ship tomorrow.

One analyst called Kog's inference squeeze the largest untapped AI cost-efficiency market if these gains generalize. I keep coming back to that framing. The harness is eating the model again.

What struck me is how quiet this is moving. No press release or keynote. Just an open engine, a few benchmarks, and a whole industry realizing its hardware sizing assumptions are obsolete.

Every data center expansion plan written last quarter assumed you needed more GPUs for more parameters. Those plans just got uncomfortable. Fitting models became cheaper than anyone budgeted for.

FreeToken sits on GitHub under MIT. The code is there. The numbers are public.

If you've been renting clusters for inference work that now fits on cards you already own, that gap is closing faster than most spreadsheets admit.