Kog Bets the GPU Arms Race Is a Software Problem

Here's what gets me about this one: a French startup thinks the GPU inference bottleneck is a software problem, not a hardware one.

Everyone watches the hardware race, Cerebras went public in May with purpose-built wafers, Nvidia sells H200s faster than they can make them, and every hyperscaler is designing custom silicon. The assumption is baked in: if you want faster inference, you buy bigger chips or build your own.

Kog founder Gaël Delalleau sees it differently. His team hit 3,000 tokens per second per request on standard AMD MI300X and Nvidia H200 GPUs using a 2-billion-parameter model called Laneformer 2B that they open-sourced on Hugging Face. The demo wasn't on exotic hardware. It was on the same GPUs enterprises already own.

The promise is "30x faster LLM inference" through the Kog Inference Engine (KIE). That's a massive claim, and the skepticism is warranted. The demo used a tiny model. Scaling the same approach to 70B or 400B parameter models is the real test. Delalleau says they're targeting the first major model at 10x speed by September.

What strikes me is the mindset. Delalleau studied solid-state physics at École Polytechnique, then spent years in offensive cybersecurity, a four-time DEFCON CTF finalist. He brings a reverse-engineering mentality to GPU optimization: understand the laws of the hardware at the assembly level, then exploit what the vendor didn't intend. "There's this mindset of understanding the laws of physics, and the laws of the GPU in order to make the most of them," he told TechCrunch.

That hands-on, low-level approach has a ceiling. "For every new GPU, we'll dedicate several weeks or even months to really dig into the details," Delalleau said. With 11 people, Kog can only support so many chips. The long-term play is agent-based pipelines that automate the GPU research, but that's later.

Meanwhile, the market signal is real. They had 200 tangible business leads after the Hacker News launch. Veteran Claude Code users know the pain: sometimes you wait hours for results. Anthropic charges a price multiple for Fast Mode. Kog targets the teams put off by those delays, professional workflows where speed equals revenue.

I keep coming back to the tension between the hardware arms race and software that unlocks what's already there. The hyperscalers are spending billions on custom silicon. Kog is betting that for most enterprises, the GPU they already bought has headroom nobody has touched yet.

If they pull off 10x on a real model by September, the conversation shifts. Not "which chip do I buy?" but "am I leaving performance on the table?"

The numbers matter because inference cost is becoming the dominant budget line for AI products. A 10x speedup isn't just a nice-to-have. It changes the unit economics of every AI feature.

I'll be watching whether Kog can translate the Laneformer demo into production-grade speed on Llama-3-70B or whatever comes next. The hardware race gets the headlines. The software squeeze might be where the money actually is.