GPT-5.6’s Benchmarks Are Great — Nobody Is Talking About Them

AI

GPT-5.6’s Benchmarks Are Great — Nobody Is Talking About Them

I scrolled through all 940 comments on the Hacker News thread about GPT-5.6. The benchmark chart came up at most a handful of times.

OpenAI had reason to expect otherwise. The company shipped three new models, Sol, Terra, and Luna, alongside a workplace tool called ChatGPT Work and a voice model, GPT-Live, that can listen and speak at the same time. Sol scores 80 on the SWE-bench coding benchmark, 2.8 points ahead of Anthropic’s Fable 5, using less than half the tokens and half the time. On ARC-AGI-3, Sol hit 7.8%, the first frontier model to clear a level at all. Pricing came out exactly where you’d guess. Sol runs $5/$30 per million tokens, Terra $2.5/$15, Luna $1/$6. Pay more, get smarter.

The release carried baggage, too. Weeks earlier, the U.S. government had asked OpenAI to hold Sol back, citing national security. Early access went to a short list of government-vetted partners, approved on a customer-by-customer basis. The restriction lifted the day before launch, and all three models shipped together.

None of that dominated the thread. What did was smaller, stranger, and closer to how people actually use these models every day.

Telling It to Be Brief Now Backfires

Buried in OpenAI’s new prompting guide is a line most readers will skim past. Avoid generic brevity instructions like “be concise” or “keep it short.” The model, apparently, is oversensitive to them. Say “be brief,” and Sol tends to say too little.

That’s a strange reversal. For two years, “keep it short” was rule one for prompting any model, because the defaults were bloated with hedges and disclaimers. Now the opposite failure shows up. Ask for brevity, and you get a worse answer, not a shorter good one. OpenAI’s fix is a separate verbosity parameter, with length and structure requirements embedded in the prompt text rather than a one-line instruction.

One commenter got at the real problem. Call it optimism bias baked into the model. It assumes it has correctly guessed what you want, and the people whose needs sit outside the mainstream distribution quietly lose out. Others pushed back. A smarter model, they argued, should calibrate its own verbosity from context, and less padding is progress, not a regression.

I side with the first camp. A model that asks when it’s unsure beats one that guesses confidently and gets it wrong, especially when the guessing happens inside a set of weights nobody outside OpenAI can inspect or tune.

Benchmarks and Real Code Live in Different Worlds

OpenAI’s chart shows Sol writes code faster and more cleanly. One developer measured it directly. Sol’s solution ran five times longer than the human reference answer, and roughly twice as bloated as Fable 5’s. The “half the tokens, half the time” chart, it turns out, likely measures Sol against its own predecessor, not against what a human would write.

This isn’t a new complaint. Another developer described an old GPT-5 habit. Ask for a plain frontend, get a purple-blue gradient interface with embossed buttons and drop shadows nobody asked for. Wrestling a plain Bootstrap layout out of the model took real effort. Guessing what “impressive” looks like and guessing what a user wants turn out to be the same failure wearing two different outfits.

Benchmarks sell the product. The people writing the code vote with their gut. This round, the gut vote wasn’t kind.

Quotas Decide More Than Scores Do

The thread’s heaviest discussion wasn’t about intelligence at all. Once frontier models converge on similar capability, the argument shifts from which model is smarter to whose scheduling layer works better, and whose quota goes further.

One developer runs a setup inside OpenCode where gpt-5.6-sol orchestrates, and deepseek-v4-pro handles subtasks. The main session gets read-only file access; everything else gets delegated through an open-source multi-agent workflow. The whole pipeline barely touched its quota ceiling. Cost efficiency beat anything running on a single vendor. The question shifted from which model wins to how to stitch several vendors’ models into a single pipeline.

Quota policy has become its own loyalty program. Some developers complained Anthropic’s weekly limits reset on the company’s schedule and keep tightening. Others ran the math and found GPT-5.6 Sol on its highest reasoning tier burns through quota at a much slower rate for the same workload, close to unlimited use. At least one previously all-in Anthropic user said this release changed their mind entirely, not because Fable got worse, but because the quota rules stopped feeling safe to build around.

Once the intelligence gap narrows past some threshold, loyalty stops tracking the leaderboard and starts tracking whoever built the higher wall and runs the tighter ledger.

Even the Names Sparked a Fight

Sol, Terra, and Luna kept the thread lighthearted. Some readers found the tiers instantly legible. Solar system bodies, size order obvious at a glance. Non-native English speakers pushed back, saying the naming carried none of that intuitive weight for them. Best joke in the thread: Terra for getting things done here on Earth, Luna for the moonshot projects, Sol reserved for whenever you’re ready to fly straight into the sun.

Benchmarks are for investors and press coverage. Names are what stick with the people who use the product every day.

Is It Really “Getting Smarter”?

A familiar argument reignited, too. Are a model’s visible “thinking” traces actual reasoning, or a trained summary performance dressed up to look like one? Calling a model “smarter,” saying it “learned something,” is that dangerous anthropomorphizing, or just accurate shorthand? Neither side moved the other. Underneath, it’s the same old fight about whether a pile of matrix multiplication earns the label "intelligence".

One question cut sharper than the rest. OpenAI’s guide claims that Sol now infers user intent better. Is that capability just the product of training on an ocean of user conversation history? Nobody had a confirmed answer, but the question itself beats “is it smart or not” by a wide margin.

A Global Launch That Wasn’t Quite Global

Back to the national security review that opened this piece. The restriction lifted, and the announcement read clean: a global public release. Reality looked messier. A user in Switzerland needed a U.S. VPN just to load the announcement page. A user in Spain, ten hours after launch, still couldn’t get in, even on a paid Pro plan. One narrative says the government signed off and the world got access. The other says ordinary users sat waiting inside a patchwork of timezone and region allowlists. That mismatch tells you more about this release than any benchmark chart does. The story moves fast. The rollout doesn’t.

A model’s day-to-day usefulness was never going to live in a benchmark table. It lives in whether the model guessed your intent correctly, whether the code it wrote is something you’d want to maintain, whether the quota math pencils out, and whether the thing loads at all where you happen to live. Those are the variables that decide which model you reach for tomorrow, not the bar chart from launch day.

My own read: Luna, the cheap tier, is worth trying for grunt work. Several developers reported clearing routine Jira tickets on it fast and cheap. Swapping an entire agent workflow for a 2.8-point benchmark gap is a different call, and I’m not making it yet. What this release signals is that model intelligence is approaching diminishing returns, while quota policy, tooling compatibility, and even a government’s approval pipeline are becoming the variables that determine what developers choose.