Tag: AI Benchmarks

  • OpenAI Jalapeño Chip Beats Blackwell 1.9x Per Watt — Ships 2027

    The OpenAI Jalapeño chip, the company’s first custom inference ASIC, delivered 1.5x to 1.9x more AI work per watt than Nvidia’s Blackwell systems in SemiAnalysis InferenceX tests published August 25, 2026. It draws 700W against GB300’s 1,400W and cut end-to-end latency by up to 3.6x. The catch: these are engineering samples. Volume deployment does not arrive until 2027.

    What is the OpenAI Jalapeño chip?

    The OpenAI Jalapeño chip is a custom inference accelerator co-developed with Broadcom and fabricated on TSMC’s N3P node. It is built to serve tokens, not train models. OpenAI published its first third-party benchmarks this week, and they are better than any first-generation silicon has a right to be.

    The headline spec: 13.4 PFLOPS of MXFP4 compute at a 700W rating, paired with HBM4 running at 15.4 TB/s of bandwidth. In sustained operation the part draws under 550W, according to the benchmark data reported by ForkLog.

    Nvidia’s GB200 rack unit pulls 1,200W. GB300 pulls 1,400W. Rubin sits between 900W and 1,150W. Jalapeño is doing its work in roughly half the power envelope.

    The timeline is the real story

    OpenAI started design in mid-2024 and handed the chip to the fab in November 2025. That is nine months from first design to manufacturing handoff, and 16 months to tape-out — a schedule that normally takes a silicon team two to three years.

    OpenAI says its own models helped design the chip. That claim is unverifiable from the outside, but the calendar is not.

    “Jalapeño can serve more AI work per unit of power, while also returning responses more quickly,” said Richard Ho, OpenAI’s head of hardware, in comments reported by TechCrunch.

    How much faster is Jalapeño than Nvidia Blackwell?

    Across three open-weight models, Jalapeño roughly doubled Nvidia’s tokens per second per kilowatt while cutting latency by 43% to 72%. The gap widens as models get larger. On DeepSeek R1 670B, Jalapeño returned a first response in 1.65 seconds against GB300’s 5.99 seconds.

    Here are the SemiAnalysis InferenceX results as reported by ForkLog:

    Model Jalapeño (mixed TPS/kW) Nvidia system Nvidia (mixed TPS/kW) Jalapeño latency Nvidia latency
    GPT-OSS 120B 85,448 GB200 44,960 1.03s 1.80s
    DeepSeek R1 670B 19,641 GB300 11,781 1.65s 5.99s
    Kimi K2.5 1T 18,195 GB300 11,862 1.56s 5.31s

    On single-user throughput, Jalapeño hit roughly 1,400 tokens per second on GPT-OSS 120B and over 700 tokens per second on DeepSeek R1 670B.

    The aggregate claims are wider still: 1.7x to 3.6x lower end-to-end latency and 2.1x to 4.1x higher performance on interactive workloads, per The Decoder. At matched decoding speeds, The Decoder reported token-throughput-per-kilowatt advantages of 54x to 104x — a number that only makes sense in the narrow regime where GPU batching collapses.

    What SemiAnalysis actually said

    “Usually first generation chips aren’t competitive, but OpenAI is beating Nvidia Blackwell and even Rubin,” SemiAnalysis CEO Dylan Patel said, per The Decoder.

    That is a strong endorsement from an analyst house that sells research to the same hyperscalers buying Nvidia racks. Take it seriously. Take it with salt.

    Why does performance per watt decide who wins?

    Because power, not silicon, is the binding constraint on AI buildouts in 2026. Data center operators are queuing for grid interconnects measured in years. If a chip does the same work at half the watts, the same substation serves twice the revenue.

    That math is why custom ASICs keep appearing. Every watt saved on inference is a watt available for a paying customer, and inference is now the majority of frontier-lab compute spend.

    OpenAI CFO Sarah Friar framed it in cost terms: custom chips give the company “greater control over inference costs” and let it match hardware to specific tasks. Friar also said the chip “complements” existing partnerships rather than replacing them — corporate language for we are still buying your GPUs, please keep taking our calls.

    We covered the same power-and-memory squeeze from the supply side in our piece on the Nvidia AI server price hike, and the economics of fast inference in Cerebras vs Groq.

    What does this do to Nvidia’s margins?

    Nothing this quarter. Nvidia reported Q2 fiscal 2027 revenue of $96.22 billion on August 26, beating the $92.07 billion consensus, with data center revenue of $89.02 billion — up 117% year over year, according to 24/7 Wall St. EPS came in at $2.22 against a $2.09 estimate.

    Guidance was louder than the beat. Nvidia guided Q3 to $108 billion plus or minus 2%, with non-GAAP gross margins near 74% and no China data center compute revenue assumed.

    “AI has reached its inflection point. It’s doing useful work. Its tokens are productive and profitable. Now, compute is revenue,” CEO Jensen Huang said on the call.

    Nvidia also disclosed supply commitments of $279 billion, largely for Vera Rubin memory. That is a company buying ahead, not one bracing for demand loss.

    The threat is 2028, not 2026

    Custom silicon does not eat Nvidia’s revenue. It eats Nvidia’s pricing power. A 74% gross margin exists because there is no substitute at scale. Jalapeño is the first credible substitute built by Nvidia’s single largest customer.

    NVDA closed at $213.05 before the print, down 3.04% on the week and up 14.37% year to date, per 24/7 Wall St. The stock has fallen after four of its last five earnings reports despite beating consensus three quarters running.

    Who wins and who loses financially?

    Broadcom is the clearest winner. It gets ASIC design revenue, a marquee reference customer, and validation that its custom-silicon business can beat the merchant-GPU incumbent on a first attempt. Nvidia is the clearest medium-term loser, though the damage lands in 2028 pricing, not 2026 volume.

    • Broadcom — books high-margin custom ASIC revenue and proves the model. We covered its financing appetite in the Broadcom AI debt deal.
    • TSMC — wins either way. N3P wafers are N3P wafers, whether the logo says Nvidia or OpenAI.
    • HBM suppliers — Jalapeño uses HBM4 at 15.4 TB/s. More custom chips means more high-bandwidth memory demand, not less.
    • OpenAI — gains leverage in every future GPU negotiation, which may be worth more than the chip itself. Its Nvidia relationship already shifted once, as we noted when Nvidia cut its OpenAI data center guarantee.
    • Nvidia — keeps the volume through 2027, then defends 74% margins against a credible in-house alternative.
    • Second-tier inference clouds — squeezed hardest. They rent GPUs at market rates and cannot design their own.

    What’s the catch with the Jalapeño benchmarks?

    Three catches, and they matter. Jalapeño exists as engineering samples only. Rubin is already shipping to customers. And the benchmark set was chosen by the chip’s owner, run on three open-weight models, with two of Nvidia’s standard optimizations absent from the comparison.

    The Decoder reported that Jalapeño lacks multi-token prediction and speculative decoding optimizations. Those are exactly the techniques that close latency gaps on GPUs. Adding them later helps Jalapeño; adding them to the comparison today would narrow the gap.

    The models tested were GPT-OSS 120B, DeepSeek R1 670B and Kimi K2.5 1T. Larger current-generation models — DeepSeek V4 Pro, Kimi K3 — were not tested at all. Neither, notably, was any GPT-5-class OpenAI frontier model, which is the workload the chip actually has to serve.

    And the deployment schedule is honest about itself: very small volumes at the end of 2026, meaningful volume in 2027. OpenAI says a second generation is in advanced development and a third is in design.

    A chip that wins benchmarks in August 2026 must still win against whatever Nvidia ships in 2027. That is a different race.

    Frequently asked questions

    Is the OpenAI Jalapeño chip available to buy?

    No. It is an internal accelerator for OpenAI’s own inference fleet, currently at engineering-sample stage. Small-volume deployment starts at the end of 2026, with wider rollout in 2027. There is no external sales channel announced.

    Who manufactures the Jalapeño chip?

    Broadcom co-developed it with OpenAI, and TSMC fabricates it on the N3P process node. The benchmarked silicon is B0 stepping, meaning at least one revision past first tape-out.

    Does Jalapeño beat Nvidia’s Rubin?

    On the perf-per-watt figures SemiAnalysis published, yes — 1.5x to 1.9x. But Rubin is shipping to paying customers now and Jalapeño is not, so the comparison is between a product and a prototype.

    Can Jalapeño train models?

    No. It is an inference-only design. OpenAI still needs GPUs for training, which is why CFO Sarah Friar described the chip as complementing rather than replacing existing supplier relationships.

    How much power does Jalapeño use?

    It is rated at 700W and reportedly sustains under 550W in operation. Nvidia’s GB200 draws 1,200W and GB300 draws 1,400W, so Jalapeño operates in roughly half the envelope.

    Did Nvidia’s earnings show any damage from custom chips?

    None yet. Data center revenue grew 117% year over year to $89.02 billion and Q3 guidance is $108 billion. Custom silicon is a 2028 margin question, not a 2026 revenue question.

    What benchmark was used?

    SemiAnalysis InferenceX, which measures mixed tokens per second per kilowatt alongside end-to-end latency. It is a third-party benchmark, but the model selection and test configuration came from the chip’s owner.

    The bottom line

    Jalapeño is the most serious first-generation AI accelerator anyone has produced, and the power numbers are the part that should worry Nvidia. Half the watts for double the tokens is not a rounding error; it is a structural argument for custom silicon at every lab large enough to fund a design team.

    But the trade here is not “sell Nvidia.” Nvidia just printed $96.22 billion in a quarter and guided to $108 billion. The trade is that Nvidia’s 74% gross margin now has an expiry date attached, and the market will start pricing that date long before 2028 arrives.

    The honest read: OpenAI has proven it can build a chip. It has not yet proven it can build ten million of them, on schedule, while Nvidia iterates annually. Benchmarks are cheap. Yield is not.

    Sources

  • Best Computer Use Agent 2026: Qwen-UI-Agent vs Opus 4.8

    Alibaba’s Qwen-UI-Agent posts the strongest mobile GUI scores anyone has published — 92.2% on MobileWorld-Real — and you cannot buy it. No weights, no API, no price. For a computer use agent you can deploy this week, Claude Opus 4.8 still owns desktop reliability at 83.4% on OSWorld-Verified, and Holo 3.1 wins on cost per task. Everything else is a benchmark press release.

    What is a computer use agent, and why is everyone shipping one now?

    A computer use agent is a model that operates a screen directly — clicking, typing, scrolling — instead of calling an API. It reads screenshots, decides the next action, and repeats. That makes it the only automation layer that works on software with no integration surface.

    The category went from research demo to procurement question fast. Temporal’s 2026 State of Development Report found daily agent usage among surveyed engineers jumped from 47.3% to 80.8% this year. Agents are now normal. Agents that drive a mouse are the next line item.

    The benchmark that actually decides purchases

    OSWorld-Verified is the number buyers use. It runs 369 real desktop tasks in a live Linux environment and scores binary success — the task worked, or it did not.

    On BenchLM’s August 2026 OSWorld-Verified board, Claude Opus 4.8 sits at 83.4%, Gemini 3.6 Flash at 83.0%, and GPT-5.5 at 78.7%. The top of that board is clustered inside about three points. That clustering is the whole story: on desktop, model choice barely moves the needle anymore. Price does.

    Is Qwen-UI-Agent actually better than Claude Opus 4.8?

    On phones, yes, and not by a little. On desktops, no. The Qwen-UI-Agent technical report from Alibaba’s Tongyi MAI team claims 92.2% on MobileWorld-Real, beating Claude Opus 4.8 by 7.5 points and GPT-5.6 Sol by 6.8. On OSWorld-Verified it lands second at 79.5%.

    Where Qwen-UI-Agent wins

    Mobile is a rout. The report puts Qwen-UI-Agent at 82.1% on MobileWorld — 14.6 points ahead of Opus 4.8, 12.0 ahead of GPT-5.6 Sol — and 97.5% on AndroidDaily.

    Grounding is the other win. It reports 81.5% on ScreenSpot-Pro under the zoom-in setting and 97.5% on ScreenSpot-V2. Grounding is the skill of hitting the right pixel, and it is where most agents quietly fail.

    It also takes WebArena at 73.6%, 1.7 points above Opus 4.8.

    Where it loses

    Desktop. Opus 4.8’s 83.4% on OSWorld-Verified beats Qwen-UI-Agent’s 79.5% by 3.9 points, and on the harder OSWorld-v2 the report concedes 13.9% binary success against a 40.0% partial-progress score. Long desktop chains still break.

    Deep research is worse. BrowseComp comes in at 64.1% — roughly 26 points behind GPT-5.6 on the same test, per eesel AI’s breakdown.

    Can you actually buy Qwen-UI-Agent right now?

    No. There is no API, no published price, and no checkpoint on Hugging Face. The Tongyi-MAI repository ships a technical report, demo videos, and cookbook notebooks. The Apache 2.0 license covers the repo contents, not a model you can run.

    What is downloadable is the previous generation: MAI-UI 8B (8.77B parameters) at 65.7% on ScreenSpot-Pro and MAI-UI 2B (2.13B) at 57.4%. The larger 235B-A22B variant tops AndroidWorld at 76.7% and is not published either.

    Treat the headline scores as a roadmap, not a product. That gap between announcement and availability is the single most expensive mistake in agent procurement this year.

    How much does a computer use agent cost in 2026?

    Between $0.03 and $30 per million input tokens, depending on tier. Screen-driving is token-hungry: every step ships a fresh screenshot, so a computer use agent burns input tokens at several times the rate of a chat workload. Input price, not output price, is what sets your bill.

    Agent / model Access Input $/M Output $/M Best published score
    Claude Opus 4.8 Hosted API (GA) $5.00 $25.00 83.4% OSWorld-Verified
    Claude Sonnet 5 Hosted API (GA) $2.00 $10.00 Mid-tier desktop
    GPT-5.6 Sol Hosted API (GA) $5.00 $30.00 92.8% Online-Mind2Web
    GPT-5.6 Luna Hosted API (GA) $0.20 $1.20 Budget browser tier
    Gemini 3.7 Flash Hosted API (preview) $0.75 $3.75 83.0% (3.6 Flash)
    Holo 3.1 Open weights + API $0.25 $1.80 82.56% OSWorld-Verified
    Qwen3.5 Flash Hosted API $0.10 $0.40 Grounding / cheap steps
    Qwen-UI-Agent Not released 92.2% MobileWorld-Real

    Claude prices come from Anthropic’s pricing documentation; OpenAI, Gemini and Qwen rates from BenchLM’s August 2026 tables.

    The token math nobody prices in

    Four multipliers decide whether your pilot survives contact with finance.

    • Tokenizer drift. Anthropic documents that Claude 4.7 and later use a newer tokenizer producing roughly 30% more tokens for the same text. Same price, bigger invoice.
    • Cache hits. Cached input bills at 0.1x on Claude and 10% of standard on OpenAI. For screenshot loops with a stable system prompt, that is the difference between viable and absurd.
    • Batch discounts. Both Anthropic and OpenAI cut 50% for asynchronous batch work. Most agent tasks are not urgent.
    • Fast mode. Anthropic’s fast tier for Opus 5 and 4.8 runs $10 input and $50 output — double the standard rate for latency you probably do not need on a background agent.

    One more piece of good news for budgets: Anthropic made Sonnet 5’s introductory $2/$10 pricing permanent, canceling the increase to $3/$15 that was scheduled for September 1. If you built a cost model around that hike, tear it up. We covered the token-efficiency side of this in Agent Skills vs MCP.

    Which computer use agent should you pick for your use case?

    Match the agent to the surface, not to the leaderboard. Desktop reliability, mobile coverage, browser scale and self-hosting are four different purchases, and no single vendor currently wins all four. Here is the split as of late August 2026.

    Use case Pick Why Cost signal
    Desktop work where errors are expensive Claude Opus 4.8 83.4% OSWorld-Verified; prompt-injection classifiers run over every screenshot $5/$25 per M tokens
    Phone plus desktop from one endpoint Gemini Computer Use Only hosted API with a documented Android environment $0.75/$3.75 (3.7 Flash)
    Browser-only automation at volume Browser Use 87.4% on Odysseys; model-agnostic, cloud supplies Chromium From $29/mo, $0.02/browser hour
    Data cannot leave your network Holo 3.1 82.56% OSWorld-Verified on a maintainer-run board; Apache 2.0 $0.25/$1.80 or self-host free
    Logged-in flows blocked by 2FA Skyvern Credentials never sent to the model; 85.85% WebVoyager From $29/mo; 2FA at $149/mo
    Commercial product, permissive license UI-TARS Apache 2.0, ~808k monthly downloads Free weights

    Are open-weight GUI agents good enough yet?

    One is. Holo 3.1 scores 82.56% on OSWorld-Verified on a maintainer-run board — within a point of Opus 4.8 — under Apache 2.0, with a hosted fallback at $0.25/$1.80 per million tokens and a free tier at 10 requests per minute. That is the only open model verified by a neutral board rather than its own vendor.

    The rest are not close. ByteDance’s UI-TARS is the most downloaded GUI agent at roughly 808,000 monthly pulls, but the downloadable checkpoint scores 27.5% on OSWorld — a 15-point gap from the published figure.

    The pattern mirrors what we found in GLM-5.3 vs DeepSeek V4 Pro: open weights close the gap on paper faster than they close it in production.

    Is a computer use agent worth it in 2026?

    Only when no API exists. The most useful finding in this cycle is that the best GUI agents avoid the GUI: 55.1% of actions on OSWorld-v2 were typed commands rather than clicks. Screen-driving is the fallback, not the strategy.

    The failure data backs that up. Roughly 52% of real-device failures trace to the screen itself and 40% to the agent, meaning half your errors come from interfaces that were never designed for a machine to read.

    The financial read is blunt. If a documented API exists, an API integration costs less, breaks less, and does not bill you for screenshots. Reserve computer use for legacy desktop software, vendor portals with no API, and mobile apps. That is a real and expensive category — it is just smaller than the demos suggest. The same discipline applies to coding agents, which we priced in Claude Code vs Codex CLI.

    Frequently asked questions

    What is the best computer use agent right now?

    Claude Opus 4.8 for desktop reliability at 83.4% OSWorld-Verified, Gemini Computer Use if you need Android, and Holo 3.1 if you need open weights. Qwen-UI-Agent has the best benchmarks and no product.

    When will Qwen-UI-Agent weights be released?

    Alibaba has not announced a date. As of late August 2026 the repository has no releases and no checkpoint on Hugging Face. Only the older MAI-UI 2B and 8B models are downloadable.

    How much does Claude computer use cost?

    Claude Opus 4.8 bills $5 per million input tokens and $25 per million output, per Anthropic’s pricing docs. Cache hits drop input to 0.1x and batch processing cuts both sides by 50%.

    Is Claude Sonnet 5 still going up in price on September 1?

    No. Anthropic canceled the increase and made the $2/$10 introductory pricing permanent. The previously scheduled $3/$15 rate will not take effect.

    Do computer use agents work on mobile?

    Partially. Gemini Computer Use is the only hosted API with a documented Android environment. Claude’s computer use has no mobile support. Qwen-UI-Agent leads mobile benchmarks but ships nothing.

    Are computer use agents safe to run on production systems?

    Only with guardrails. Anthropic runs prompt-injection classifiers over every screenshot automatically; Gemini’s injection detection is off by default, and OpenAI leaves safety implementation to you. Scope credentials tightly and log every action.

    What is OSWorld-Verified?

    A 369-task benchmark that scores computer use agents on real desktop work in a live environment, pass or fail. It is the closest thing the category has to a purchasing standard.

    The bottom line

    Buy Claude Opus 4.8 for desktop work where a failed task costs more than a token bill — the 83.4% OSWorld-Verified score and automatic injection screening justify $5/$25. Buy Holo 3.1 if data residency or cost dominates: 82.56% for $0.25/$1.80 is the best price-per-point in the category by a wide margin.

    Ignore Qwen-UI-Agent until weights land. A 92.2% score you cannot run is worth exactly zero. And before you buy any of them, check whether the target system has an API — because 55.1% of the winning agent’s own actions were typed commands, which is the benchmark quietly telling you to skip the screen entirely.

    Sources

  • Best o3 Replacement: What to Use After the August 26 Cutoff

    OpenAI retires o3 from ChatGPT on August 26, 2026. The best o3 replacement for most teams is GPT-5.6 Terra at $2/$12 per million tokens — not the officially recommended Sol at $5/$30. Terra posts 90.4% on GPQA Diamond against o3’s 87.7%, at 60% less output cost. API users are not on the same clock: their o3 shutdown is December 11.

    The reasoning model that defined 2025 is being switched off. And the migration advice OpenAI published is the expensive option.

    Here is what actually changes on Tuesday, what each o3 replacement costs, and which one wins for your workload.

    What exactly happens to o3 on August 26, 2026?

    o3 disappears from the ChatGPT model picker on August 26, 2026. That is a product change, not an API shutdown. OpenAI’s release notes from May 28, 2026 confirm o3 is “retired from ChatGPT on August 26, 2026 following a 90-day sunset period.” Developers calling o3 over the API keep working past that date.

    The ChatGPT side: the picker already stopped naming models

    ChatGPT users lose nothing they can still see. Since the June 10, 2026 model picker update, OpenAI stopped exposing version numbers entirely.

    The picker now offers Instant, Medium, High, and Extra High, plus Pro Standard and Pro Extended on Pro plans. Reasoning is sold as effort, not as a model name.

    So for consumer subscribers, the o3 replacement is already installed. Selecting High is the closest analogue to what o3 Thinking used to do.

    The API side: your real deadline is December 11

    This is where most coverage gets it wrong. OpenAI’s deprecations page lists o3-2025-04-16 with a shutdown date of December 11, 2026, migrating to gpt-5.6-sol. o3-pro-2025-06-10 follows the same date, moving to Sol with reasoning.mode: pro.

    What does die on August 26 is the legacy Assistants API. Anything built on Assistants must move to the Responses and Conversations APIs by Tuesday. That is the deadline worth panicking about.

    Which o3 replacement is best for most workloads?

    GPT-5.6 Terra. OpenAI names Sol as the official successor, but Sol is priced for frontier reasoning at $5/$30 per million tokens. Terra sits at $2/$12 and clears o3 on the benchmarks that mattered to o3 users. For the overwhelming majority of o3 traffic, paying Sol rates is a rounding error you repeat a million times.

    Terra got cheaper on July 30, 2026, when OpenAI cut its price roughly 20% as part of the GPT-5.6 pricing refresh. Luna fell about 80% in the same announcement, to $0.20/$1.20.

    The quality case is not a stretch either. OpenRouter’s provider data puts Terra at 90.4% on GPQA Diamond and 75.3% on TAU-Bench. o3 scored 87.7% on GPQA Diamond at launch.

    Why Terra beats Sol on cost per task

    Reasoning models bill you for tokens you never see. A 500-token visible answer can consume 2,000+ tokens once hidden reasoning is counted.

    That multiplier is exactly why output price dominates your bill. At $12 versus $30 per million output tokens, Terra cuts the expensive half of the invoice by 60%.

    Terra also carries a 1,050,000-token context window with 128,000 max output — over five times o3’s 200K context. You are not trading capability down.

    How much does each o3 replacement cost?

    Prices below are list rates per million tokens as of August 24, 2026, pulled from provider pricing pages. Cached input bills at 10% of standard rates on OpenAI, and the Batch API halves both sides.

    Model Input Output Context Notes
    o3 (retiring) $2.00 $8.00 200K API shutdown Dec 11, 2026
    o3-pro (retiring) $20.00 $80.00 200K API shutdown Dec 11, 2026
    GPT-5.6 Terra $2.00 $12.00 1.05M Cut ~20% on Jul 30, 2026
    GPT-5.6 Sol $5.00 $30.00 1.05M Official o3 successor
    GPT-5.6 Luna $0.20 $1.20 1.05M Cut ~80% on Jul 30, 2026
    Claude Opus 5 $5.00 $25.00 200K Fast Mode is $10/$50
    Claude Sonnet 5 $2.00 $10.00 200K $2/$10 now permanent
    Gemini 3.7 Flash $0.75 $3.75 Doubles Jan 1, 2027
    Kimi K3 $2.60 $13.00 1M Open weights, 2.8T params

    Read that table one way and the story is obvious: o3 at $2/$8 was cheap, and every direct successor except Luna and Gemini Flash costs more per output token. Migration is a price increase unless you choose deliberately.

    Is GPT-5.6 Sol worth $5/$30 in 2026?

    Only for the top slice of your traffic. Sol is the frontier tier and the only model OpenAI formally maps o3-pro onto, via reasoning.mode: pro. If you were paying o3-pro’s $20/$80, Sol at $5/$30 is a 75% input cut and a 62.5% output cut.

    If you were on standard o3, Sol is a 150% input increase and a 275% output increase. Same model family, opposite financial outcome.

    There is also a long-context trap. Requests beyond the standard threshold reprice: Sol rises to $10/$45, Terra to $4/$18, Luna to $0.40/$1.80. Feeding a million-token repo into Sol is a different product than a 20K-token prompt.

    Which o3 replacement should you pick for your use case?

    Match the model to the job, not to the vendor’s migration note. Below is where each option earns its price, based on published benchmarks and list pricing. Benchmark your own top 20 prompts against Terra before escalating anything to Sol.

    Use case Pick Why
    General o3 traffic GPT-5.6 Terra 90.4% GPQA Diamond at $2/$12; 1.05M context
    Hardest reasoning, o3-pro traffic GPT-5.6 Sol Official reasoning.mode: pro path; 62.5% cheaper output than o3-pro
    High-volume classification GPT-5.6 Luna $0.20/$1.20 after the ~80% July cut
    Agentic coding, terminal work Gemini 3.7 Flash 85.8% on Terminal-bench 2.1 at $0.75/$3.75
    Long document analysis Claude Opus 5 $5/$25 with 200K context, cheaper output than Sol
    Balanced daily driver Claude Sonnet 5 $2/$10 locked permanently
    Self-hosting, data residency Kimi K3 2.8T open weights, 1M context, $2.60/$13 hosted

    What about Claude and Gemini as an o3 replacement?

    Both are live options, and both moved on price in the last two weeks. Anthropic canceled a scheduled increase; Google launched a discount with an expiry date attached. Those two facts change the math more than any benchmark did.

    Claude Sonnet 5’s price freeze quietly killed a 50% increase

    Sonnet 5 launched at $2/$10 as introductory pricing set to expire August 31, 2026, with a jump to $3/$15 scheduled for September 1. Anthropic’s pricing documentation now states that increase “will not occur” and $2/$10 is the standard price.

    That makes Sonnet 5 an exact price match to o3 on input and 25% more on output — the closest financial like-for-like swap available. We broke down how it stacks up against Google’s cheap tier in Gemini 3.7 Flash vs Claude Sonnet 5.

    Gemini 3.7 Flash is the cheapest credible option — until January

    Gemini 3.7 Flash shipped August 13, 2026 with a 50% introductory cut to $0.75/$3.75. On January 1, 2027 it reverts to $1.50/$7.50, and context caching moves from $0.075 to $0.15.

    Its numbers are strong where agents live: 85.8% on Terminal-bench 2.1 and 65.3% on DeepSWE v1.1, though only 43.6% on FrontierCode 1.1 Main. Build your 2027 budget on the standard rate, not the promo.

    How do you migrate off o3 without breaking production?

    Treat this as a pricing audit, not a find-and-replace. The single most expensive mistake is routing all o3 traffic to Sol because the deprecation table said so. Work through it in this order:

    1. Split the Assistants API work out first. It dies August 26, 2026 — 107 days before o3 does. Move to Responses and Conversations now.
    2. Pull your last 30 days of o3 token spend and split it by input versus output. Output volume decides which tier you can afford.
    3. Replay your top 20 prompts against Terra. If quality holds, you are done at $2/$12.
    4. Escalate only the failures to Sol. Route by task difficulty, not by default.
    5. Push bulk classification to Luna or Gemini 3.7 Flash. At $0.20/$1.20, Luna makes some batch jobs nearly free.
    6. Turn on prompt caching and the Batch API. Cache hits bill at 10%; batch halves everything.

    Teams running agentic coding harnesses should also re-check their tooling layer, not just the model. We covered that trade-off in Claude Code vs Codex CLI.

    Frequently asked questions about the o3 replacement

    Is o3 gone from the API on August 26, 2026?

    No. August 26 is the ChatGPT retirement date. OpenAI’s deprecations page lists the o3-2025-04-16 API shutdown as December 11, 2026, with gpt-5.6-sol as the migration target.

    What is the cheapest o3 replacement?

    GPT-5.6 Luna at $0.20/$1.20 per million tokens, following its roughly 80% price cut on July 30, 2026. For work needing more reasoning depth, Gemini 3.7 Flash at $0.75/$3.75 is the next step up.

    Does GPT-5.6 Terra actually beat o3?

    On GPQA Diamond, yes: 90.4% for Terra on OpenRouter’s provider data versus 87.7% for o3 at launch. Terra also carries a 1.05M context window against o3’s 200K.

    What replaces o3-pro?

    GPT-5.6 Sol with reasoning.mode: pro, per OpenAI’s deprecations table. At $5/$30 versus o3-pro’s $20/$80, it is a substantial price cut for that specific tier.

    Will Claude Sonnet 5 get more expensive in September?

    No. Anthropic canceled the September 1, 2026 increase to $3/$15. The $2/$10 rate is now permanent per its pricing docs.

    Is there an open-weights o3 replacement?

    Kimi K3 is the closest: 2.8 trillion parameters, 1M context, released July 16, 2026, and $2.60/$13 hosted on OpenRouter. Chinese open-weight coders are also competitive — see our GLM-5.3 vs DeepSeek V4 Pro breakdown.

    What breaks on August 26 if I do nothing?

    Two things: o3 vanishes from the ChatGPT picker, and the legacy Assistants API stops working. API calls to o3 itself keep running until December 11, 2026.

    The bottom line

    Move general o3 traffic to GPT-5.6 Terra at $2/$12 and stop there. It beats o3 on GPQA Diamond, gives you five times the context, and costs 60% less per output token than the Sol tier OpenAI points you at.

    Send only o3-pro-class work to Sol, where $5/$30 is genuinely a 62.5% output discount on what you were paying. Send bulk work to Luna at $0.20/$1.20.

    The decision depends on exactly one number: your output-token share. Above roughly 30% of spend, tier choice dominates everything else on your invoice. Below that, input caching matters more than which model you pick.

    And fix your Assistants API code before Tuesday. That is the only hard deadline this week. For more on where frontier pricing is heading, see our analysis of DeepSeek’s vision model against Claude Opus 4.8.

    Sources

  • DeepSeek Vision Model vs Claude Opus 4.8: 23x Cheaper Per Image

    DeepSeek’s vision model just split a 2-2 draw with Claude Opus 4.8 on visual benchmarks while charging roughly 23 times less per image. V4-Flash-Vision-Exp processes 1,000 images for about $0.17 at peak rates. Claude Sonnet 4.6 charges around $4 for the same work. If your workload is image-heavy and volume-driven, the math is no longer close.

    The DeepSeek vision model shipped on August 21, 2026. It is experimental, API-only, and it does something DeepSeek has done before: it removes a price floor the incumbents were standing on.

    What is DeepSeek’s new vision model?

    V4-Flash-Vision-Exp is an experimental multimodal extension of DeepSeek’s V4-Flash, adding image understanding to a model that was previously text-only. It is available on DeepSeek’s paid developer platform by setting model='deepseek-v4-flash-vision-exp', per the official API changelog.

    The target is not image generation. It is visual agent work — reading charts, parsing screenshots, driving interfaces, and running multi-step tasks that require seeing what happened.

    DeepSeek’s own framing is careful. The changelog calls it “a significant leap over DeepSeek-V4-Flash, bringing its multimodal agent capabilities close to Opus-4.8,” while noting pure-text performance is unchanged from the base model.

    What’s under the hood

    The foundation is V4-Flash, released in April 2026: a 284-billion-parameter mixture-of-experts model built from multiple 13-billion-parameter networks, according to SiliconANGLE.

    The vision specs are aggressive. A 1-million-token context window, up to 384,000 output tokens, and up to 600 images per request. Each image is capped at 384 tokens regardless of size — that cap is the entire reason the pricing works.

    Supported formats are JPEG, PNG, GIF and WebP, with a maximum edge of 8,192 pixels, dropping to 4,096 pixels once you send 15 or more images in a single call.

    Did the DeepSeek vision model actually beat Claude Opus 4.8?

    On two benchmarks yes, on two no. It is a genuine 2-2 split, not a sweep, and the margins are thin in both directions. DeepSeek wins the agentic and hard-visual tests. Opus 4.8 holds the chart-reading and structured-reasoning tests. Anyone claiming a decisive winner is selling something.

    Benchmark DeepSeek V4-Flash-Vision-Exp Claude Opus 4.8 Margin
    Agents’ Last Exam (ALE) 27.3 25.7 +1.6 DeepSeek
    ZeroBench (Pass@5) 35.0 34.0 +1.0 DeepSeek
    ApexBench (Pass@1) 36.5 39.4 +2.9 Opus 4.8
    Chartography 64.3 65.0 +0.7 Opus 4.8

    Scores as reported by XenoSpectrum’s breakdown. DeepSeek’s changelog separately lists a Terminal Bench 2.1 score of 83.9.

    Context on the tests matters. ALE contains more than 1,000 multi-step tasks; ZeroBench is 100 image-analysis tasks deliberately built to be hard, SiliconANGLE reports. Those are the two DeepSeek won.

    The pattern is consistent with what we saw when GLM-5.3 went up against DeepSeek V4 Pro: Chinese labs are now trading benchmark wins with frontier Western models rather than trailing them.

    How much does the DeepSeek vision model cost?

    Very little, and the structure is unusual. DeepSeek runs peak and off-peak pricing. Off-peak, input is $0.22 per million tokens and output is $0.66. During peak hours — 01:00–04:00 and 06:00–10:00 UTC — both double, to $0.44 and $1.32. Cache hits cost $0.007 off-peak.

    The image economics fall out of the 384-token cap. A single image costs $0.00008448 off-peak and $0.00016896 at peak.

    Model ~Cost per 1,000 images (1024×1024) Input / output per 1M tokens Context
    DeepSeek V4-Flash-Vision-Exp (off-peak) ~$0.09 $0.22 / $0.66 1M
    DeepSeek V4-Flash-Vision-Exp (peak) ~$0.17 $0.44 / $1.32 1M
    Google Gemini 3.1 Pro ~$0.50 $2.00 / $12.00 1M
    OpenAI GPT-5.4 Vision ~$1.90 $2.50 / $15.00 1.1M
    Anthropic Claude Sonnet 4.6 ~$4.00
    Anthropic Claude Opus 4.8 $5.00 / $25.00 1M

    Per-image costs are as calculated by BigGo Finance; Opus 4.8 token rates are $5.00 input, $25.00 output and $0.50 cached input, per OpenRouter.

    Set those side by side. On output tokens, DeepSeek off-peak at $0.66 against Opus 4.8 at $25.00 is a 37x spread for a model that just won two of four visual benchmarks against it.

    Why the 384-token image cap matters

    Most vision APIs price images by resolution. Send a bigger image, pay more tokens. That makes cost forecasting miserable and punishes exactly the high-resolution documents that need vision in the first place.

    DeepSeek caps every image at 384 tokens no matter its size, up to an 8,192-pixel edge. A dense scanned contract and a low-resolution thumbnail cost the same.

    For finance and operations teams, that turns a variable line item into a fixed one. You can price a document-processing product per page before you have seen a single customer’s files, which is not something the resolution-scaled competitors let you do.

    Is the DeepSeek vision model cheap enough to switch?

    For high-volume image pipelines, yes — the gap is too large to argue with. For low-volume work where quality variance costs more than tokens, no. The deciding number is how many images you process per month, because below roughly 100,000 images the absolute savings stop mattering.

    Run it concretely. A million images per month costs about $170 at DeepSeek’s peak rate, $90 off-peak, and roughly $4,000 on Claude Sonnet 4.6. That is a $46,000 annual difference on a single pipeline.

    • Document and invoice processing at scale. Switch. The 384-token cap makes cost per page nearly flat.
    • Screenshot-driven agents and RPA. Switch. ALE and ZeroBench are exactly this workload, and DeepSeek won both.
    • Chart and dashboard extraction. Stay. Opus 4.8 leads Chartography 65.0 to 64.3, and errors here are expensive.
    • Regulated or air-gapped work. Stay. This is a Chinese paid API with no open weights announced.
    • Batch archive backfill. Switch, and schedule it off-peak. You halve the bill by choosing when to run.

    That off-peak lever is genuinely underrated. Almost no batch job cares what hour it runs, and DeepSeek is effectively paying you 50% to move it.

    Which vision model should you use for what?

    Match the model to the failure cost, not to the benchmark chart. Where a wrong answer is cheap and volume is enormous, take the cheapest capable model. Where a wrong answer triggers a human review cycle or a compliance event, the token savings are irrelevant.

    Use case Pick Why
    Bulk OCR, receipts, invoices DeepSeek V4-Flash-Vision-Exp ~$0.09–$0.17 per 1,000 images; flat 384-token cost per image
    Visual agents, UI automation DeepSeek V4-Flash-Vision-Exp Wins ALE (27.3) and ZeroBench (35.0) head-to-head
    Financial charts, data extraction Claude Opus 4.8 Leads Chartography 65.0 and ApexBench 39.4
    Mixed text-plus-vision reasoning Claude Opus 4.8 DeepSeek’s text performance is unchanged from V4-Flash
    Mid-volume general vision Gemini 3.1 Pro ~$0.50 per 1,000 images with Western data residency
    Production systems needing stability Claude Opus 4.8 or Gemini DeepSeek’s model is explicitly labeled experimental

    What are the catches?

    Three, and none of them are small. The model is experimental, it is closed, and the benchmark margins are inside the noise band. Anyone budgeting a migration on a 1.0-point ZeroBench edge is over-reading the data.

    Start with the “Exp” in the name. DeepSeek ships experimental endpoints and retires them. Building a production pipeline on one is a scheduling risk, not just a quality risk.

    Second, there are no open weights. SiliconANGLE notes DeepSeek “may release a free version later on,” but nothing is confirmed. That kills the self-hosting escape hatch that made Qwen3.8-Max interesting for teams with data-residency constraints.

    Third, the wins are narrow. +1.6 on ALE and +1.0 on ZeroBench are real but slim, and DeepSeek loses ApexBench by a wider 2.9 points than either of its victories. The honest summary is parity, delivered at a fraction of the price — which is still a devastating result for Anthropic’s pricing power.

    Frequently asked questions

    What is DeepSeek V4-Flash-Vision-Exp?

    An experimental multimodal model released on August 21, 2026, extending DeepSeek’s V4-Flash with image understanding. It handles up to 600 images per request with a 1-million-token context window, available only through DeepSeek’s paid API.

    Is the DeepSeek vision model better than Claude Opus 4.8?

    It splits 2-2. DeepSeek wins Agents’ Last Exam (27.3 vs 25.7) and ZeroBench Pass@5 (35.0 vs 34.0). Opus 4.8 wins ApexBench Pass@1 (39.4 vs 36.5) and Chartography (65.0 vs 64.3).

    How much does the DeepSeek vision model cost per image?

    A single image is capped at 384 tokens, costing $0.00008448 off-peak and $0.00016896 at peak. That works out to roughly $0.09 to $0.17 per 1,000 images.

    What are DeepSeek’s peak and off-peak hours?

    Peak hours run 01:00–04:00 and 06:00–10:00 UTC, when input and output rates double to $0.44 and $1.32 per million tokens. Everything outside those windows bills at $0.22 and $0.66.

    Are the weights open?

    No. V4-Flash-Vision-Exp is available only through DeepSeek’s paid developer platform. Reports suggest a free version may follow, but nothing has been confirmed.

    Does it hurt text performance?

    No. DeepSeek’s changelog states the model stays “on par with the official DeepSeek-V4-Flash” on agent reasoning, world knowledge, and other text-based tasks. The vision capability is additive.

    Should I migrate my production vision pipeline?

    Only if you process more than roughly 100,000 images per month and can tolerate an experimental endpoint. Below that volume, the absolute savings do not justify the migration and stability risk.

    The bottom line

    Move your high-volume image workloads to DeepSeek V4-Flash-Vision-Exp and keep your judgment-heavy visual work on Claude Opus 4.8. That is the split the benchmarks actually support, and it is where the money is.

    The specific trigger: if you process more than 100,000 images a month and a wrong answer costs you less than a human review, switch, and schedule the batch off-peak to halve the bill again. Below that volume, or where a visual error creates a compliance event, the $4-per-1,000 Anthropic bill is cheap insurance.

    The larger read is about pricing power, not vision. Anthropic charges $25 per million output tokens on Opus 4.8. DeepSeek charges $0.66 off-peak and just traded benchmark wins with it. The same compression already hit text and coding models, and multimodal was the last premium tier standing.

    Sources

  • Nvidia AVO Hits 100% on ARC-AGI-3. The Model Alone Scored 30.2%.

    Nvidia AVO — Agentic Variation Operators — scored 100.00 RHAE on the ARC-AGI-3 public set on August 21, 2026, clearing all 183 levels across 25 environments in 6,624 actions, roughly 12% fewer than the VISTA baseline. The same base model, Claude Opus 5, scores 30.2% on its own. The harness did the work, and that changes where agent money goes.

    What is Nvidia AVO?

    Nvidia AVO stands for Agentic Variation Operators. It is not a model. It is a general-purpose coding-agent system that wraps an existing frontier model in a loop — inspect, plan, implement, evaluate — plus persistent memory and a supervisor that intervenes when progress stalls. Nvidia published the results on August 21, 2026.

    The base model inside the winning run was Anthropic’s Claude Opus 5. Nvidia also ran limited experiments with GPT-5.6 Sol on a subset of games, and labeled those findings preliminary.

    That detail is the whole story. Nvidia did not train a better reasoner. It built better scaffolding around someone else’s reasoner.

    How the AVO loop works

    AVO runs a four-step cycle: inspect the current context, plan a change, implement it, then evaluate the result against the environment.

    Two additions separate it from a standard agent loop, according to Nvidia’s technical blog:

    • Persistent memory that carries forward prior implementations, evaluation results and reasoning across the whole run, not just the current context window.
    • A supervision mechanism that watches the trajectory and redirects the agent when it detects the run has stopped making progress.
    • Variation operators that generate structured alternatives rather than retrying the same failed approach.

    Why the supervisor is the expensive part

    Long-horizon agent failure is rarely a single wrong answer. It is a slow drift — the agent loops on a dead approach and burns tokens without noticing.

    A supervisor that detects stagnation is cheap to describe and hard to build. It is also the component least likely to transfer cleanly to another benchmark.

    What is ARC-AGI-3 and why does a 100% score matter?

    ARC-AGI-3 is ARC Prize’s interactive reasoning benchmark: 25 pixel-art puzzle environments containing 183 public levels. Agents get no instructions, no rules and no goal labels. They must infer the mechanics purely by playing. When the benchmark launched, humans cleared 100% of environments and the best AI managed 0.37%.

    That 0.37% figure is why this result registered. ARC-AGI-3 was designed as the benchmark models could not touch.

    How RHAE scoring works

    The metric is RHAE — Relative Human Action Efficiency. It combines task completion with how many actions the agent needed per level, measured against initial human performance, then aggregates across every level and environment.

    So a 100.00 does not just mean “finished everything.” It means finishing everything at roughly human action efficiency. ARC Prize published its human performance dataset specifically so this number would have a floor to sit on.

    How much did the harness add versus the raw model?

    The gap is 30.2% to 100.00 — the same model class, wrapped differently. ARC Prize reported Claude Opus 5 at 30.2% on ARC-AGI-3 in July 2026, which it called a genuine reasoning leap at the time. Nvidia’s harness took that model to a clean sweep of the public set.

    SystemARC-AGI-3 resultActions usedReported by
    Best AI at benchmark launch0.37%ARC Prize
    Claude Opus 5 (bare model)30.2%ARC Prize, July 2026
    VISTA baseline agentCleared same level sets7,542Nvidia
    Nvidia AVO (Claude Opus 5 inside)100.00 RHAE, all 183 levels6,624Nvidia, Aug 21 2026

    For context on the base model’s ceiling elsewhere: Claude Opus 5 at maximum reasoning effort scores 97.5% on ARC-AGI-1 and 90.4% on ARC-AGI-2, per The New Stack. ARC-AGI-3 was the one that stayed hard.

    Nvidia’s own framing, from the blog post: “The model matters, but the model is not the entire agent.”

    Is the 100% score real, or is it benchmark theater?

    It is real, and it is narrower than the headline suggests. The score covers the ARC-AGI-3 public set only — not the semi-private or private competition sets that ARC Prize maintains precisely to catch overfitting. Nvidia says so in its own post.

    The public set is not the hidden exam

    Benchmark designers split datasets for a reason. A public set is a practice test with the answers eventually leaking into the ecosystem around it.

    One commenter on Nvidia’s announcement put it bluntly, as flagged in explainX’s write-up: “I would not file this as solved AGI… if you post 100 like it is the hidden exam.” Until AVO posts a semi-private number, that objection stands.

    These are not controlled ablations

    Nvidia explicitly labels its comparisons as not controlled ablations. That matters more than it sounds.

    The AVO-versus-VISTA action count — 6,624 against 7,542 — varies agent backends, observation formats, memory systems and reasoning settings all at once. The 100.00-versus-30.2% comparison swaps the entire system architecture and the reasoning-effort setting simultaneously.

    Neither number isolates how much the harness itself contributed. The honest reading is “a well-built harness closed a very large gap,” not “the harness is worth exactly 70 points.”

    Nvidia also disclosed no compute cost, no token usage and no wall-clock runtime for the 6,624 actions. For anyone pricing an agent product, that is the number that actually matters — and it is missing.

    Has AVO done anything useful outside a puzzle benchmark?

    Yes, and this is the part investors should read twice. Nvidia ran AVO continuously for seven days on GPU-kernel optimization, exploring more than 500 optimization directions. The system produced kernels that beat FlashAttention-4 by up to 10.5% on NVIDIA DGX B200 hardware.

    FlashAttention is not a soft target. It is hand-tuned infrastructure that the entire industry’s inference economics rest on.

    A 10.5% kernel improvement compounds across every token served on that hardware. If it holds in production, it is worth more to Nvidia than the benchmark headline — and it lands in the same week the company has been buying capability outright elsewhere.

    Who wins and who loses financially?

    The winner is whoever owns the orchestration layer. If a 30% model becomes a 100% agent through harness design, then value is accruing above the weights, not inside them. That is bad news for anyone whose entire moat is a checkpoint.

    Nvidia is climbing the stack

    AVO did not appear in isolation. On the same day, Nvidia paid $6 billion to license Poolside’s model-development software and invested $1 billion more in the startup, according to PYMNTS.

    A chip company publishing frontier agent architecture and licensing a model factory in the same 24 hours is not a coincidence. It is a company that has watched its customers capture the margin its silicon creates — the same dynamic behind its recalculated OpenAI data center guarantee.

    Model labs keep pricing power, for now

    Note who supplied the brain: Anthropic. AVO’s best run needed Claude Opus 5, and the harness could not manufacture reasoning that was not already there — the 0.37% launch-day figure is proof that scaffolding alone does nothing on a weak model.

    So frontier labs still sell the scarce input. What they lose is the claim that the model is the product, which shows up quickly in cheaper models closing capability gaps.

    Agent startups just got a harder question

    Three practical implications for anyone building or funding an agent company:

    1. Harness design has not hit diminishing returns. A 30-to-100 jump says the scaffolding layer is still under-engineered — which is opportunity and commoditization risk in the same sentence.
    2. Your differentiator may be a blog post away from replication. Persistent memory plus a stagnation supervisor is a describable architecture, not a trade secret.
    3. Nvidia is now a potential competitor, not just a supplier. It has the hardware, the capital, and as of August 21, published frontier agent research.

    The cost question decides all three. Running a supervised, memory-heavy loop for 6,624 actions is not free, and the economics look very different depending on whether the underlying tokens cost $2 or $60 per million — the same math that drives coding-agent unit costs and inference vendor selection.

    Frequently asked questions about Nvidia AVO

    Is Nvidia AVO a new AI model?

    No. AVO is an agent system — a harness — that runs on top of existing frontier models. The reported 100.00 RHAE run used Claude Opus 5 as its base model.

    Did Nvidia AVO solve AGI?

    No. The score covers ARC-AGI-3’s public set of 183 levels across 25 environments. ARC Prize also maintains semi-private and private sets, and AVO has not posted a result on those.

    What does RHAE mean?

    Relative Human Action Efficiency. It scores both whether an agent completes a level and how many actions it needed relative to initial human performance, aggregated across the benchmark.

    Can developers use AVO today?

    Nvidia’s August 21 post describes the architecture and results. It does not announce a code or weights release, so treat AVO as published research rather than a shippable dependency.

    How much does an AVO run cost?

    Nvidia did not disclose compute cost, token usage or wall-clock time for the benchmark run. Without those figures, the result cannot be compared on a cost-per-task basis against cheaper agent harnesses.

    What was the FlashAttention-4 result?

    Running for seven days across 500-plus optimization directions, AVO produced GPU kernels that outperformed FlashAttention-4 by up to 10.5% on NVIDIA DGX B200 hardware.

    Does this make Claude Opus 5 look better or worse?

    Both. The model was capable enough to be driven to 100.00 RHAE, and weak enough on its own to score 30.2%. The delta belongs to the harness, not the checkpoint.

    The bottom line

    Nvidia AVO is the most important agent result of the month, and the headline number is the least interesting part of it.

    A 100.00 on a public set with no controlled ablations and no disclosed cost is a demonstration, not a benchmark victory. Anyone treating it as “ARC-AGI-3 is solved” is reading a press release as a result.

    What survives scrutiny is the gap: 30.2% to 100.00, same model, different scaffolding. That gap is the clearest evidence yet that in 2026 the agent layer, not the model layer, is where the remaining engineering leverage sits.

    And the FlashAttention-4 kernels are the tell. Nvidia did not build AVO to win a puzzle leaderboard. It built AVO to make its own hardware faster — and, at a moment when record sums are being raised to finance AI chip capacity, to stop being only the company that sells the machines.

    Sources