OpenAI’s Jalapeño Chip Turns Watts Into Faster Agent Steps

OpenAI said its Jalapeño inference chip did 1.5 to 1.9 times more AI work per watt than Nvidia’s GB200 and GB300 systems on a public test last week. End-to-end latency was 1.7 to 3.6 times lower across the same three models.

Richard Ho, who leads OpenAI’s hardware program, told reporters the chip offers the “best of both worlds,” higher throughput and lower latency, because most serving systems “have to make a trade-off between the two.” The same briefing put only small volumes into OpenAI’s fleet by the end of 2026, with a larger ramp in 2027, and Ho said Nvidia remains a partner for training and inference.

Three Open Models Sat on the Public Scoreboard

OpenAI published the InferenceX results on August 25 at Hot Chips, with an appendix for each model at a nominal 8,000-token prompt and 1,000-token reply. The runs used single-token prediction. GPT-OSS 120B, OpenAI’s own open-weight model, was scored against a GB200 rack rated at 1,200 watts. DeepSeek R1 670B and Moonshot AI’s Kimi K2.5 1T, a trillion-parameter model, were scored against GB300 racks rated at 1,400 watts.

INFERENCEX, MATCHED TO PACKAGE POWER

The latency gap is widest on the two larger models, which is the opposite of a chip that only looks good on a house model. For highly interactive loads, OpenAI put the gain at 2.1 to 4.1 times, and the DeepSeek row is the top of that band: 1.43 ms versus 5.90 ms between tokens, or about 700 versus 169 tokens per second per user.

Some recaps led with a 104.3 times throughput jump on DeepSeek. That figure is real in the appendix, 12,258 versus 118 mixed tokens per kilowatt, but only at GB300’s old interactive setting of 169.41 tokens per second per user. It is a matched-point result, not the everyday serving ratio. The load-bearing range remains 1.5 to 1.9 times more work per watt at peak.

The Test Leaves Nvidia’s Next Chip off the Board

OpenAI ran the public InferenceX benchmark because it scores a full request, prefill through decode, and then divides useful work by each chip’s published power rating. Jalapeño is rated at 700 watts. OpenAI said measured draw stayed at or below 550 watts on these jobs, but the comparison still uses 700 watts, which is the conservative choice for Jalapeño if the Nvidia side is also taken at sticker power rather than a measured plug.

Ho said the posted numbers are single-token prediction, with no speculative decode. Multi-token prediction on rival stacks can add about a three to five times boost, which is why a raw read of the charts can overstate the gap. OpenAI also showed a stricter cut on DeepSeek, Jalapeño in single-token mode against GB300 in multi-token mode, and still claimed about 1.5 times more peak work per kilowatt and 2.2 times lower end-to-end latency. The lead shrinks. It does not flip.

THE SCOREBOARD’S FINE PRINT

• The rival generation: The published Pareto line is July 2026 GB200 and GB300, not Nvidia’s HBM4 Rubin racks that have started to ship.
• The power math: Results are divided by package ratings of 700 watts, 1,200 watts, and 1,400 watts, not by a shared measured wall-plug figure.
• The job shape: Each run is one turn at 8,000 input tokens and 1,000 output tokens, which is easier to tune than a long, multi-turn agent trace.
• The missing suite: Lab visitors confirmed the InferenceX jobs in person and have not posted AgentX numbers for Jalapeño, the long-context replay that now dominates production serving.

Nvidia is already publishing Vera Rubin NVL72 throughput claims on that agentic class of test, including large DeepSeek and Kimi workloads, which is the comparison Jalapeño still has to make in public. A first-generation ASIC that beats last year’s rack on a single-turn chart can still lose the year if Rubin’s software stack keeps climbing while OpenAI is qualifying boards.

A 700-Watt Package Keeps the KV Cache Local

Jalapeño is an inference ASIC, a chip cut for running a trained model rather than teaching a new one. Ho’s team designed it around a question the company now repeats in almost every hardware briefing: what would you build if the main job were serving modern language models, especially agents that take many steps in a row.

Language-model serving splits into phases that hate each other. Prefill, reading the prompt, is heavy on compute. Decode, emitting one token after another, is heavy on memory bandwidth. When those phases live on different boxes, the key-value cache that holds conversation state has to move, and cores sit idle while they wait. OpenAI’s answer is to keep that cache in place and change which mix of compute, memory, and network is awake inside one chip, with idle blocks gated instead of paid for as a second accelerator’s baseline draw.

THE JALAPEÑO PACKAGE

• Compute: 13.4 petaflops of MXFP4 matrix work in a 700-watt package.
• Memory: six HBM4 stacks, 216 GiB, at 15.4 TB/s of bandwidth.
• Scale-up: 128 chips in the local domain, 2,048 chips on a global fabric built with Broadcom Tomahawk 6, at 600 GB/s locally and 200 GB/s globally.
• Full domain: 27 exaflops and 432 TiB if the 2,048-chip system is filled.

Hot Chips slides put a bandwidth-only ceiling at 1,000 to 2,000 tokens per second per user without speculative decode, and 5,000 to 10,000 with it. OpenAI said the real system lands well below those rates, which is a quiet admission that wires, fences, and placement still eat the theoretical peak. The architecture bet is that a spatial layout, each core slice paired with an HBM slice plus a fast collective path for the common cross-core patterns, wastes less of that peak on waiting.

Nine Months to Tape-Out, Two Months to Port the Models

OpenAI says Jalapeño went from initial design to manufacturing tape-out in nine months, with its own models used to explore implementations, shorten verification loops, and pack more arithmetic onto the die. The results post is blunt about the programming target: local tensors, explicit communication, and predictable sync, so that both people and models can map work onto the chip.

That claim has a concrete follow-up. Using Codex with GPT-Astra, the hardware team brought GPT-OSS 120B, DeepSeek R1, and Kimi K2.5 to high performance in two months, even though those three were not in the original production plan. For selected GPT-OSS attention and mixture-of-experts blocks, AI-written kernels ran 1.5 to 1.8 times faster than the existing expert-written versions. Those figures apply to the selected blocks, not the full model.

The speed of the cycle is the part other chip teams will copy first. A nine-month ASIC used to be a punchline. If the next stepping can be compiled the same way, OpenAI does not need Jalapeño to replace Nvidia this winter. It needs the loop to stay shorter than Nvidia’s software treadmill.

Hock Tan Promised Gigawatt-Scale Racks This Year

OpenAI unveiled Jalapeño with Broadcom on June 24, with Celestica named for boards, racks, and system build. OpenAI designed the architecture. Broadcom did the silicon implementation and the Tomahawk networking. The June note already pointed at gigawatt-scale deployment with data-center partners across several generations, which is a supply-chain story more than a die shot.

This is just the beginning of a multi-generation roadmap. By co-developing our industry-leading silicon directly with OpenAI, we are enabling the deployment of gigawatt scale data centers with Microsoft and other partners beginning in 2026.

Hock Tan, President and CEO, Broadcom

Greg Brockman, OpenAI’s president and co-founder, tied the chip to a “long-term full-stack infrastructure strategy to make compute more abundant.” The stack he means is models, products, serving software, chips, memory, networking, and racks designed together. The company that collects a toll on that custom path is Broadcom, which already does similar work for other large inference ASICs. Nvidia still sells the general-purpose training and serving GPUs. Broadcom sells the special case.

TSMC is the manufacturing destination for the tape-out, which puts Jalapeño in the same wafer, HBM4, and advanced-packaging queue as everyone else who is trying to get out from under GPU lead times. Escaping Nvidia’s board does not escape the line at the foundry.

Small Volumes This Year, Nvidia Still in the Rack

OpenAI plans to start putting Jalapeño into its own compute infrastructure by the end of the year, then raise volume in 2027. It has not said how many chips that ramp is. Gen 2 is deep in development. Gen 3 is taking shape. The results post still says the company “will continue to widely deploy accelerators from NVIDIA and other partners for both training and inference workloads.”

THE JALAPEÑO CLOCK

1. June 24, 2026: OpenAI and Broadcom unveil Jalapeño, with Celestica on racks, and engineering samples already running GPT-5.3-Codex-Spark in the lab.
2. August 25, 2026: Ho’s team posts the three-model InferenceX appendix at Hot Chips and says small volumes arrive by year end.
3. Late 2026: OpenAI begins production qualification, software bring-up, and a limited fleet install.
4. 2027: Volume ramps while Gen 2 and Gen 3 move through the same partner chain.

Jensen Huang, Nvidia’s chief executive, answered a day after the slides. He said he does not “have to take anything personally,” that “lots of projects get started” and lots get canceled, and that Nvidia still has “the supply chain and the technology scale to be their largest supplier.” On the earnings call he drew a sharper line: many of these XPUs are inference-specific chips for one cloud or one service, while Nvidia sells a platform across the whole AI life cycle.

WHERE EXPERTS DISAGREE

• OpenAI’s view: One balanced chip can sit on the Pareto frontier for both cheap tokens and fast tokens, so the company can serve more demand from the same power budget.
• Nvidia’s view: A single-customer inference ASIC is a workload slice, not a replacement for a platform that trains, serves, and ships into every cloud.

Both can be true at once. Jalapeño does not have to empty Nvidia’s racks to change what OpenAI pays for the next training cluster. A credible in-house option is itself a price. Sam Altman, OpenAI’s chief executive, put none of that in the public note.

we made a chip and it is fast

— Sam Altman (@sama) August 25, 2026

Why Agent Loops Care About Those Milliseconds

OpenAI’s own write-up spends more time on agents than on chat replies. An agent that has to search, call a tool, read the result, and call the model again pays the latency bill on every hop, so a 1.65-second DeepSeek round trip against a 5.99-second one is not a brag about snappy autocomplete. It is a claim about how many steps fit inside a user’s patience and a data hall’s watt cap.

That is also why the missing AgentX numbers sting. A single 8,000/1,000 turn does not stress routers, prefix cache, or the messy hit rates of a real coding session. The chip still has to prove it can hold the cache local when the conversation is long and the tools keep coming back. Until those traces are public, the agent story is an architecture argument plus a single-turn chart.

Coding agents feel that wait first, which is why the same silicon already maps those watts into faster agent steps once a tool loop starts stacking calls. Ho’s line about “faster responses, more responsive agents, and more reliable access as the demand grows” only holds if the 2027 racks actually show up, and if the software that placed those open models in two months can do the same for the models OpenAI serves to paying users.

OpenAI is still qualifying production parts, maturing the stack, and checking more models. Nvidia still owns training and a large share of serving. Broadcom still owns the custom path. The scoreboard from August 25 is a first-generation ASIC that won a public test against last year’s racks, then booked a small install for later this year.