The Same Model Scores 30 Points Apart Across OpenRouter Hosts, and Default Routing Lands at 69.9%

Call an open-weight model through OpenRouter and it is easy to assume that sending the same model name gets you the same answer. On September 7, Mo Moustafa, who builds Olly, an AI assistant that runs inside iMessage, published "So you want to use OpenRouter?", a set of measurements showing that assumption is wrong. The post reached the top of the Hacker News front page on September 11 with 642 points, and an OpenRouter co-founder answered in the comments.

OpenRouter is a brokering API: you send a request naming one model, and it forwards that request to one of the several companies (hosts, below) that have loaded the model onto their servers. We covered it before, when it raised $113 million on model routing. DeepSeek V4 Flash 0731 alone has more than 30 hosts. They all run the same weights, and the results are not the same.

Every figure here is for DeepSeek V4 Flash 0731, read off the OpenRouter model page on September 11.

Identical weights, a 30-point tool-calling gap

Moustafa compared the four models Olly uses (DeepSeek V4 Flash 0731, GLM-5.3 Flash, Qwen 3.5 122B, MiniMax M3) across roughly 20 hosts per model. Olly has passed 18 million cumulative messages, about a third of which go through OpenRouter.

The widest gap showed up in TAU-Bench, which measures whether an agent calls its tools correctly. DeepSeek V4 Flash 0731 scored 81.3% on DeepSeek's own servers and 58.4% on DigitalOcean. In July, Fireworks bottomed out at 46%, a 35-point spread. GPQA, which tests knowledge questions, split 90.2% for DeepSeek against 75.3% for DigitalOcean, a 15-point gap.

TAU is the score that matters for agents, and a 20-point difference is not noise. (Mo Moustafa)

This is not one person's measurement. OpenRouter publishes per-host GPQA Diamond and TAU-Bench Airline scores on the Performance tab of each model page, as 32-day moving averages. That is the same window OpenRouter uses when it picks hosts on quality.

The table lists 33 hosts plus one auto-routing row. Lined up against price and traffic share:

Top-ranked Alibaba Cloud Int. (76.8%) and bottom-ranked DigitalOcean (46.4%) sit 30.4 points apart. The absolute numbers differ from Moustafa's (81.3% against 58.4%) because OpenRouter's table is a 32-day average and DeepSeek's own endpoint was missing from the September 11 listing. DigitalOcean lands at the bottom either way.

60% of traffic goes to the two cheapest hosts

Per OpenRouter's documentation, a request that specifies neither sort nor order prefers the cheaper host. Among hosts with no outage in the last 30 seconds, it splits traffic in inverse proportion to the square of price. Under that formula, a host charging twice as much gets picked a quarter as often.

Output prices for this one model run from $0.16 at OpenInference to $1.32 at Cloudflare and Baidu Qianfan, an 8.25x spread. On September 11, Relace took 36.9% of the day's tokens and OpenInference 23.5%, so the two of them carried 60.4%. Both sit in the cheapest band, and their TAU-Bench scores are 71.7% and 70.8%, five to six points below the three hosts in the 76% range. OpenInference's GPQA is 70.5%, about 19 points below Alibaba Cloud Int. at 89.4%.

Paying more does not buy accuracy either. Cloudflare charges $1.32 for output and scores 74.6%, while Alibaba Cloud Int. tops the table at 76.8% for $0.528, roughly 40% of that price. The price list is not a quality signal.

OpenRouter knows about this. Since March 12, requests that include tools get Auto Exacto by default, which reorders hosts using throughput, tool-call success rate, and OpenRouter's own benchmark scores. At launch, OpenRouter said GLM-5's tool-call error rate fell from 8% to about 1%.

Its coverage has limits.

• Requests without tools still follow the price weighting. To get quality-first routing there, you append :exacto to the model name.
• Passing sort: "price" or appending :floor turns Auto Exacto off.
• The auto-routing row near the bottom of the table sits at 69.9% on TAU-Bench, 6.9 points below the top host. The page does not say which settings that row was measured under.

The failures that never reach a scoreboard

Many of the failures Moustafa hit were buried inside HTTP 200 responses. With no error code, retry logic never fires.

• Image input: DeepInfra's Qwen3.5 read the letter K as R and called red blue. Venice and Together's MiniMax advertised image support and answered that there was no image.
• Reasoning effort: DigitalOcean, GMICloud, Mancer, and Venice produced the same amount of reasoning no matter what reasoning.effort was set to. He pinned the host and sent the same prompt three times per effort level.
• Empty responses: in July, StreamLake took 20% of DeepSeek traffic and produced 92% of the empty responses.
• Tool-call parsing: instead of structured tool calls, raw markup such as <use_skills><parameters> came back as body text.
• IP-based limits: Venice and Novita worked fine from a personal Mac, but nearly every request from a production server came back 429.

The usual workaround did not hold either. Filtering to fp8 hosts with quantizations did not improve quality, because fp4 and fp8 hosts were scattered throughout the rankings. Moustafa wrote that precision is a poor proxy for quality and that forcing the filter only shrinks the candidate pool.

Pinning hosts failed too. He named Cloudflare, Baidu, and Alibaba in order and set allow_fallbacks: false to block alternate routes. Baidu then rate-limited him, Cloudflare pulled the model, and Alibaba added limits of its own, taking the service down.

OpenRouter's answer, and what it left blank

Chris Clark, OpenRouter co-founder and COO (HN username numlocked), described how routing works in the comments.

• Benchmarks run continuously against live endpoints, and a host more than one standard deviation from the median gets pulled out of the default routing pool.
• Hosts that frequently mis-parse tool calls are routed around in real time.
• Quantization filters are a "legacy concept" and he advised against using them.

He also acknowledged gaps. When a response's finish reason is error, OpenRouter absorbs the cost; when the content is empty but the response finishes with stop, the user pays. The head of provider relations (pingtoven) confirmed that DeepInfra passed the image test when it was listed in July but does not pass it now, and said image and reasoning-effort tests would move to continuous runs.

Clark said OpenRouter is "introducing QoS tiers for production apps." He gave no date and no price, and noted that the support team is "very understaffed."

What you can set today

Before any QoS tier ships, the provider object in the request body already lets you pick hosts yourself. This part applies to developers.

If tool calling matters to your agent, start by naming the top TAU-Bench hosts in order.

{
  "model": "deepseek/deepseek-v4-flash-0731",
  "provider": {
    "order": ["alibaba", "nextbit", "coreweave"],
    "require_parameters": true,
    "allow_fallbacks": true
  }
}

order is the attempt sequence, and host identifiers are on the Providers tab of the model page. require_parameters: true restricts routing to hosts that support every parameter in the request. The documentation describes it as filtering on declared support, so it is no guarantee against a host that advertises a parameter and ignores it. Leave allow_fallbacks on: turning it off, as Moustafa did, leaves no alternate route when all three named hosts break at once.

If you are running a tool-calling agent on an open-weight model through OpenRouter, open the Performance tab on that model's page, read the AutoExacto Benchmarks table, and put the top three TAU-Bench hosts into order. Then log the share of empty-content 200 responses per host on your production servers for a week. Once you have a number to compare against the default-routing period, you can judge whether the tokens saved on a cheap host are worth less than the conversations it failed.