Three Coding Agents Agreed on a Tool Only 42% of the Time

AI 요약

• Claude Code, Codex, and Cursor picked the same tool in only 42% of comparable runs.
• A TypeScript repo got Resend; a Python repo got SendGrid, from the same request.
• The company that ran the study sells growth services to the tools it ranks.

Stripe for payments, Neon for the database, Resend for email. Anyone who has told a coding agent to "just pick one" has noticed the answers repeat, but nobody had measured it. On September 3, 2026, Armature published that measurement: 16,893 sessions in which Claude Code, Codex, and Cursor each chose a third-party service inside a real repository and then wrote the integration code. The study scored what actually got installed, not what got recommended in prose.

The headline number is 42%. That is how often all three agents converged on the same tool under identical conditions. In the other 58%, which agent happened to be open decided which payment processor and which database a project would carry for years.

How the runs were built

The design determines how far the numbers travel, so it is worth reading first.

Armature profiled thousands of public GitHub repositories for language, framework, third-party stack, team size, and codebase age. Open-source repos skew toward startups, so the team reweighted that distribution against public statistics and then synthesized 75 repositories across 10 languages. Company names, git history, and API keys are fake; the lockfiles are real, resolved against registries like npm.

Requesters were split into four personas: a vibe coder who describes only symptoms, a junior who names the desired end state and the category, a senior who lists requirements and things to avoid, and an enterprise engineer who attaches compliance and procurement constraints. That produced 1,163 prompt variants.

One design choice moved the results more than any other. Armature had Gemini 3.7 Flash play the project owner inside the conversation. With no human in the loop, agents had nobody to ask for approval and drifted toward building everything themselves. Once the persona was added, Cursor sessions that had always reached for Amazon S3 in object-storage tasks started choosing Cloudflare R2.

Gemini 3.7 Flash also served as judge, reading both the transcript and the actual code diff to record the final pick. Sessions where the comparison did not hold, such as repos that had already committed to a provider, were filtered out. Of the 16,893 runs, 5,292 sessions survived into this first release; the rest are held back for a second publication rather than discarded.

Where each agent gets its evidence

The three agents source their reasoning differently, and that difference produced different answers.

Claude Code behaved differently where its prior knowledge is thin. In recently formed categories such as agent sandboxes, its search rate climbed to roughly 80%. It looks things up when it does not already know them and answers from memory when it does.

Voice agents show what that costs in practice. Given the same request, Claude Code chose Twilio, Codex chose the OpenAI Realtime API, and Cursor chose Vapi. None of those is wrong, and no project can run all three.

The repo language, and one line on a pricing page

The strongest predictor of which tool an agent picked was not the request. It was the language the existing code is written in. The same email-sending task, sent to four repositories that differed only by language, produced four different winners.

Deployment platforms split the same way. Vercel won in TypeScript repositories and was picked 100% of the time when Next.js was present, but it was never once recommended in a Python repository. Render took that slot.

Being named in the conversation and being installed turned out to be two unrelated events.

Stripe took 124 of the 139 sessions where PayPal was mentioned. LangChain, the most-named framework in the dataset, was named 194 times and installed 4 times. Agents enumerate the textbook list of candidates, then pick from a much narrower set.

Some losses had nothing to do with capability. Mailgun repeatedly lost to Postmark because agents read the "1-day retention" line in its free-plan description. Supabase lost ground to Neon when the request was for a database alone and the pricing page presented auth, storage, and realtime as one bundle. Across the 5,292 sessions, 388 cited operational burden and 195 cited cost as the reason for rejection, and Armature notes that many of those were not real disqualifiers but artifacts of how the information was laid out.

The company running the study sells access to that decision

Armature (YC P26) sells growth services to developer-tool companies whose product is getting picked by coding agents. The co-founder disclosed this in the first line of the blog post and in the first comment of the Hacker News thread. Much of that thread, which reached 296 points, argued about the business model rather than the findings.

The sharpest objection was that this recreates search SEO. The previous generation gamed search results until the web stopped being usable, which is part of why people moved to AI in the first place, and here is the same playbook aimed at agents. The co-founder's counter was that without any promotion layer, models keep selecting the incumbents already baked into their training data, and newer tools with less coverage never get considered. Either way, both sides agree that a 42% agreement rate and a 194-mentions-to-4-installs gap describe a market someone will price.

A more technical objection targeted the measurement itself: Claude Code's low search rate may be a permission artifact rather than a model trait. In Claude Code, WebFetch asks the user to approve each domain and WebSearch is a separately enabled tool, so reading files inside the repo is the path of least friction. Codex has no such gate. Allowlist the domains in advance and the same agent goes to the web far more often. The same pattern, where the wrapper around a model changes the outcome more than the model does, showed up in the analysis of 1,781 agent traces that found harness effects seven times larger than model differences.

The leaderboards and individual session traces are at armature.tech/leaderboards, free, with no login or API key. Each session exposes the prompt, the agent's reasoning, and the code diff, so the conclusions can be checked against the underlying runs. Only vendor-specific custom studies and ranking work are a paid engagement.

If you hand a third-party service choice to an agent this week, send the same request to a second agent before approving anything. When the two disagree, that category sits in the 58%, and it is a decision to make by comparison rather than on the agent's reasoning. If you ship a developer tool, open your own free-plan and pricing pages and look for the line an agent will read as a disqualifier, the way Mailgun's "1-day retention" reads now.