Five LLMs split on 67% of fact-checks, and AI search absorbs the cost

Lenz Research published a frontier LLM fact-check disagreement study on May 21, 2026. The study asks a question AI search products cannot avoid: not whether a model can write a convincing answer, but whether frontier models reach the same verdict when they see the same real-world claim. The headline is blunt. Across 1,000 actual user-submitted fact-check requests, five models failed to reach the same conclusion on 672 claims, or 67% of the corpus.

The evaluated models were GPT-5.4, Claude Opus 4.7, Gemini 3 Pro, Gemini 3 Pro + Search, and Sonar Pro. Each model received the same claim and date anchor, then had to choose exactly one of four labels: True, Mostly True, Misleading, or False. There was no option to explain, hedge, or abstain. That setup strips away long-form persuasion and citation style so the study can isolate the final label a product might show to a user.

The first line of the original table compresses the whole result. All five models agreed on 328 claims, or 33%. One model dissented from the majority on 224 claims, or 22%. Two models dissented on 316 claims, or 32%. Another 132 claims, or 13%, produced a 2-2-1 or 2-1-1-1 split with no strict majority at all. Lenz reported ordinal Krippendorff's alpha of 0.639. The panel was not random, but it was not consistent enough to treat the models as interchangeable judges.

The more uncomfortable figure is 34%. Lenz found 343 claims where the two most distant models were separated by at least two label buckets. That does not mean every case was a clean True versus False fight, but it does mean more than one-third of the claims had disagreement too large to dismiss as wording preference or minor confidence calibration. A product can display "sources checked," yet still produce a different final verdict depending on which model interprets the evidence.

The dataset source is part of the story. Lenz used 1,000 recent claims submitted by real users to its fact-checking platform. The research note says the claims were not older than February 15, 2026, and excluded private claims, staff or API submissions, pending or hidden items, and near-duplicates. That matters because public benchmark data can be contaminated by training exposure or canonical answer keys. This corpus is closer to what a live AI search or verification product receives in production: messy, recent, and not pre-labeled for the model.

Lenz also avoided a common shortcut. The study did not treat the model majority as ground truth. The authors explicitly note that the majority verdict can be wrong and that a minority model can be right. This is not a leaderboard claiming GPT-5.4 beats Claude, or that Gemini with search is more factual than Sonar Pro. The measured object is structural: when the same claim enters a frontier panel, how often does the panel fail to land in the same bucket?

The downloadable CSV shows that disagreement is partly about label habits. Gemini 3 Pro chose True for 539 claims and False for 401, while using Mostly True and Misleading only 30 times each. Gemini 3 Pro + Search also leaned toward the edges, with 520 True labels and 351 False labels. Claude Opus 4.7 used Mostly True 258 times and Misleading 193 times, while Sonar Pro also used the middle buckets more often.

That does not prove one model is more careful than another. Lenz forced a one-label response, and retrieval-enabled models were not constrained to fetch the same material. For product teams, the signal is still practical. A model that uses Misleading broadly will drive a different UI, escalation policy, and user action than a model that cuts directly to False. The same claim can become a warning badge, a blocked answer, a neutral caveat, or a human-review task depending on which label habit the system inherits.

AI search products already operate in this territory. A search results page with an answer box, a support agent that judges customer requests, and an internal RAG tool summarizing incident logs all perform claim verification. Questions such as "Does this policy apply today?", "Is this library version vulnerable?", and "Does this contract clause apply to this customer segment?" are not simple text generation tasks. They are verdict tasks with dates, evidence, and cost attached.

For fact-checking and compliance domains, citations are necessary but not sufficient. Two models can inspect the same URL and still disagree because they apply different time anchors, interpret Mostly True differently, penalize incomplete evidence differently, or weigh source authority differently. Lenz included an "as of YYYY-MM-DD" anchor for each claim because political, financial, legal, and technical statements can change truth value over time. A claim without a date can be true in one release cycle and false in the next.

The appendix makes the problem concrete. Example claims include World Bank portfolio figures for Nigeria, scientific statements, political claims involving India or Kenya, and technical responsibility questions about organizations. Some cases produced a maximum bucket distance of three, with one model choosing True, another choosing False, and another landing on Misleading. To a user, that looks like one question. Inside the system, it is several incompatible judgment rules colliding.

The result should not be reduced to "LLM fact-checking does not work." Lenz did not include human-labeled ground truth in this v1.0 snapshot. The FAQ says a follow-up study will compare human labels, the frontier panel, and Lenz verdicts. The current release does not establish the model error rate. If five models split on a claim, the only lower-bound statement is that at least one verdict is label-inconsistent under the four-bucket rubric.

That limitation leaves builders with a more realistic problem. Production AI search and verification products usually answer before ground truth exists. If the answer were already known, the product would not be needed. The system therefore has to record and show more than "the model is confident." It needs agreement level, evidence overlap, time anchor fit, retrieval trace, and the reason a middle bucket was chosen.

The Hacker News response focused on the same tension. The post appeared on May 28, 2026, and the Algolia API snapshot cited in the research note showed 489 points and 341 comments the next day. The discussion split across the risk of single-model fact-checking, the fact that search-connected models still did not guarantee the same conclusion, and the counterargument that fact-check labels are difficult for humans too. Lenz makes that caveat directly by noting that human-annotated fact-check corpora such as AVeriTeC do not produce perfect annotator agreement either.

Three design changes are available without waiting for a follow-up study. First, fact-checking pipelines should store dissent metadata beside the final verdict. Even a single-model setup needs prompt version, model version, retrieval trace, timestamp, and rubric version if the team wants reproducible audits. Second, multi-model systems should not expose only the majority label. Vote distribution, maximum bucket distance, and no-majority state are separate product facts. A 13% no-majority rate is large enough to require a policy for automatic publishing, automatic blocking, and human escalation.

Third, user-facing language should shrink the "verified" badge. Legal, health, finance, and politics claims have different false-positive and false-negative costs than most technical claims. Lenz's corpus distribution included General at 179 claims, Health at 171, Politics at 168, Science at 151, History at 131, Tech at 77, Finance at 75, and Legal at 48. The same disagreement rate should not lead to the same UI in every domain. A health claim may need an escalation state where a tech claim can show a caution label and linked evidence.

The cost structure is the reason AI search teams cannot ignore this result. It is tempting to assume that one more model call will create consensus. Lenz shows the opposite case: five model calls still left 67% of claims non-unanimous. Multi-model verification adds latency and API cost, then forces the product to decide what split verdicts mean. Without that policy, the bill rises while the user still sees one smooth sentence.

RAG evaluation also needs different metrics. Many internal evals stop at answer correctness, citation recall, and groundedness score. Claim verification workflows need disagreement rate, maximum bucket distance, no-majority rate, and domain-specific abstain or escalation rates. Lenz intentionally removed an abstain option and forced four labels. Real products need the missing fifth path: recognizing when the system should not produce a final verdict.

The conservative reading is not "the models are broken." It is "the verdict interface is underbuilt." Once Lenz adds human labels, the accuracy discussion can become sharper. The existing snapshot is already enough to make single-answer fact-check copy risky. If 672 of 1,000 real user claims do not receive the same label from a frontier panel, a product should not hide that disagreement behind a confident sentence.

The same lesson reaches AI developer tools. Coding agents also make claims: whether a dependency is vulnerable, whether a license allows a use case, whether release notes changed a behavior, or whether a cloud incident explains an outage. A log line that says "searched and verified" is too thin. The useful trace includes the claim date, sources, verdict rubric, dissent handling, and the threshold for human review. Lenz v1.0's 67% figure is not a declaration that AI search has failed. It is a line item in the real cost of building verification products.