Condition under which link failure nearly disappears. On 435 ExpertQA questions (a 20% sample), Claude Sonnet 4.5, Gemini 2.5 Pro and GPT-5.1 answered with cited URLs while a URL checking tool (urlhealth: HTTP check plus Wayback Machine lookup) was available as a callable tool, and could verify and replace their own citations over several rounds. Classified with the same tool before and after on the same questions, the non-resolving rate fell for GPT-5.1 from 16.0% to 0.6% (26x), for Gemini from 6.1% to 0.1% (79x) and for Claude from 4.9% to 0.8% (6.4x), all p < 10^-35 in a two-proportion z-test. Table 3 prints final shares of LIVE 78.0 to 88.9%, DEAD 0.1 to 0.6%, LIKELY HALLUCINATED 0.4 to 1.8% and UNKNOWN 10.3 to 20.2% of 4,203 to 7,985 URLs per model. A smaller model (gpt-5-nano) in the same loop ended at a 7.5% not-live rate, with 48 hallucinated URLs persisting across up to 14 correction rounds. The measurement concerns whether links resolve; whether the replacement links support the claims was not measured. Automated measurement, no human or LLM rater.

Authors are at the University of Pennsylvania (DARPA SciFy funding); arXiv preprint under review, not peer reviewed. They are not the vendor of any evaluated model but are the makers of the tool whose effect is measured here, so independence is recorded as positioned for this result. The prose rates after mitigation (0.6, 0.1, 0.8%) do not equal DEAD plus LIKELY HALLUCINATED in Table 3 (2.4% for GPT-5.1, 0.7% for Gemini, 0.5% for Claude); the text does not explain the difference. UNKNOWN responses (10 to 20%) sit outside the non-resolving count; a headless-browser audit of 600 sampled UNKNOWN URLs found 11.0% [8.5, 13.7] genuinely dead. The pre-tool rate for GPT-5.1 on this subset under urlhealth classification (16.0%) is not the main-pipeline rate on the full set (8.47%); classification rules and question sets differ. Gemini ran in two phases because its API does not allow search grounding and custom tools together. The Claude run stopped at 658 questions on an API usage limit, hence the common subset of 435. Tool and data are announced under MIT license, so the counter-check (rerun) is open.

A recomputation from the released data in which DEAD plus LIKELY HALLUCINATED plus the dead share of UNKNOWN exceeds 1% for any of the three models (Table 3 already sums to 2.4% for GPT-5.1), or an independent browser-based liveness check of the final citations finds more than 1% not resolving. As a condition for the stock's question it falls if a support audit shows that tool-verified links resolve but support their sentences no better than before. Query to run: browser liveness check of the final URL sets, and per-citation support rating before versus after the urlhealth loop.

Broken and invented links are an inherent defect of language models. Too coarse: given a link checker as a tool and the competence to use it, three current models brought confirmed-broken citations down to below 1 percent by the paper's count and at most 2.4 percent by its table, so link failure is a tooling condition; support is a separate quantity that the check does not touch.

https://arxiv.org/pdf/2604.03173v1 p. 7, Section 5.1 Results; p. 8, Table 3 | 2026-04-03 · arXiv 2604.03173 · Rao, Wong, Callison-Burch, Detecting and Correcting Reference Hallucinations in Commercial LLMs and Deep Research Agents