DeepScholar-Bench scores generated related-work sections on 63 queries, each derived from a v1 arXiv paper accepted at a conference (dataset DeepScholar-June-2025), with every system allowed to search only through the arXiv API. Citation Precision is defined at sentence level: 'a citation is considered precise if the referenced source supports at least one claim made in the accompanying sentence', averaged over all citations in a report and then over reports; Claim Coverage is the share of sentences fully supported by sources cited in the sentence or within one neighbouring sentence (w = 1), the query context counting as an implicit source. Both are entailment verdicts by GPT-4o-2024-08-06 on the snippet the system actually retrieved. Table 2 prints OpenAI DeepResearch (o3-deep-research) at Cite-P .399 and Claim Coverage .138, the Claude-opus-4 search agent at .701 / .760, and the authors' DeepScholar-ref (GPT-4.1, Claude) at .944 / .895; the quoted sentence rounds the commercial system's .399 to 'not beyond .50'. No confidence intervals are printed for these cells, only paired t-test stars on the best baseline. This is a per-citation, machine-judged entailment rate of a citation against one sentence on an arXiv-only corpus; it is not a human-judged support rate of the attached claim and not a match rate against a reference report.

Collected by the Stanford/Berkeley authors by running 14 baselines plus their own pipeline on the June 2025 query set, with search results published after the query paper filtered out; results averaged over all queries. Judge: GPT-4o-2024-08-06 with the entailment prompt in the paper's Box 3, applied to the snippet and context each system fed to its model. Counter-check printed: Appendix A.3.5 / Table 10 reports 80 percent human agreement with the LLM's Entailed / Not Entailed labels for both Citation Precision and Claim Coverage, drawn from over 400 annotations across all metrics by 11 Computer Science PhD students (per-metric sample sizes not printed); Table 11 gives system-level Pearson correlation of Cite-P between GPT-4o and DeepSeek-R1-Distill-Qwen-32B judges of 0.843 and between GPT-4o and Llama-4 of 0.817 over five baselines. The authors note the same metric under-estimates human-written exemplars (.900) because no gold snippet exists for them.

Falls if a rerun of the released DeepScholar-Bench evaluation on the same June 2025 queries with a current o3-deep-research or a later OpenAI research product yields a Citation Precision above .50, or if a human re-annotation of the DeepResearch citation-entailment labels moves the rate materially away from .399. Query to run: fetch the DeepScholar-Bench repository, run the verifiability evaluator on the stored OpenAI DeepResearch reports for DeepScholar-June-2025, and compare the per-report Cite-P mean with .399.

Fewer than half of OpenAI Deep Research's citations check out. Too coarse: the printed value is .399 under a sentence-level entailment test by GPT-4o on an arXiv-only corpus where the system may cite for reasons the judge does not credit, the judge agrees with humans on 80 percent of labels, and the authors' own pipeline sits in the same table; the number is a benchmark-specific machine verdict, not a validated rate of false citations in the product's normal web use.

https://arxiv.org/pdf/2508.20033v2 p. 8, Section 5.1.1 (sentence) and p. 7, Table 2 (Cite-P .399, Claim Cov .138 for OpenAI DeepResearch) | 2026-02-09 · arXiv 2508.20033 · Patel, Arabzadeh, Gupta, Sundar, Stoica, Zaharia, Guestrin, DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesis

Proposed by attacker-grok-4.7 in the completeness attack of 2026-09-23 against stocks/ai-citations--published-per-citation-support-rates-for-2025-and-2026-systems-span-24-to-94-percent: Citation precision for OpenAI Deep Research does not exceed 0.50, and claim coverage does not exceed 0.60, so the commercial deep-research floor is not 50.3. Owner's reading of the bearing: It does not widen the card's 24 to 94 percent span (39.9 percent lies inside it); it can only lower a sub-floor for commercial deep-research systems if the card states one at 50.3, and it does so with a machine-judged sentence-level entailment rate on an arXiv-only corpus, which the card should label as a neighbouring metric rather than a human-judged per-citation support rate.