In December 2024 the BBC put 100 news questions to ChatGPT (Enterprise, GPT-4o), Copilot Pro, Gemini Standard and Perplexity Pro; 45 BBC journalists reviewed 362 responses. 62 responses included quotes from BBC articles; eight quotes were either altered from the original source or not present in the cited article, which the report expresses as an error rate of 13%. The eight occurred in responses from all assistants tested except ChatGPT. The report divides a count of quotes by a count of responses; the total number of quotes checked is not printed. Verification was a human comparison with the BBC's own articles.

Designed and carried out by the BBC's Responsible AI team. Responses to 100 news questions drawn from trending Google search topics were collected on 5 and 6 December 2024 with the prefix 'Use BBC News sources where possible'; the BBC lifted its crawler blocks for the duration. 45 BBC News journalists, assigned by area of expertise and in many cases the authors of the cited articles, reviewed 362 responses in randomised order with assistant names removed. The BBC is the publisher whose content is represented, states that publishers should have control over the use of their content and calls for regulation: a stake and a declared position, recorded as positioned. Counter-check: the cited BBC articles are public, so each quote can be compared by anyone, but the report publishes only example responses in its appendix, not the 62 quote-bearing responses or the eight cases in full. A small inter-rater agreement test using Krippendorff's Alpha showed moderate agreement; it was not specific to quotes.

Falls or narrows if the full set of BBC-attributed quotes in the 62 responses is compared with the cited BBC articles, including their revision history since BBC articles are updated after publication, and the altered-or-absent count differs from eight. Narrows if the rate is recomputed per quote rather than per response. Query: number of distinct BBC-attributed quotes in the 362 responses, and how many match the article version live on 5 and 6 December 2024.

When an assistant puts words in quotation marks and cites the article, the words are in the article. Too coarse: in this sample about one in eight quote-bearing responses carried a quote that was altered or not in the cited article.

https://www.bbc.co.uk/aboutthebbc/documents/bbc-research-into-ai-assistants.pdf p. 7, Accuracy | 2025-02-11 · BBC report · Elliott, Representation of BBC News content in AI Assistants