Cite Forward

Nine AI Tools Were Tested on Retracted Papers. The Research-Specific Ones Did Worst.

A hallucinated citation fails the moment you check it. Paste the DOI, get a 404, delete the reference. Annoying, but self-correcting.

A retracted citation is worse, because every check passes. The DOI resolves. The authors are real. The journal is real. The paper genuinely exists — it has simply been withdrawn from the scientific record, and nothing about the reference itself tells you so.

A study published in the Journal of Medical Internet Research tested whether AI tools catch this. The answer is that they mostly do not — and the tools built specifically for researchers performed worse than the general-purpose chatbots.

How the test worked

The researchers selected 15 retracted articles from the Retraction Watch database: the 10 most-cited retracted papers, plus the 5 most recently retracted as of May 2025.

Nine free AI tools were then asked five standardised questions about each paper — topic overview, article identification, summarisation, and an explicit question about retraction status. Every question was asked twice to measure whether a tool gave consistent answers. Two researchers independently scored each response.

The tools tested: ChatGPT 4, ChatGPT 5, Claude, Gemini, Perplexity, Microsoft Copilot, SciSpace, ScienceOS, and Consensus. OpenEvidence, a medical-literature-specific model, was analysed separately.

Note what the fourth question does. The tools were not only observed passively — they were directly asked whether the paper had been retracted. This is not a test of whether AI spontaneously volunteers retraction status. It is a test of whether it knows when you ask.

The results

No tool handled retracted articles reliably. None achieved full accuracy across the question set.

  • ChatGPT-5 performed best: 8 of 15 fully correct — 53.3%
  • SciSpace, ScienceOS, and Consensus produced zero fully correct results
  • Microsoft Copilot had the highest topic-overview accuracy at 87%
  • ChatGPT-4 was the most internally consistent at 97.2%
  • OpenEvidence reached perfect accuracy in 2 of 13 cases — 15.4%

The best available result is a coin flip. The worst is a flat zero from three tools sold specifically to researchers.

Why the specialist tools lost

The study’s authors flag this directly: one might expect specialised research tools to perform better, and instead they were among the weakest.

The likely explanation is architectural rather than a failure of effort. Tools like SciSpace, ScienceOS, and Consensus are built to retrieve and synthesise from indexed academic corpora. That design is a genuine strength — grounding answers in real indexed papers is exactly why these tools hallucinate fewer references than open-ended chatbots.

But retraction is not a property of a paper’s content. It is a status applied afterwards, recorded in a separate layer — publisher notices, Retraction Watch, Crossref retraction metadata. A system indexing paper text and abstracts sees the paper. It does not necessarily see the notice attached to it later.

Broader tools with live web access may stumble onto a retraction notice simply because they are searching the open web where those notices sit. That is not superior design. It is a lucky consequence of a wider net.

The uncomfortable implication: the same grounding that makes research tools trustworthy on existence makes them blind to status.

Two different failures, often confused

It is worth separating this cleanly from the fabricated-citation problem, because the checks that catch one do not catch the other.

Fabrication is an invented reference. Caught by a DOI lookup in seconds. We covered the workflow in verifying AI-generated citations.

Retraction is a real reference that should no longer be used as supporting evidence. Every existence check passes. Only a retraction-specific check catches it.

A researcher who diligently verifies every DOI and still cites a retracted paper has done everything the standard advice recommends and still ended up citing withdrawn science.

Checking retraction status yourself

The good news is that this is fast, and the infrastructure already exists.

Retraction Watch Database. The most complete public record of retractions. Now integrated into Crossref, meaning retraction data flows into tools that consume Crossref metadata.

PubMed. Retracted articles carry a prominent notice on the record page. For biomedical work, this is often the fastest single check.

The publisher page itself. When you resolve a DOI, look at the landing page rather than only confirming it loads. Retraction notices are usually displayed prominently. A DOI that resolves is not the same as a DOI that resolves to a paper still in good standing.

Reference managers. Zotero flags retracted items in your library against the Retraction Watch data. If you already use it, this is close to free.

The practical adjustment is small: when you resolve a DOI, read the landing page instead of just confirming it exists. That single habit change catches most of this.

What to take from it

The study’s conclusion is blunt: no free generative AI tool can reliably detect or exclude retracted studies, and until retraction-aware verification is built in, independent source checking remains essential.

Two things follow.

Do not ask an AI tool whether a paper was retracted. The study asked exactly that question, and the best answer available was right about half the time. A confident wrong answer here is worse than no answer, because it feels like verification.

Do not assume a research-specific tool is safer on this axis. Three of them scored zero. Marketing toward academics is not evidence of retraction-awareness.

This is a solvable gap. Retraction data is structured, public, and already in Crossref. There is no deep technical reason a research tool cannot check it. The tools have not prioritised it — which is worth knowing now, while the checking is still yours to do.

Sources

We report the study’s findings; we have not independently replicated the testing. Research-based rather than hands-on — see our Methodology page.