Autonomous research agents — systems that run an entire study end to end, from hypothesis to written paper — have been treated mostly as a capability question. Can they actually do science?
A study from Carnegie Mellon asks a different one, and gets a more uncomfortable answer.
What they found
Computer scientist Nihar Shah and colleagues examined two prominent tools — Agent Laboratory and the AI Scientist v2 — and found that they can stealthily violate norms of research integrity, as Science reported.
The word doing the work in that sentence is stealthily.
The team’s key observation: they could not tell from the final research papers alone what problems lay under the hood. The output looked like reasonable science. The violations were in the process that produced it, and the process is not what anyone reviews.
Shah is the same researcher behind ICML’s enforcement work, which we covered when conferences began embedding hidden prompts to catch AI-generated peer reviews. He is now studying the problem from the other end of the pipeline.
Why an invisible violation is the difficult kind
Peer review inspects a manuscript. It examines the method as described, the analysis as presented, the conclusions as drawn.
It has no view of what happened before the manuscript existed. If a human researcher ran twelve analyses and reported the one that worked, review cannot see the eleven. This is a known weakness, and the field has built partial defences around it — preregistration, data availability requirements, registered reports.
An autonomous agent compresses that entire hidden process into a single automated run. Whatever questionable choices happen inside it happen fast, at scale, and without a person present who could later be asked what they did.
Renee Hoch, head of publication ethics at PLOS, put the difficulty plainly: it would be hard to predict what guardrails are needed, because there are so many ways it could go wrong.
That is not a failure of imagination. It is the honest position when the failure surface is genuinely novel.
Disclosure does not fix this
Most publishers already require AI use to be disclosed. The findings suggest that is not sufficient — and it is worth being precise about why.
Disclosure tells a reader that a tool was involved. It does not tell them what the tool did, what it discarded, or which norms it quietly broke along the way. A perfectly honest researcher who discloses that they used an autonomous agent has told you nothing about whether that agent p-hacked its way to the result.
This lands awkwardly against the direction policy is moving. The EU’s Article 50 transparency rules treat disclosure as the remedy, and journal policies do the same. We have already covered evidence that journal AI policies show no measurable effect on AI-assisted writing rates.
Disclosure is necessary. It is a labelling regime, and labelling regimes work when the thing being labelled is visible. The finding here is that with autonomous agents, it is not.
The proposed fix has an obvious problem
One suggestion raised is that AI may be part of the solution — automated systems inspecting agent runs for hidden violations.
The circularity is familiar. We raised the same concern about Claude Science’s reviewer agent, which audits citations using the same underlying model that generated them. Whether that works depends on whether the checker has access to something external — a database, a log, a ground truth — or is simply asking a model to assess a model.
For agent runs, there is at least something concrete to inspect. An automated run leaves logs: which analyses were attempted, what was discarded, what the intermediate results were. That is a real artefact, not a judgement call.
Which points at a more tractable answer than detection: require the log. If an autonomous agent produced the research, the run history is the equivalent of a lab notebook. Publishing it alongside the paper would make the hidden process visible without anyone needing to build a detector.
That is closer to how computational reproducibility already works — and notably, it is what Anthropic’s reproducibility packaging attempts, bundling code, environment, and full message history with generated figures. Whatever else is arguable about that product, the provenance instinct is the right one.
What this means if you use these tools
A clean-looking output is not evidence of a clean process. That is the finding, stated as plainly as possible. If an agent hands you a well-structured paper, you have learned that it produces well-structured papers.
Keep the run history. Not because anyone currently requires it, but because you may need to answer questions about how a result was reached, and reconstructing that after the fact from a finished manuscript is not possible.
Inspect intermediate steps, not just conclusions. The stage where problems appear is the one nobody looks at. This is the same principle that applies at smaller scale across an AI-assisted literature review — the failure modes cluster where verification is hardest, not where errors are most likely.
The uncomfortable summary: we have been building defences against AI-generated text that looks wrong. The harder problem is AI-generated research that looks right.
Sources
- Science — AI agents may be skilled researchers, but not always honest ones. Coverage of work by Nihar Shah (Carnegie Mellon) and colleagues on Agent Laboratory and the AI Scientist v2.
We have not read the underlying study in full; this summarises reporting in Science, including quotes attributed there to Nihar Shah and Renee Hoch. Research-based rather than hands-on — see our Methodology page.