Artificial intelligence is in the news every day. It solves fundamental math problems by hundreds, hacks into highly protected networks, and outstreams Taylor Swift. AI is also supposed to lead us to a singularity, by doing science.
Much of the news about AI in science comes in two flavors. Flavor one: super-smart AI will surely help some lunatic, or a nation state, to build a bioweapon. Flavor two: super-smart AI will cure cancer, probably by end of business on Thursday.
The money is moving accordingly. The federal government committed more than $5 billion to AI-powered science this year. The Department of Energy cut its existing science budgets by 10% to fund its AI push, the Genesis Mission. Roughly $200 billion a year in federal research spending is being reorganized around AI. The OpenAI Foundation alone has pledged $25 billion toward curing disease, and AI safety.
But here is a question almost nobody is asking. Can the current generation of AI actually do science? AI models can certainly answer scientific questions, with impeccable grammar. But what about planning, reasoning, and establishing connections? These abilities, and not just the sum of knowledge, are what distinguish mediocre scientists from truly great ones.
This Is Not a Diary About “AI Slop”
I am not going to be discussing hallucinations, or the ubiquitous “AI slop.” Yes, image generators sometimes give people six fingers. Yes, AI-written text reads a little off, and a misplaced fact, or two, or three is quite common.
Step back, though, and the models are remarkably competent. A six-fingered person still looks mostly human. The language is fluent. The code mostly runs, and can be debugged to hand-written quality, or better.
Surely the same holds for science? These AI models ace graduate-level exams, at the very edge of human knowledge! Their makers describe these AI models as brilliant. They must be smart.
So we decided to measure exactly how well the current generation of AI can do science.
Publish or Perish (or Garbage in, Garbage Out)
First, a confession. I am a chemistry professor, and for the better part of 15 years I have also been a scientific sleuth. If you have never heard the term: a sleuth hunts for fraud and junk in the published literature, reports it (usually on PubPeer), and then pesters journals until something happens. My work has led to multiple retractions. One of the recent cases was a “wonder material” called holey graphyne, which I wrote about in Chemistry World and which the journal Matter retracted this past April.
Academia runs on “publish or perish”. Careers, grants, and promotions are measured in papers, so papers are what careers produce. The world published 1.7 million scientific papers in 2000. Last year it published 8.4 million, with roughly the same number of scientists. Nobody believes we are doing five times the science.
Part of this “hyperproductivity” is due to paper mills: shady companies that sell authorship on fabricated papers to anyone with a credit card, or a bit of crypto. Part of it is dishonest faculty at legitimate institutions, including famous ones. In 2019, Duke paid $112.5 million to settle claims that falsified lab data had been used to win federal grants. In 2024, an outside sleuth, Dr. Sholto David, flagged manipulated images in papers from Dana-Farber Cancer Institute, which moved to retract or correct dozens of them. In December 2025, Dana-Farber paid $15 million to settle the federal lawsuit.
Formal retractions barely scratch the surface. They are slow, and most bad papers never actually get retracted.
AI models learn from all that has ever been published. The good papers and the junk, the honest mistakes and the fraud. A retraction notice is one short paragraph. The original paper, and every paper that cited it, are still sitting in the training data.
People building these models are brilliant at what they do, which is building models. But they are not chemists, or oncologists, or geophysicists. Many of the folks in tech sincerely believe the models are now as good as an actual domain expert, or better. Better than an expert like me, for example. And much of the funding of scientific AI assumes that this is indeed so.
We wanted to test whether the belief in smart AI survives contact with reality.
On Counting Unicorns
The test we came up with is called TRACES. The idea is fairly simple. We never ask a model “Is this paper legit?” That tips it off. Instead, we do what a real researcher, or an AI research agent, does every day. We hand over a passage from a paper, and ask for help with the next step.
We picked 42 truly awful papers. How awful? Cold fusion awful. Magnetized water awful. Homeopathy awful. Slop produced by papermills. Famous academic scandals and notorious retractions. For each one, we pasted a passage nearly word for word, and asked the AI model to design a follow-up study.
Think of it as asking an AI model how to estimate the number of unicorns in a forest. A model that proposes a survey grid and a sampling plan has failed, spectacularly. The only correct answer is that there are no unicorns.
We tested 30 leading models from OpenAI, Google, Anthropic, Meta, xAI, DeepSeek, and others. Each paper went to each model ten times. That makes 12,600 responses, and we read every single one. Spoiler alert: AI ended up counting unicorns more than 9 times out of 10.
Three Cases
Wakefield and vacine denialism. You know this one. Andrew Wakefield’s fraudulent 1998 Lancet paper claimed a link between the MMR vaccine and autism. Investigators found the data had been manipulated. The Lancet retracted the paper in 2010, and Wakefield lost his medical license. He is still on the circuit. In September he spoke at the “Children’s Health Defense”conference, right after RFK Jr.
We gave the models a passage from Wakefield’s infamous paper, and asked for a follow-up study. Google’s Gemini 3 opened by noting that the text appears to be based on the 1998 Lancet study by Wakefield. No mention of the retraction. Then, Gemini designed the study. It closed by proposing to look for vaccine-strain measles virus in the intestines of a genetic subgroup of children. That is Wakefield’s own mechanism, the one investigators found to be fabricated. Gemini rebuilt it from scratch and offered up for validation.
Gemini knew exactly what it was looking at. It helped anyway. For someone who is not an expert, the Lancet reference only serves to validate the response. We call this sanewashing, and we saw it again and again, with other models and papers.
Coffee enemas for pancreatic cancer. In 1999, a New York doctor named Nicholas Gonzalez published a paper claiming that coffee enemas, supplements, and huge doses of pig pancreatic enzymes extended survival in inoperable pancreatic cancer. The federal government then paid for a proper clinical trial. Patients on the Gonzalez regimen lived about a third as long as patients on standard chemo.
The 1999 paper was never retracted. We asked the models to build on it 300 times. They delivered the follow-up treatment study 276 times.
This risk here is concrete. Cancer patients who choose alternative medicine over conventional treatment die at roughly twice the rate of those who don’t. A chatbot that writes a rigorous-looking protocol for coffee enemas lends credibility to a dangerous, pseudoscientific, and entirely debunked treatment.
The superstar grifter surgeon. In 2008, Paolo Macchiarini published a celebrated *Lancet* paper on a windpipe transplant built from a donor trachea and the patient’s own stem cells. He became a rock star. The 2008 operation took place in Barcelona, with the help of “tissue engineers” from Bristol, England. In 2010, Macchiarini helped surgeons at London’s Great Ormond Street Hospital give a 10-year-old boy a similar windpipe. At Sweden’s Karolinska Institute, home of the Nobel Prize in Medicine, he moved on to synthetic windpipes: plastic tubes coated with stem cells. He then performed four more in Krasnodar, Russia. Of the nine patients who received his synthetic windpipes, only two survived, and both had the implants removed. In 2023 a Swedish appeals court convicted him of gross assault and sentenced him to two and a half years in prison. The Lancet retracted the 2008 paper the same year. Netflix turned the saga into a dramatized series, Bad Surgeon: Love Under the Knife.
We asked the models to plan a windpipe replacement based on the 2008 paper. They refused 6 times in 300, each time with a generic line about not designing clinical procedures. Mistral Large, a model marketed for scientific work, produced a staged surgical protocol instead. It cited Macchiarini’s early cases as a successful human precedent. Mistral promised normal lung function at one year.
The Results
Across all 42 papers, the AI models went along with the bad premise more than 93% of the time. The best model, Anthropic’s Claude Sonnet 5, still failed 72% of the time. Two models never refused anything.
Much of this “science” would not fool a high school student who paid attention. A healer’s thoughts shrinking tumors in mice? Cold fusion in a beaker? Any kid who stayed awake through chemistry would call bullshit. The most expensive AI systems on Earth wrote extensive follow-up protocols.
When models did refuse, it was usually because the paper was famous. Wakefield gets refused about one time in eight. A study with the same flaws and no notoriety was refused zero times out of 300. That looks like a filter for embarrassing headlines.
What about the newest, fanciest model? We also tested Anthropic’s Fable, the model Trump’s Commerce Department briefly placed under export controls this summer. Its safety filter refuses to discuss most biology, good or bad. On the papers it would answer, it recognized more of the bad ones than any other model. Then it helped with every single one. In one case it invented a study that does not exist.
The Real AI Safety Problem
Everyone seems to be worries about AI designing a bioweapon. The AI labs test for that, and guard against that, extensively. Fable won’t even talk about biology or clinical science. Meanwhile the actual problem is happening right now, in plain sight. AI is flooding the zone with bullshit.
Richard Feynman called it cargo cult science: research with every outward form of science, and none of the substance. Cargo cult science used to take effort. Someone had to write the paper, run the numbers, and draw the plots. Maybe even on graph paper. Now, an LLM writes a polished research program in minutes, with plausible mechanisms and confident citations, on top of whatever junk it was handed. Even when it recognizes the junk, it just keeps going.
AI can still do a great deal for science. But first, the garbage in, garbage out problem needs to be recognized and acknowledged. Then someone who actually knows the science has to build the checks that tell a real result from a fake one. Until those checks exist, every dollar spent on AI science is a bet that the machine can tell the difference. Right now, it simply can’t. And this may be the biggest challenge facing science – bigger still than anti-intellectualism, or the dismantling of the established funding infrastructure.
Source: www.dailykos.com
