By Hamza Faisal
Nearly half of what the AI world thinks it knows about large language models may already be out of date. Stanford researchers Owen Queen and James Zou retested 50 widely cited findings about LLMs on 20 newer models — and found that 44% of them no longer hold.
The study, titled “The shelf life of LLM findings” and published on November 2, took 50 well-known claims and ran each experiment again on models released between July 2024 and September 2026 from eight developers. A finding counted as “expired” when its effect had shrunk to one-fifth or less of its original size by September 30, 2026.
The scoreboard is stark: 22 of the 50 findings, or 44%, had expired. Another 34% had weakened. Only 16% still held at full strength, while 6% had grown stronger. Among the 22 findings whose expiry date could be estimated, the average lifespan was just 15.7 months, with a median of 14.7 months. Four percent of claims expired within six months, 16% within a year, and 39% within two years.
What expired — and what survived
The findings that faded fastest were about capability failures: claims that models could not count letters, could not do arithmetic in other number bases, could not find a fact buried in a long document, invented citations, or gave up a correct answer when pushed back. Reasoning-related weaknesses expired fastest of all: five of nine had expired, and none persisted or grew stronger.
The clearest example is fabricated citations. A 2023 study found GPT-3.5 invented 55% of the works it cited. When Queen and Zou repeated the same prompts, GPT-3.5 Turbo fabricated 59% of 210 references — but Claude Opus 4.6 fabricated only 4% of 302. Not every model has improved equally, though: Qwen3.7-Flash still fabricated 69% of its references.
The findings that proved most durable were about preferences and judgment rather than raw capability. Sycophancy, evaluator bias and left-leaning political bias were among the claims that persisted, suggesting some behaviours are baked into training and alignment rather than fixed by scale alone.
The 15-month shelf life
The study’s central message is that an empirical finding about an AI model is a snapshot of one model on one day, not a law of nature. Models are retrained, fine-tuned and replaced so quickly that a result published this year can be history by the next. The researchers fitted trends against model release dates; each claim was tested with only one protocol, and substitute models stood in where the originals had been retired.
The pair used coding agents, including Claude Code and Codex, to run the retests far faster than manual work would allow. The project is published as an interactive page with open code and data under an MIT licence, so other researchers can check the classifications themselves. It has not yet been peer-reviewed.
What the researchers want changed
The pair call on researchers to treat AI findings as observations tied to a specific model and date: document the exact model version, publish the prompts and the analysis code, and retest claims on new model versions instead of citing old results as settled fact. For anyone building products, writing policy or quoting AI research, the implication is plain — check the date on the finding before trusting it.
The study does not mean 44% of all AI research is obsolete — its 50 claims were hand-picked, not a random sample. But as a warning about how fast AI knowledge decays, it is hard to ignore: in this field, a finding’s half-life is now measured in months.
By Hamza Faisal



