Some Signs Of Ai Model Collapse Begin To Reveal Themselves

Opinion I use AI a lot, but not to write stories. I use AI for search. When it comes to search, AI, especially Perplexity, is simply better than Google.

Ordinary search has gone to the dogs. Maybe as Google goes gaga for AI, its search engine will get better again, but I doubt it. In just the last few months, I’ve noticed that AI-enabled search, too, has been getting crappier.

Guide for the perplexed – Google is no longer the best search engine

READ MORE

In particular, I’m finding that when I search for hard data such as market-share statistics or other business numbers, the results often come from bad sources. Instead of stats from 10-Ks, the US Securities and Exchange Commission’s (SEC) mandated annual business financial reports for public companies, I get numbers from sites purporting to be summaries of business reports. These bear some resemblance to reality, but they’re never quite right. If I specify I want only 10-K results, it works. If I just ask for financial results, the answers get… interesting,

This isn’t just Perplexity. I’ve done the exact same searches on all the major AI search bots, and they all give me “questionable” results.

Welcome to Garbage In/Garbage Out (GIGO). Formally, in AI circles, this is known as AI model collapse. In an AI model collapse, AI systems, which are trained on their own outputs, gradually lose accuracy, diversity, and reliability. This occurs because errors compound across successive model generations, leading to distorted data distributions and “irreversible defects” in performance. The final result? A Nature 2024 paper stated, “The model becomes poisoned with its own projection of reality.”

Model collapse is the result of three different factors. The first is error accumulation, in which each model generation inherits and amplifies flaws from previous versions, causing outputs to drift from original data patterns. Next, there is the loss of tail data: In this, rare events are erased from training data, and eventually, entire concepts are blurred. Finally, feedback loops reinforce narrow patterns, creating repetitive text or biased recommendations.

I like how the AI company Aquant puts it: “In simpler terms, when AI is trained on its own outputs, the results can drift further away from reality.”

I’m not the only one seeing AI results starting to go downhill. In a recent Bloomberg Research study of Retrieval-Augmented Generation (RAG), the financial media giant found that 11 leading LLMs, including GPT-4o, Claude-3.5-Sonnet, and Llama-3-8 B, using over 5,000 harmful prompts would produce bad results.

RAG, for those of you who don’t know, enables large language models (LLMs) to pull in information from external knowledge stores, such as databases, documents, and live in-house data stores, rather than relying just on the LLMs’ pre-trained knowledge.

You’d think RAG would produce better results, wouldn’t you? And it does. For example, it tends to reduce AI hallucinations. But, simultaneously, it increases the chance that RAG-enabled LLMs will leak private client data, create misleading market analyses, and produce biased investment advice.

As Amanda Stent, Bloomberg’s head of AI strategy & research in the office of the CTO, explained: “This counterintuitive finding has far-reaching implications given how ubiquitously RAG is used in gen AI applications such as customer support agents and question-answering systems. The average internet user interacts with RAG-based systems daily. AI practitioners need to be thoughtful about how to use RAG responsibly.”

That sounds good, but a “responsible AI user” is an oxymoron. For all the crap about how AI will encourage us to spend more time doing better work, the truth is AI users write fake papers including bullshit results. This ranges from your kid’s high school report to fake scientific research documents to the infamous Chicago Sun-Times best of summer feature, which included forthcoming novels that don’t exist.

What all this does is accelerate the day when AI becomes worthless. For example, when I asked ChatGPT, “What’s the plot of Min Jin Lee’s forthcoming novel ‘Nightshade Market?'” one of the fake novels, ChatGPT confidently replied, “There is no publicly available information regarding the plot of Min Jin Lee’s forthcoming novel, Nightshade Market. While the novel has been announced, details about its storyline have not been disclosed.”

Once more, and with feeling, GIGO.

Some researchers argue that collapse can be mitigated by mixing synthetic data with fresh human-generated content. What a cute idea. Where is that human-generated content going to come from?

Given a choice between good content that requires real work and study to produce and AI slop, I know what most people will do. It’s not just some kid wanting a B on their book report of John Steinbeck’s The Pearl; it’s businesses eager, they claim, to gain operational efficiency, but really wanting to fire employees to increase profits.

Quality? Please. Get real.

We’re going to invest more and more in AI, right up to the point that model collapse hits hard and AI answers are so bad even a brain-dead CEO can’t ignore it.

How long will it take? I think it’s already happening, but so far, I seem to be the only one calling it. Still, if we believe OpenAI’s leader and cheerleader, Sam Altman, who tweeted in February 2024 that “OpenAI now generates about 100 billion words per day,” and we presume many of those words end up online, it won’t take long. ®

Original Source