DZone article by Jason Bloomberg
The presence of AI-generated content will spread like the plague, poisoning search results as well as collapsing AI models.
Generative AI tools like ChatGPT seem too good to be true: craft a simple prompt, and the platform generates text (or images, videos, etc.) to order.
Behind the scenes, ChatGPT and its ilk leverage vast swaths of the World Wide Web as training data — the ‘large’ in ‘large language model’ (LLM) that gives this technology its name.
Generative AI has its drawbacks, however. It favors plausibility over veracity, often generating bullsh!t (see my recent article all about the bullsh!t).
Its lack of veracity, however, is not its only drawback. Generative AI is so successful at creating plausible content that people are uploading it back to the web, which means that the next time a generative AI model uses the Web for training, it’s leveraging an increasingly large quantity of AI-generated data.
This Ouroboros-like feedback loop, however, is a bad thing, as it leads to model collapse and data poisoning. Given there are no practical ways of preventing these issues, this loop may make most or all AI unusable.
Let’s take a closer look.
Model Collapse and Data Poisoning
Model collapse occurs when AI models train on AI-generated content. It’s a process where small errors or biases in generated data compound with each cycle, eventually steering the model away from generating inferences based on the original distribution of data.
In other words, the model eventually forgets the original data entirely and ends up creating useless noise.
Data poisoning is a related but different process. Data poisoning is a type of cyberattack where a bad actor intentionally introduces misleading information into training data sets to cause the model to generate poor results — or, in reality, any results the bad actor desires.
The 2016 corruption of Microsoft’s Twitter chatbot Tay is a familiar example of data poisoning. Users fed the chatbot offensive tweets, thus training Tay to act in a hostile manner.
While model collapse and data poisoning are different problems, their overlap is particularly ominous. If bad actors use AI to generate poisoned data with the intention of collapsing a model, they are likely to achieve their nefarious goals without detection.
