Decoherence grows relative to the rate of change of a system, and it increases faster the more decoherent a system is, resulting in runaway exponential growth. It is not a new phenomenon, and humans have always countered it through carefully finding ways to do more with less, unifying disparate patterns and models into a cohesive whole. However, the widespread use of AI is currently producing it at a rate that is unsustainable.
The good news is that this rules out the specter of runaway recursive self-improvement, the intelligence explosion that goes parabolic before anyone can react, as decoherence both prevents AI from improving in a bubble and limits the rate at which it can improve outside of it. The bad news is that we are barreling toward a different kind of disaster: Decoherence Collapse.
The article makes general assertions about systems. It gives plumbing, software, and machine learning as examples, and I simply add the examples of human learning and human societies.
An agent specifying the criteria of a test inspired by the real world and then evaluating the performance of a bunch of other agents is equally information-incestuous when those agents are LLMs as when they are humans. If the observations of the article must hold true for LLM training, then they must hold true for humans. If they do not hold true for humans, then they do not prohibit the existence of LLM training programs that likewise avoid the decoherence trap.
Given the fact that evolution was able to find human creativity and art as forms of cohering enrichment of human performance beyond the limits of the training data of a world without art, I would expect that there is a good chance tens of thousands of engineers and billions of dollars of compute will be able to find some form of abstract expression and learning as a form of cohering enrichment of LLM performance beyond the limits of the training data of a world without that abstract LLM expression.