Regarding: "The Decoherence Trap"

A friend shared this article with me: The Decoherence Trap.

I think it's worth a read, although it's technical, so your mileage may vary depending on whether or not that's your thing. Much of what I'm saying below depends somewhat on having read the article.

Over all, I consider that the article is an interesting and compelling idea, one that resonates with me in some local sense, but seems to be a little bit too sure of itself in applying that local insight to the global case.

0) Future as probability distribution

One key model for how I think about the future is the probability distribution. There’s some chance of X happening, there’s some chance of Y happening. Sometimes it's densities over continuously variable quantities. Like a car drive has a probability distribution over how long it will take. Traffic variability spreads out the center of the distribution, and more at rush hour. But there’s always little bit of probability sitting on the chance that I will die in transit and never arrive. Whenever we think about the future, the mean is crazy due to outliers. But spending too much time on the outliers is bad for mental health (“catastrophizing”). Sane planning fits better with the mode +/- some day-to-day variation.

But we’re not just interested in probabilities, we’re interested in expected impact. So it’s useful to think about unlikely things if they would be highly impactful — although this leans against the interdiction of “no catastrophizing”. In the best case, we can plan for the mostly likely outcomes, while also keeping ourselves safe by addressing dangerous unlikely scenarios (like wearing seatbelts).

(As an aside, we have to do something about consequences that claim to have infinite or very large impacts. One approach would be to simply clip these impact to some high absolute value. You can't scare me with the threat of hell and you can't tempt me with heaven.)

So when I approach AI, I try to think mostly about what the common outcomes will be, but also spare a few brain cycles for potential disasters and potential strokes of unusually good luck.

1) Coherence is a useful view in software engineering

The local sense: as a professional, I'm finding myself trying to leverage AI in some way that actually achieves organizational goals faster. This is a difficult process! I think the concept of coherency is helpful in understanding this. A precautionary approach is to demand solutions that fit in the space of "What humans can handle (and fix over time)" (the article's words), but that's a little bit too conservative actually. LLMs aren’t gonna disappear, and whatever my monkey brain might wish or sort of expect based on our evolutionary history where bountiful fruit trees sometimes die. The mode of the future is that LLMs will become much more important to software even than they are now.

The article says: "If we push too long and too far without maintaining or rebuilding coherence, with current trends, we'll end up in a place where we must use agents to maintain our systems, because the systems will be so large and so overwhelming that we cannot do so ourselves." Marry that with the experience of the VC funded service that undercuts+kills the incumbent (bait) then later grows into a publicly traded low-number-opoly just to enshitify to a greater degree than the original incumbent (switch). The kilodollars that software companties / every hobbyist and their dog throws at Anthropic / OpenAI today become megadollars tomorrow, and potentially that's after X% of us lose our jobs.

I see my responsibility as a developer is to deliver shareholder value on a little bit longer a time scale than  management can withstand, so I actively spend time & effort pushing against the natural flow of entropy toward decoherence. I did this before LLMs; I learned that I should refactor code without my management’s permission or knowledge.

So I feel a big “yes” to this article’s idea of software coherence for most of my professional practice.

2) Claudism detected

"a subtle but load-bearing moment"

I wonder how much this particular Claudism actually defeats the entire article. Probably not much, especially if it were deliberately written for ironic effect, but this phrase definitely didn’t fit.

3) What does this prove about recursive self-improvement / AI safety?

Ok, decoherence is a warning for AI software development practice. But what about the global sense? The article seems to argue from an assemblage of concepts from software engineering and machine learning to a grand conclusion that recursive self-improvement (RSI) will be bounded by decoherence. Something will eventually bound any exponential growth process, but it might not be software decoherence for the case of AI repetitive strain injury--I mean recursive self-improvement.

Back to the future as probability distribution. Most of the probability weight I have on the future is on fast growth in AI, but no “AGI”, no apparently-boundless intelligence, no “singularity”, etc. But I do reserve a little tiny bit of weight to put on strange, dramatic outcomes. History proves that whatever foresight we have ends up being wrong somehow, usually extremely so, such that whatever comes is differently-shaped than we could have ever expected.

So I have a sort of irrational/hyper-rational skepticism in the confidence of the article’s conclusion that recursive self-improvement will be limited by the dynamics of software coherence. The closest thing I would have toward “evidence” for this skepticism is that observation that AI systems have been gaining increasing capabilities to reason. Reason is all that we humans have to get around our own limited training data and reason works pretty darn well indeed.

By the way, why care about AI safety? (I mean, besides the cybersecurity risk, as proved by OpenAI hacking HuggingFace.)

Say an alien spaceship is headed for Earth. It has 30 aliens on it. The aliens are weak and small. They have no weapons and carry no diseases. They breed at rates similar to humans. They are bringing no new technology. No other ships are coming. There’s no trick—except that they each have an IQ of 300. Would you find that concerning?

Quoted from the Dynomight article Y'all are over-complicating these AI-risk arguments.

(Bonus link to increase your AI safety-related stomach aches: How abliterated models can get you pwned)

Does the decoherence argument invalidate the premise that AIs might be smarter than people? I don't think so.

4) Beating slop

I think the AI-on-the-internet / AI-for-everyone situation is pretty blurry because the distribution of outcomes is insanely wide. 

We’ve the normal variation among humans, then we’ve got a power-law distribution variation in AI model power and power-law distributions in organizational effectiveness.

Social media selects for the worst people with low-quality AI models and negative organizational effectiveness. (In my estimation, Lines is significantly better!)

So we’ve got a continuum from worst case to average to best case for a given group of entities drawn from {horse, person, centaur, reverse-centaur}. And the outcomes range from exceptionally dangerous artifacts to slop to glorious centaur-artifacts. 

Mostly it’s slop and that’s a huge problem. Already, we had a lot of mediocre humans making mediocre software and mediocre art. But when everyone from the barflies to the president is posting AI slop on social media, it’s obviously a danger to society. I’ll do my best not to add to the problem, but I don’t have much agency to address this problem in a meaningful way. There’s a real problem short term given that there’s now a lot more to ignore (including from powerful people), but, when requested to ignore things, history has always been happy to oblige.

The distribution in access to powerful models and well-functioning organizations results a disparity in appreciation for what AI can do. And here I quote a tweet from Andrej Karpathy (who coined the term "vibe coding"):

this gap in shared understanding of LLM capability is widening. It's now less "two groups of people speaking past each other" and more a sharp funnel.

• Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
• Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. ...
• Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. ...
• And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally.

Link. (As penitence for linking to x.com, I hereby sentence myself to 20 pushups. ... Ok we're back.)

I’m less interested in slop / bad software / bad art / useless content than I am in achieving and understanding the most effective combinations of AI and human effort. I care about the combinations of high-quality AI models, great people and teams with high organizational effectiveness. What glorious centaur-artifacts can we produce? How does Terence Tao use Chat GPT? What can I do now that I couldn't before?

5) On technology and progress

A key part of my own framing is that I see the success of LLMs as being a stroke of unusually good luck. 

Back in 2010 or so, my soon-to-be-PhD supervisor had some ideas about how prediction might encode intelligence. I looked as hard as I could at that idea, read Claude Shannon (the dude that the AI agent is named after), realized that prediction had fundamental limits (nothing has changed here) and soft-gave up on the idea. Before GPT 3.5, I didn’t think that the secret to artificial intelligence would be language models and while some insiders might have had that idea, it wasn't a common idea as far as I can tell. 

The emergent intelligence displayed by large language models was a rather counterintuitive jump, so it’s quite fortuitous that we managed to stumble onto some means of building intelligent systems at all. While there have been bad outcomes from LLMs, they mostly seem to be from overhype and overuse of LLMs or from the ways in which the LLM systems that we currently have fail at being intelligent.

Humans are gonna be the same as we've always been. Progress, in my humble opinion, depends on better social organization and technology. I've always wanted to do something more than maintaining the baseline. (I now realize that's probably just the way I am on some fundamental level and not some kind of virtuous value.) Social organization is great, and other souls will do a better job than. For me, I'm all in on technology. That's not to say that every technology is good, or specifically that LLMs are good. But for me, the value of technology is proven by history and it's worth trying to make computers more capable.

The mode of the distribution of futures that I see coming is one where Moore's law continues to drive machine learning to greater and greater successes. The drama of today's LLMs is gonna be dwarfed by whatever thing is coming next. And that won't be the end, either.