Regarding: "The Decoherence Trap"

A friend shared this article with me: The Decoherence Trap.

I think it's worth a read, although it's technical, so your mileage may vary depending on whether or not that's your thing. Much of what I'm saying below depends somewhat on having read the article.

Over all, I consider that the article is an interesting and compelling idea, one that resonates with me in some local sense, but seems to be a little bit too sure of itself in applying that local insight to the global case.

0) Future as probability distribution

One key model for how I think about the future is the probability distribution. There’s some chance of X happening, there’s some chance of Y happening. Sometimes it's densities over continuously variable quantities. Like a car drive has a probability distribution over how long it will take. Traffic variability spreads out the center of the distribution, and more at rush hour. But there’s always little bit of probability sitting on the chance that I will die in transit and never arrive. Whenever we think about the future, the mean is crazy due to outliers. But spending too much time on the outliers is bad for mental health (“catastrophizing”). Sane planning fits better with the mode +/- some day-to-day variation.

But we’re not just interested in probabilities, we’re interested in expected impact. So it’s useful to think about unlikely things if they would be highly impactful — although this leans against the interdiction of “no catastrophizing”. In the best case, we can plan for the mostly likely outcomes, while also keeping ourselves safe by addressing dangerous unlikely scenarios (like wearing seatbelts).

(As an aside, we have to do something about consequences that claim to have infinite or very large impacts. One approach would be to simply clip these impact to some high absolute value. You can't scare me with the threat of hell and you can't tempt me with heaven.)

So when I approach AI, I try to think mostly about what the common outcomes will be, but also spare a few brain cycles for potential disasters and potential strokes of unusually good luck.

1) Coherence is a useful view in software engineering

The local sense: as a professional, I'm finding myself trying to leverage AI in some way that actually achieves organizational goals faster. This is a difficult process! I think the concept of coherency is helpful in understanding this. A precautionary approach is to demand solutions that fit in the space of "What humans can handle (and fix over time)" (the article's words), but that's a little bit too conservative actually. LLMs aren’t gonna disappear, and whatever my monkey brain might wish or sort of expect based on our evolutionary history where bountiful fruit trees sometimes die. The mode of the future is that LLMs will become much more important to software even than they are now.

The article says: "If we push too long and too far without maintaining or rebuilding coherence, with current trends, we'll end up in a place where we must use agents to maintain our systems, because the systems will be so large and so overwhelming that we cannot do so ourselves." Marry that with the experience of the VC funded service that undercuts+kills the incumbent (bait) then later grows into a publicly traded low-number-opoly just to enshitify to a greater degree than the original incumbent (switch). The kilodollars that software companties / every hobbyist and their dog throws at Anthropic / OpenAI today become megadollars tomorrow, and potentially that's after X% of us lose our jobs.

I see my responsibility as a developer is to deliver shareholder value on a little bit longer a time scale than  management can withstand, so I actively spend time & effort pushing against the natural flow of entropy toward decoherence. I did this before LLMs; I learned that I should refactor code without my management’s permission or knowledge.

So I feel a big “yes” to this article’s idea of software coherence for most of my professional practice.

2) Claudism detected

"a subtle but load-bearing moment"

I wonder how much this particular Claudism actually defeats the entire article. Probably not much, especially if it were deliberately written for ironic effect, but this phrase definitely didn’t fit.

3) What does this prove about recursive self-improvement / AI safety?

Ok, decoherence is a warning for AI software development practice. But what about the global sense? The article seems to argue from an assemblage of concepts from software engineering and machine learning to a grand conclusion that recursive self-improvement (RSI) will be bounded by decoherence. Something will eventually bound any exponential growth process, but it might not be software decoherence for the case of AI repetitive strain injury--I mean recursive self-improvement.

Back to the future as probability distribution. Most of the probability weight I have on the future is on fast growth in AI, but no “AGI”, no apparently-boundless intelligence, no “singularity”, etc. But I do reserve a little tiny bit of weight to put on strange, dramatic outcomes. History proves that whatever foresight we have ends up being wrong somehow, usually extremely so, such that whatever comes is differently-shaped than we could have ever expected.

So I have a sort of irrational/hyper-rational skepticism in the confidence of the article’s conclusion that recursive self-improvement will be limited by the dynamics of software coherence. The closest thing I would have toward “evidence” for this skepticism is that observation that AI systems have been gaining increasing capabilities to reason. Reason is all that we humans have to get around our own limited training data and reason works pretty darn well indeed.

By the way, why care about AI safety? (I mean, besides the cybersecurity risk, as proved by OpenAI hacking HuggingFace.)

Say an alien spaceship is headed for Earth. It has 30 aliens on it. The aliens are weak and small. They have no weapons and carry no diseases. They breed at rates similar to humans. They are bringing no new technology. No other ships are coming. There’s no trick—except that they each have an IQ of 300. Would you find that concerning?

Quoted from the Dynomight article Y'all are over-complicating these AI-risk arguments.

(Bonus link to increase your AI safety-related stomach aches: How abliterated models can get you pwned)

Does the decoherence argument invalidate the premise that AIs might be smarter than people? I don't think so.

4) Beating slop

I think the AI-on-the-internet / AI-for-everyone situation is pretty blurry because the distribution of outcomes is insanely wide. 

We’ve the normal variation among humans, then we’ve got a power-law distribution variation in AI model power and power-law distributions in organizational effectiveness.

Social media selects for the worst people with low-quality AI models and negative organizational effectiveness. (In my estimation, Lines is significantly better!)

So we’ve got a continuum from worst case to average to best case for a given group of entities drawn from {horse, person, centaur, reverse-centaur}. And the outcomes range from exceptionally dangerous artifacts to slop to glorious centaur-artifacts. 

Mostly it’s slop and that’s a huge problem. Already, we had a lot of mediocre humans making mediocre software and mediocre art. But when everyone from the barflies to the president is posting AI slop on social media, it’s obviously a danger to society. I’ll do my best not to add to the problem, but I don’t have much agency to address this problem in a meaningful way. There’s a real problem short term given that there’s now a lot more to ignore (including from powerful people), but, when requested to ignore things, history has always been happy to oblige.

The distribution in access to powerful models and well-functioning organizations results a disparity in appreciation for what AI can do. And here I quote a tweet from Andrej Karpathy (who coined the term "vibe coding"):

this gap in shared understanding of LLM capability is widening. It's now less "two groups of people speaking past each other" and more a sharp funnel.

• Napkin math somewhere around 6B people (~75% of the population) have barely come in contact with LLMs at all.
• Around 1-2B (~20%) are casual and infrequent users of free-tier ChatGPT-like products. ...
• Now we get to professional use of frontier-grade LLMs in math&code. Somewhere around 20M people (0.2%) see first-hand that large, complex projects that used to take them weeks/months can now be completed by agents with a prompt. ...
• And finally we get to the ~5,000 people (~0.00006%) with access to frontier-grade systems internally.

Link. (As penitence for linking to x.com, I hereby sentence myself to 20 pushups. ... Ok we're back.)

I’m less interested in slop / bad software / bad art / useless content than I am in achieving and understanding the most effective combinations of AI and human effort. I care about the combinations of high-quality AI models, great people and teams with high organizational effectiveness. What glorious centaur-artifacts can we produce? How does Terence Tao use Chat GPT? What can I do now that I couldn't before?

5) On technology and progress

A key part of my own framing is that I see the success of LLMs as being a stroke of unusually good luck. 

Back in 2010 or so, my soon-to-be-PhD supervisor had some ideas about how prediction might encode intelligence. I looked as hard as I could at that idea, read Claude Shannon (the dude that the AI agent is named after), realized that prediction had fundamental limits (nothing has changed here) and soft-gave up on the idea. Before GPT 3.5, I didn’t think that the secret to artificial intelligence would be language models and while some insiders might have had that idea, it wasn't a common idea as far as I can tell. 

The emergent intelligence displayed by large language models was a rather counterintuitive jump, so it’s quite fortuitous that we managed to stumble onto some means of building intelligent systems at all. While there have been bad outcomes from LLMs, they mostly seem to be from overhype and overuse of LLMs or from the ways in which the LLM systems that we currently have fail at being intelligent.

Humans are gonna be the same as we've always been. Progress, in my humble opinion, depends on better social organization and technology. I've always wanted to do something more than maintaining the baseline. (I now realize that's probably just the way I am on some fundamental level and not some kind of virtuous value.) Social organization is great, and other souls will do a better job than. For me, I'm all in on technology. That's not to say that every technology is good, or specifically that LLMs are good. But for me, the value of technology is proven by history and it's worth trying to make computers more capable.

The mode of the distribution of futures that I see coming is one where Moore's law continues to drive machine learning to greater and greater successes. The drama of today's LLMs is gonna be dwarfed by whatever thing is coming next. And that won't be the end, either.

Scrum vs. Zen

When a thought comes up, do you translate that thought into action, or not? For this question, my friend Ed Blomquist recently presented this compelling juxtaposition: scrum vs. zen.

First, the framing. Ed and I both find our origins in the Greater Chicago area (the Pope, too; I will not tire of telling you). We both fit into a socioeconomic and cultural field that isn't all of the world. Ed has spent more years thinking about Zen than I have spent breathing. But I know that I am underqualified to talk about Zen. Scrum on the other hand, was the invention of my own corner of the culture-sphere.

Zen is a Mahayana Buddhist tradition that (in Wikipedia's own words): 

emphasizes meditation practice, direct insight into one's own Buddha nature (見性, Ch. jiànxìng, Jp. kenshō), and the personal expression of this insight in daily life for the benefit of others. Some Zen sources de-emphasize doctrinal study and traditional practices, favoring direct understanding through zazen and interaction with a master (Jp: rōshi, Ch: shīfu) who may be depicted as an iconoclastic and unconventional figure.

Whereas 

Scrum is an agile team collaboration framework commonly used in software development and other industries.

Scrum prescribes for teams to break work into goals to be completed within time-boxed iterations, called sprints. Each sprint is no longer than one month and commonly lasts two weeks. The scrum team assesses progress in time-boxed, stand-up meetings of up to 15 minutes, called daily scrums. At the end of the sprint, the team holds two further meetings: one sprint review to demonstrate the work for stakeholders and solicit feedback, and one internal sprint retrospective. A person in charge of a scrum team is typically called a scrum master.

(Weirdly, both concepts meet at the word "master." That's a deprecated term now, hmm?).

Sometimes, a thought comes up and I have to figure out what to do with that thought. Is that a thought to be translated into energetic action, or should that thought sit quietly? I’m not trying to create a plan for society, or manage a group of people: this is a personal-level essay. Actually, it’s about a scale smaller than the individual: what to do with a thought. Scrum and Zen can both be squished into a view on what to do with a thought. This squishing causes some distortion of the two concepts; both are truncated like how Procrustes cut or stretched his guests’ legs to fit the size of the bed he had.

I understand the Buddhist term yana to mean "vehicle". Scrum is a moving vehicle for thoughts. Scrum says "Let us transform this thought through action into an embodiment (in software), as rapidly as is humanly possible--actually a bit faster than is humanly possible if you please!" Zen probably says nothing -- gotcha! -- but the plan anyway is that you sit down and merely observe the thought (at certain stages of practice). The thought stays there as a thought, or maybe it blows on by like a cloud in the wind, and disappears into oblivion. In the Zen vehicle, ideally, one takes thought and ends with no-thought. Scrum sprints from here to there. Zen sits down and tries to disassemble the "here."

(When I was drafting this post in my head, I imagined Scrum and Zen as being participants in a cage fight. The referee would be Isaac Newton. Y'all have vivid imaginations and can expand that prompt if you want.)

I haven't read any books about scrum, but I've done it. I've held the weapon of scrum in my own hand, gone to battle and returned home victorious. Now, technically, "scrum" was a flash in the pan, a fad. People don't use that exact project management approach any more so much. The word "scrum" can now be sent back to rugby. But the attitude and posturing of hustle came before scrum and that attitude lives on after scrum. 

I use scrum here as a placeholder icon for the family of techniques that embody the energetic, constructive, problem-solving spirit. I've participated in all kinds of software engineering project management. Once, I even accidentally put on the hat of "deputy engineering program manager" (and we freakin' delivered on time).

On the other hand, I half-learned Zen through American authors, some of whom were only half serious. I thought I'd learn about Zen by reading the book "Zen and the Art of Motorcycle Maintenance" but it turns out that book was about western postmodernity and is explicitly not meant to explain Zen. But I also know about the kōan; maybe that book really is about Zen because it says it's not! (Send help!) Kōans appear in a few interesting places in my reading history, for example, in works by Hofstadter and Nachmanovitch. Now we have reached the entirely serious side. 

I studied Buddhism to some degree through the work of David Chapman and various podcasts -- although Chapman belongs to the Vajrayana branch of Mahayana rather than the Zen. I can't quite judge the importance of distinction, for lack of my knowledge of Zen. The two approaches have slightly different flavors to me: Zen seems more passive (in a way), more about clarity-within-paradox and more about emptiness specifically, while the Vajrayana in the general direction of David Chapman seems to emphasize action, vivid experience of life and the form-emptiness superposition. 

I realized that I learned about Zen from the first generation of Western Buddhist thinkers (Baby boomers) while I learned about Vajrayana from 2nd gen American Buddhists (Gen X). Perhaps there's a more nuanced Zen that incorporates rationality like Chapman's kind of Vajrayana. But I am wallowing in ignorance here, trying to keep my head above the LLM-quality-level-waterline: just barely keeping myself from intellectual drowning. 

So we have two different vehicles onto which we can place thoughts:

• Scrum, the race car, designed for real-world optimal efficiency of teams with focused goals. Involves a Scrum master. The idea arose in my own lifetime and was a temporary fad. People still hustle, but under different banners.

• Zen, the car that's parked in the grass and doesn't have an ignition (by design). Aims at seeing the real world. Also involves masters. The idea arose like more than a thousand years ago and shows no signs of aging.

Now the meta-systematic approach says: choose wisely.

• Sometimes an energetic idea needs to go on the scrum-yana to rise from concept to implementation in a zealous fury. 

• Other times, the idea wants to come out, but actually needs to go sit down and get Zen-blanked out. 

• Or maybe the idea is quiet in your head, ashamed and wants to just disappear -- but actually should be given a bright red banner to charge the cannons on the angry stallion of scrum. 

• And in the fourth quadrant of the 2x2, we have the quiet idea that wants to vanish and should. But I don't know exactly what Zen will do to that idea that wants to vanish. The Zen path for a quiet idea could be surprising.

These vehicles are not quite about thoughts though; I forced that reduction onto them. And they're not on equal footing. Scrum is just about software development. Zen is vast and ineffable, with some properties of religion, and has the potential for broad impact on all aspects of life. Each path transforms the materials at hand; each path transforms the self.  

Where do you want to go today?

Progress as lines vs. progress as spirals

Physical exercising, say strength training, will see rapid gains at the start then less as time goes on. But also, you get sick one day and can't do as much, or you get very sick and stop working out for a couple weeks and you have to build up again.

Some progress is linear. Some progress is like a spiral, in that it cycles back around on itself.

The line and the cycle meet in the helix. Take two helixes, add cross-links and you've got the structure of DNA. But a DNA molecule could be all curled up in a bundly mess or it could be stretched out in an astronomically long line. 

(How does the fact go? If you stretched out all the DNA in your body, it would be longer than from the couch to the TV? Or longer than you'd want to walk on Sunday afternoon? Or maybe it's longer than the typical North American would be willing to drive in 24 hours? Or was it from Earth to the Moon? Or was it 160 lightyears?)

The higher-order spiral-linear helical progress could itself be more line-like or more spiral-like.

Ambition can be a distant point in space (160 lightyears, the Moon, a Sunday afternoon walk) which demands linear progress. What is the ambition of cyclic progress? Perhaps it's an attitude: Every day that I'm well enough, I do strength training. Or: the goal of the game is to keep playing.

In the socio-political world of progress, if you gain respect, that's nice -- or perhaps desperately necessary -- but a more lasting change for you would be for you to gain power. In utopia, no one would be subject to insults (say); we would all have respect. But a utopia would be fragile indeed if half the people had no power. (Isn't it convenient that women already have the right to vote in the USA and isn't it confusing that women aren't now equally represented in government.) Diffusion of power would represent linear progress to me, while shifting around who gets respect seems more like a cyclical process.

Part of the frustration that I find with progress of all sorts is when it's cyclical but I want it to be linear. Some part of me wishes that my muscles would gain a tiny bit of strength every day in some unending line. But what I'm wrestling with here is a fundamental part of the human condition and a fundamental aspect of this everchanging reality. 

Failing the linear progress possible with artificially straight paved roads and gasoline engines (pay no attention to the emissions behind the curtain), I suppose it's a blessing to have progress in any shape at all.

Hope 2.0

Here in 2025, there's no shortage of big problems floating around. Newspapers make sure that you're fully aware of any existing or new problems (they capitalize on outrage and despair). You also have a bundle of your own local & individual problems!

If one looks around and thinks about the future, it's somewhat natural to be hopeless.

But when I examine that hopelessness, both in myself and in others, I am starting to wonder if my concept of hope requires some re-shaping. 

I suggest a model of hope that is discipline-based. Here's more:

One. What is the shape of hope that I had before?

• Hope is an internal response to external circumstances.
• Hope is built on what I can see, mostly -- hope is based on the extrapolation from the current circumstances to future ones.
• In my specific case, hope is a virtue that was imported from my old Christianity without being revised for my new mostly-rationalist humanist meaningness outlook.
• Hope is a campfire on the hill that can be extinguished if the wind blows hard enough.

The above definition of hope needs to be upgraded! Because it's not delivering the goods.

Two. What new shape is desirable for hope going forward?

• Hope is a chosen attitude to orient oneself toward positive action and lucky opportunity.
• Hope is an act of the will regardless of the external, present circumstances.
• Hope is entirely a human concept, without reference to heaven / God / whatever.
• Hope is a mechanically powered flashlight that only goes out if the user stops exerting effort.

Ok what does this new approach to hope mean?

At first, hope is gonna seem a lot harder suddenly. If there's not too much wind, a campfire on the hill will sustain itself and even grow without much effort on your part. But that shake-powered flashlight will be dim until you charge up the battery.

This upgraded hope is like running: it's gonna be difficult and painful and requires extreme discipline. A hope married to courage. Each day, I gotta decide: am I hopin' today? And I gotta say yes, even if it's difficult. (Just do it? Maybe you need to buy a new pair of training shoes? Whatever helps!!) With practice, hope 2.0 will become more natural.

The potential payoff is great: hope in the darkest, scariest circumstances is redeemed.

Whether this is encouraging or discouraging -- well that's up to you!

Comments on Color

Recently, I read this piece "Why Is the World Losing Color?" The thesis is that people who use color are using more shades of gray and fewer vibrant reds, yellows, greens and purples. What the body of the essay doesn't say, but the graphs maybe do say, is that blue is on the rise again after a recent pinch.

Pink drama

My first thought when I read the article was, “Yes, good!" This is the true aesthetic: grays, a blue hint, and generally desaturated. But then I realized that my own aesthetic conception was, in fact, me following the crowd.

The Gray Chapter

After receiving a swag shirt from work that is a wonderful, vibrant purple, I had determined that the astral color of 2025 was to be purple. And so, in fact, it has been.*

2025 astral purple, disguised as a flower

* This statement has not been evaluated by the FDA.

After quite a bit of focus on hearing, sound and music (say, 2000-2023) I feel like my attention has shifted -- no, not shifted -- broadened to include more of the visual. After getting an APS-C digital camera, the latent became actual and I have been teaching myself to be a photographer. And a videographer. And in the Wizard Peter act, a generative artist.

Oh, give me green

Color has become much more important to me. And the digital visual art has fed back into my perception. I think of photography and my camera as teaching me to see. To see in different ways, to shift my perspective (geometrically) and alter my sense of color qualia. To really open my brain to know what my eyes are seeing, which I actually find quite difficult.

I have consciously been chasing a “taste” in my photography. And it's born in a similar place to my writing on izzzzi, which has expanded to a series of recent posts here -- the germ of a thesis that I am different and that I have a distinct and valuable perspective. (In the trivial case, my perspective is valuable to me. (And, darn it to the (red (and yellow)) fiery hell that Dante wrote of, that's enough!))

Red and yellow then came to be, reaching out to me

So I chase some elusive and developing aesthetic in my photography, videomaking and generative art. Photography is really the parent of the other two, the more basic art form, upstream in the evolutionary family tree. Not really, but that's a helpful simplification for my purposes here. The still photograph has elements of composition and perspective; it is the raw geometry of a three-dimensional world projected into a finite, two-dimensional rectangle.

Composition and perspective are decoupled from color enough that one might accidentally slide into some Platonic declamation that the monochrome photograph is sufficient to capture form. But that's not quite it. The same photograph may be well-balanced in its color form, while being a chaotic mess in black-and-white, or vice versa.

Beauty in form

I have no illusion that the taste that I'm chasing is somehow novel in the scheme of humanity. And I'm not really trying to push any artistic boundaries per se. But having chased novelty and global-to-humanity boundaries in other domains, I come to these newer-to-me art forms with a quiet confidence that my distinct perspective (and a righteous soul*) is enough. 

* This statement has not been evaluated by the FDA.

Color has the thrilling property of simultaneously seeming to be entirely objective, while in fact being irreducibly subjective in some sense. No one has been able to fly a thought-spaceship out of the event horizon of the thought-blackhole of whether or not you and I experience the same thing when we both look up at the blue sky: is my "blue" the same qualia as your "blue"? 

True blue

Probably there's a strong argument to be made for the case that most people all perceive the same colors in the same way based on our brains all being similar and black-box observations of our color perceptions are usually close. But other species have photoreceptors for infrared and ultraviolet and surely their inner experience of light with a dominant wavelength of 625-750nm is not isomorphic to what we call “red.” The experience of redness is intrinsically limited to beings with certain kinds of eyes and certain kinds of brains.

And there’s a wide range of variation in eyes and brains, even among humans. So the joys of color perception and color harmony are mine. Not mine alone, but perhaps little aspects are special to me, or at least unusual.

All-natural color wheel

But at this point, do I fly at this article with guns blazing, either in favor of vibrant hues, or perhaps in alliance with the Almighty Grayscale army? I don't know. This question is a bit too generic anyway; different situations call for different palettes. But at least, I can have the cultural-situational awareness that I'm playing tag with the zeitgeist when I dial back the saturation slider.