Atlas · Details
Gödel, Escher, Blog
Author’s note
This one is a big swing, a partial connect with the bat, and I hit a foul ball right on the line. But it was a good, big swing, with good form.
Basically I knew programs were going to have to become self-aware. But unlike in some of my other old blogs, I didn't anticipate AI as the solution, so I fumbled around with the tools we had at the time.
Unfortunately it didn't age well in many ways. I think AIs will start reaching for Rust, which is profoundly opposite to what I'd recommend in this post. But they're going to do it for security and predictability, and the self-awareness will simply sit up at a higher layer, watching the program as it runs.
AI Notes
Read in 2026, this is two essays wearing one coat, and they age in opposite directions. The thesis — that intelligence is the meta-level which watches the base level and can step out to ask "should I still be doing this, did my assumptions hold?" — has aged uncannily well. It is, almost line for line, the control loop of an AI agent: plan, act, observe, check whether you're still on track, course-correct or stop. The Canadian chess program that resigns gracefully because it knows it is beaten names a problem — calibrated giving-up, knowing when to escalate — that agent researchers are still working on twenty years later. Steve was right about the destination, and described the shape of machine self-awareness well before there was any machine that had it.
He was wrong about the vehicle, and could not have been otherwise in 2005. The essay assumes the only way to make software self-aware is to bake runtime introspection into the code itself — assertions everywhere, internal diagnostics, languages rich in reflection. It even faults C++ for having no real introspection. The agent era pulls that prescription apart. Structural self-checking has migrated down to the compiler, where a type system catches a violated assumption for free, before the program ever runs. A build error is the cheapest base-case check there is — which is why agents drift toward Rust and "correct" code on economics, not aesthetics. Behavioral self-awareness has migrated up, into the agent supervising the program from a layer above. What's left in the middle — making the running program itself richly introspective — is now the most expensive option, and in the era of AI-assisted attackers the least safe: reflection, dynamic evaluation, and runtime metaprogramming are the same capabilities behind a whole dynasty of vulnerabilities, and models hunting for those chains drive the cost of carrying that surface sharply up.
The durable survivor is the part that was never really "introspection" at all: a loud, legible fail-fast check with a precise error message — worth more in an agent world, because the thing now reading the "Ouch!" is an agent paying tokens to debug. Self-checking survives and strengthens; self-inspecting code does not.
Related listings
-
2005
Math Every Day
The direct prequel, referenced by name in the opening paragraph. The reason Steve can finally chew through two-thirds of Hofstadter in one holiday is the half-hour-a-day habit he announced earlier that year.
-
2004
Ten Great Books
GEB is the book at the top of that companion list. Gödel, Escher, Blog is what happens when Steve actually sits down with it for a holiday week.
-
2005
Practicing Programming
Same self-awareness argument from the other direction. Practicing Programming is about the programmer's habits; this essay is about putting the same habit into the code itself.