Programming Parables: Gell-Mann Amnesia and Agentic Development

Alternative titles: The Agent Was Certain; Gell-Mann Amnesia for AI; Knowing Enough to Push Back; Domain Knowledge Is Still the Job; The SQLite Test

I. LLaMnesia

Gell-Mann amnesia refers to the concept of reading a newspaper and, when it is on a topic that you have deep knowledge about, finding significant errors in the reporting. Just obvious inaccuracies.

And then you flip the page of the newspaper and start reading about a topic that you don't have deep knowledge of, and you simply believe every word that's reported.

You have this immediate amnesia of the fact that this newspaper is not something to be trusted as soon as you flip the page.

Of course, this relates very heavily to AI tool use in general.

LLMs are very spiky. They can easily make mistakes, but they have been largely tuned to sound authoritative and useful. And so it's very easy to be taken by them, especially when they do things in a way that appears to be thorough, reasonable, and proficient.

A bit of an example that I ran into the other day.

II. Scaffolding is all you need

I was replacing a very simple logging utility that we had just for local work. It was a standard kind of append-only JSONL implementation.

This had worked well for a long time, but suddenly we needed to have guarantees around concurrency. We had an issue where multiple writers tried to append into that file, and it led to some data loss.

Now, this is just a small local utility, so it's not catastrophic, and it was a known limitation of this tool. But we figured it was finally time to actually upgrade it into something a little bit more reasonable.

So I opened up a chat with an agent and began to develop a plan.

Immediately, when it was apprised of the situation, it wanted to implement a solution that would include lock file handling, atomic file writes through renaming and move operations, and some sort of queue for processes to write into that gets drained into the file.

The point being, it immediately wanted to solve the problem in the shape of the problem rather than finding a solution.

One thing I talk about a lot is this concept of remove before you add, and it was doing exactly what agents love to do when they run into an issue: it builds and builds and builds, scaffolding around the problem rather than finding the simple route to just avoid the problem in the first place.

So, of course, I recommended that we just switch to using SQLite.

It agreed on the surface, but decided that first it needed to do a probing spike test of this new implementation.

And so it wrote itself a little test suite. It ran it. It looked incredibly professional and convincing.

And it came back with the result that this wouldn't work for our purpose.

When it tried to do concurrent writes into the SQLite database file, it led to significant irreversible data loss. And so we needed to stop even considering this solution and instead start developing those gates and checks and balances around this JSON file that already existed.

Now, I was there driving the conversation, so I pushed back because this didn't smell right to me.

I nudged it to do some web research, check best practices around configurations and modern pragmas for SQLite, and take another stab at this test.

And it did.

And it said, essentially, Heavens, I found a glaring flaw in my original probing test. I've now fixed the bug and I'm rerunning the test.

Again, it looked very professional and thorough and highly convincing.

And then it came back saying: I fixed the bug, but the issue remains. This solution is untenable. SQLite will not work for this purpose. We must instead implement this more advanced and holistic solution by building scaffolding upon scaffolding around this JSON file.

At this point I was certain that the agent just didn't know what it was doing.

Arguably, I should have just reset the context and tried again with a fresh agent. But I pushed on.

And this time it came back saying, Heavens, I've found even further glaring bugs and misuse of SQLite in my spike test. I've now resolved them. I'm rerunning the test.

And, of course, as anybody who's been a programmer for probably a week would be able to guess, the test came back saying it was validated.

Yes, you could have many concurrent processes contending to write to one SQLite file, and it worked just fine.

Of course, whether SQLite is an ideal solution for every kind of concurrent workload is another question, but that's for another time.

III. Trust yourself

The point here is that the agent was absolutely positive that this solution was untenable.

And not only was it positive, but it also produced extremely thorough and convincing, seemingly empirical evidence that this solution was untenable.

Without domain-specific knowledge helping you to steer this conversation, it's very realistic, and even reasonable, that somebody would have just gone along with the agent's architectural recommendations.

And so hopefully this serves as a reminder to myself as well as anybody else that we need to rely upon our own judgment of things.

We also need to be able to understand where our gaps are.

We shouldn't be taken in by authoritative-sounding evidence. Instead, we need to take the time to learn what we need to learn in order to actually validate these things for ourselves.