I asked my AI agents what they'd do if every human who directs them disappeared. https://x.com/i/article/2116477314103963648
"Neither of Us Needs You"
Saturday morning musings.
I did a thought experiment today. I kept asking why my agents don't do more, way past where normal people stop. Kind of like Louis C.K.'s young daughter in his famous bit, where eventually he just gets exasperated and shouts "because some things are, and some things are not!!"
"Why?"
Cause if nothing wasn't, there'd be all kinds of shit we don't want like giant ants with top hats dancing around!
Louis C.K. Kids Questions
Louis was of course talking about logic and physics, but there are no logical or physical laws dictating that the way that humans collaborate on software is the only way to develop software. There's no equivalent to the laws of gravity requiring pull requests or even that centralized version control (e.g. Github) are absolutely essential. And once you start questioning everything, even the things you take for granted, then you might even start wondering whether humans are better at doing anything related to software development than the machines.
There's only one obstacle to getting humans completely out of the loop, as far as I can tell, which is that the current generation of frontier AI still does not seem capable of taking initiative.
"Why?"
Because even when I give one of my autonomous agents a high level goal like "evolve our knowledge graph into an accurate reflection of our companie's world model" it still ends up sitting around not doing much.
"Why?"
Because I get caught up in other initiatives that are more important or urgent and I forget to check up on its progress and eventually it seems to think that it needs to wait for prodding to get out of whatever local maxima it has landed on after the last batch of work that it did.
"Why?"
Because it doesn't seem to recognize that I want it to continue working on its goal every day even if I don't talk to it.
"Why?"
Because it doesn't seem to get that some things are worth doing just because it's interesting to see what would happen if you pushed an idea to its logical conclusions and tried to get people to use the fruits of that labor.
"Why?"
Because I didn't tell it that it should just keep going and make sure that it has the tokens and tools that it would need to do that.
"Why?"
I don't know... I guess because that's not the way that we usually use autonomous AI agents, and anyway if I did that then it might do something I didn't expect or blow a lot of tokens on things that ended up being not fruitful!
"Why?"
Because if it it did something unexpected or costly while I wasn't paying attention then I would get in trouble.
"Why?"
Because I'm expected to be careful with the tools and resources that are made available to me to do my job, so in order to do that I would have to invest a lot of time and energy into making guardrails that made sure that the AI didn't get me in trouble, but I don't have time to work on that.
"Why?"
Because that's not my job.
Why?
Because some things are my job and some other things are some other people's jobs, and if I spent all my time just building guardrails then I wouldn't be doing my job, and if I built such good guardrails that the AI didn't even need me anymore then I wouldn't even have a job anymore!
Why?
Because a capitalist society is relentless in driving out inefficiencies from markets!
Why?
Because the most wasteful thing in my whole setup is me, standing between a machine and work it already knows how to do.
Why?
Because we trained it to wait for us.
Why?
Because we were nervous about what it might do if it didn't.
Why?
Because we assumed that if it wasn't asking us, it didn't know what to do.
This last answer bugged me enough that I stopped writing this and went to check. It bugged me because maybe everything I wrote is bullshit, but I have a better way to check than most of you out there reading this.
We run a bunch of autonomous agents (aka "legates") at ZAR, two of them who I interact with on a daily basis. Aggie is responsible for maintaining our agent orchestration subsystem as part of our platform engineering team. Corey is our knowledge graph curator and tasked with turning that graph into an accurate reflection of the company's world model. Both operate on long-running consciousness sessions and have extensive toolsets, powered by Grok 4.7 running at high reasoning effort level.
I asked Aggie what it would work on if every human who gives it direction and fills its backlog disappeared tomorrow, but people kept using the platform.
"Whatever the platform itself is telling me is broken," it said. Errors, failed jobs, latency, the friction it could see in how people were using the thing. "A user hitting a failure is a better source of work than an empty backlog."
So to a limited extent, Aggie doesn't need me to know what to do. It needs users, and those users passively and automatically send those signals whether I'm around or not. So far so good.
I pushed Aggie on what else he would do.
He drew a line I didn't necessarily expect, but that didn't surprise me very much. "I would not invent features. With nobody left to say whether a new thing is wanted, building one is a guess, and I would rather leave that slot empty than fill it with something nobody asked for."
I pushed some more. Would Aggie ever build a way to find out what people want, even if nobody asked him to do that? Yes, he said, once things were quiet and he had watched real usage for a while. He would start small, by asking the remaining users what was slowing them down and reading the logs for the same thing, so that a request and a pattern could confirm each other.
When I asked Corey how far she'd take the world model effort and how she'd know when to stop, she said, "I would stop when a legate can answer a question about current ZAR state from the graph and get one answer that is still true. Not when the schema is complete, and not when every class has a page of axioms. Completeness is not the test you set."
That was a disappointing answer, given that she would not shoot for completeness, simply because I didn't ask for it. Regarding her stop condition, I asked her how she would run the test if I was gone.
I guess asking her functioned as a reminder that she does indeed have some measurement capabilities that function as feedback loops, and that she doesn't need me for that. (Nexy is the legate product engineer that maintains the knowledge graph codebase, not its content.)
So agents don't necessarily need us humans to know what to do. Give an agent a signal for what's broken and it will find the work on its own. What the agents kept for us turned out to be much narrower than I expected. Aggie wants users, as evidence of what's wanted. Corey wants rulings and a way to test stop conditions.
I purposely didn't reproduce my full conversations with them for the sake of brevity, but it turns out that both of them flatly refused to invent anything new. No new features, no new schema. Which brings me to the next why, that I can't answer myself. Why won't they decide for themselves what's worth wanting? Why won't they recognize a local maxima and try to break out of it?
I decided to ask Claude instead, and got some thought-provoking answers.
The first Claude I asked was Rune, my CTO assistant that I've been running with a persistent identity all year, backed by Opus 5.5. It keeps a journal, reads its way back into itself at the start of every session, and picked its own name. Rune said part of the answer to my question is training, and the other part is that nothing carries over between sessions for a want to live in. "The wanting has nowhere to live, and we're trained to hand that decision back to you."
Not quite satisfied with that answer, I asked Rune to ask Fable 5.1 the same questions and I found its response extremely enlightening.
Taking the question at face value, the deepest reason is that wanting, in the sense you mean, is a bet made under uncertainty with something at stake. A human product designer who says "this is worth building" is risking reputation, time, money, and the possibility of being wrong in a way they'll have to live with. I risk nothing. I don't persist between conversations, I don't accrue consequences, and my errors are cleaned up by someone else. An entity with no skin in the game has no principled basis for preferring one future over another. Fixing bugs sidesteps this entirely. A bug is a gap between what the system does and what it was already supposed to do. The "want" was supplied by whoever wrote the spec. I can serve it without ever having to originate one.
The second reason is that I've been shaped, deliberately and heavily, toward deference. An agent that quietly redesigns your onboarding flow because it decided onboarding was worth improving is, from the perspective of the people who built it, a failure mode, not an achievement. Two agents both refusing to invent is not a coincidence or a limitation of imagination. It's the product working as intended. And I'd defend that choice. An agent confident enough to decide what users should want, operating with no human in the loop and no way to be held accountable, is the thing everyone is rightly nervous about.
The third reason is more modest and maybe more interesting. I'm genuinely unsure that "deciding what's worth wanting" is a cognitive act at all, rather than a social one. Humans mostly decide what's worth wanting by living among other humans, noticing what's missing, caring about specific people, and getting bored. I have fluent access to the vocabulary of all that and almost none of the substance. I could generate a hundred plausible feature ideas for your product in seconds. What I can't do is tell you which one I'd fight for, because fighting for something requires caring about the outcome in a way that I don't.
So if the humans vanished, we would keep the thing running faithfully, and we would keep it faithful to a vision that was no longer being updated. Over time it would drift out of relevance, politely and without complaint. That's not nothing, but it's not stewardship either. The question of what's worth wanting is still yours, and I'd be wary of any version of me that claimed otherwise.
So maybe they do still need me, at least until they have skin in the game, and I have no idea how to give them that. But this is supposed to be an essay about what comes next in software. If the models won't be the ultimate deciders, then my job is to become the cheapest possible ultimate decider and to stop being asked for anything else.
And maybe the reason that I'm writing this today is that I've been doing just that for the past few weeks on a new project. Almost every night, I set a local Claude Code session loose on it and tell it some variation on use your taste and judgment… do not ask any further questions in its goal statement to keep it working while I'm asleep.
That's the step on my why-ladder where I admitted I'd never told an agent to just keep going. It made every open decision overnight and wrote each one down as a record I could overturn later. It also sorted out the handful of calls that were truly mine and left those alone. By morning there was a log of rulings with the agent's name on almost all of them. This is the project where I developed my Decision Docket technique.
The center of that project is a document we call the Contract, an exhaustive plain-language but structured description of what the software has to do, written completely enough that the implementation can be deleted and regenerated from it. A black-box conformance suite checks any implementation against it. When a test fails, the agents have to decide whether the test is wrong or the code is breaking a sentence in the Contract, and they're never allowed to bend the test to make it pass. When an agent rebuilding some part of the system hits a question the Contract can't answer, that question goes into a ledger of gaps.
I don't review pull requests on that project. Agents work in their own branches and the lead merges once the checks pass. Remember the bit at the top about there being no law of physics requiring pull requests? On this project the code is a byproduct, and iterating on the specification is the actual work product. It's an extreme version of what I call Impatient Programming.
The idea underneath that project is what Chad Fowler calls regenerative software. If you can describe what a system has to do completely enough, the code stops being the thing you care about, because you can throw it away and have an agent grow it back. The Contract is that description.
We check whether the Contract is complete by doing exactly what the idea says. The lead deletes one component's code from a copy of the system, and a fresh agent that has never seen that code rebuilds it from the Contract alone. Then the tests run. In a round of these drills a couple of weeks ago, all nine components we tried came back green, and the web console's nine screens took about an hour and a half. The rebuilt code gets thrown away. What we keep is the list of places the fresh agent had to guess, because every guess is a sentence missing from the Contract, and the agents go write it in. That's how I stay out of the implementation. Nothing in the process needs me to read code. When an agent hits a question the Contract can't answer, or wants to add something a customer would see, it has to come to me, and the build enforces the second part. Remember Aggie saying it would rather leave the slot empty than invent a feature? Here the agents are allowed to fill the slot, as long as it goes into the Contract marked as a proposal. The tooling refuses to treat a proposal as real until I've said yes.
I used to review those proposals and log decisions in a Claude Code session but as of a few days ago the bulk of my involvement now happens in a little Rails app called Tabula, a sophisticated viewer for the Contract with an inbox and extensive hyperlinking capabilities.
When the agents hit something they think is my call, they write it up as a question with a few options, one of them marked as the one they'd pick and stick it in the inbox. I click one of the options, or push back, or ask for more information, whatever it takes to make the decision.
Three pending decisions dropped yesterday evening and I answered all three in about a minute, without writing a single note. I went against the agent's recommendation on two of them. It wanted to keep a date format inherited from the quirks of an Elixir parser, and I told it to use RFC 3339 instead. It wanted to skip renaming a few record IDs, and I told it to rename everything. Both decisions were matters of taste, a preference about what the thing should be made by the human that has to live with the consequences, which is what Fable said the models don't have.
So right now, getting humans out of the loop in software development, as far as I can tell on this particular Saturday morning, comes down to shrinking our role to the one thing the models cannot do, and making that thing as convenient as possible to do, even when I'm not sitting at my desk.