I tried to stop being the bottleneck in my own software projects. Here's what two weeks of building an AI agent orchestra taught me. https://x.com/i/article/2120594145962053632
Building an Orchestra with Agents
The Idea
One fine day, I decided to try Grok Bot. And, to my surprise, I got completely hooked. The tool is simply too good. Easy to use, proactive, and extremely well built for what it sets out to do. It was pretty much love at first prompt.
And it was during that honeymoon that an idea started taking shape in my head: what if I automated the entire software development process? What if I stopped being the bottleneck?
The idea seemed too good not to explore. But, for all the excitement, Grok Bot still had a few limitations that bothered me quite a bit. The agents burned through way too many tokens. Organizing multiple workstreams was a pain, unless you were willing to create 10921092 different bots. The agents asked for permission all the time. And, to top it off, you couldn't freely choose which models and subscriptions to use.
That last one, in particular, bothered me the most. For anyone who likes surfing the subsidized tokens of different subscriptions and trying different models for each kind of task, losing that freedom is almost a crime.
So I decided to combine two things: my new addiction to autonomous agents and my endless curiosity about how far software development automation could go.
That's how Orquestra was born.
Orquestra is a bot responsible for (who would have guessed) orchestrating my Codex and Claude Code subscriptions to solve software development problems with as little human intervention as possible. The pitch was fairly simple: I would write a project brief, define what needed to be built, and Orquestra would take care of everything else.
Or at least, that's what I hoped.
As you can probably imagine, reality wasn't that simple.
I'm not going to say I built something capable of "replacing an entire engineering team," let alone show you miraculous productivity charts. I didn't measure anything rigorously enough to back that kind of claim, and that was never the goal of the experiment. What I want to share here is the process: what worked, what broke, which design decisions made a difference and, above all, what I would do again. In fact, a good chunk of what works today only exists because something went very wrong before.
In the end, Orquestra turned out to be much more than an automation experiment. It was a deep dive into harness engineering, agent architecture, team topology, and AI applied to software development. And the lessons from that journey are exactly what I want to talk about in this article.
Orquestra
If you've never used it: Grok Bot is an AI assistant you talk to over chat. The difference is that it has its own Linux computer, which just sits there, persistent, with a terminal, files, a desktop, and a browser. It installs and runs real things on that computer.
For Orquestra, four things made all the difference:
- Its own computer. Claude Code and Codex live there, logged in to my subscriptions.
- Routines and webhooks. A routine is a prompt that runs without me, on a schedule or when an event arrives. A webhook is how an outside program triggers that routine. That's how Orquestra gets woken up when something happens.
- Memory and skills. The bot remembers things across conversations and has skills, which are packaged instructions for repeated tasks.
- Several bots talking to each other. Each bot has its own job and persona, and they message each other. Orquestra is one of my bots. In fact, it was another one of my bots that built the first version of Vigia and handed the instructions over to Orquestra.
So that's Orquestra: a Grok Bot that acts like a seasoned software engineer. It plans, defines the architecture, has tests written, implemented, and reviewed, and ships. It's a generalist, not built for a single project.
After a lot of trial and error, this was the end result:
Lesson 1: Not everything needs to be an LLM
On day one, I asked Claude for an architecture diagram. After a while I asked: "is claude running?". It wasn't. The run had died instantly because of an argument-order error in the command, and Orquestra had no way of knowing.
The obvious fix is to have the coordinator check in every so often. Except every check-in is a big model, with a big context, burning tokens to find out nothing changed. That idea quickly became a problem…
That's how Vigia ("Watchman") was born. It's a Python program, standard library only, rule-based. Zero LLM. Every Claude Code or Codex run goes through it, and it looks at objective signals:
- inactivity: by default, 10 minutes with no new bytes and no CPU usage;
- max time: by default, 20 minutes of wall-clock time;
- loops: the same error or the same action repeated, or the scope growing beyond what was agreed;
- and, if I turn it on, test protection: the implementing agent can't edit test files.
When a limit is hit, Vigia saves the evidence, kills the process, checks that it actually died, and sends a webhook to Orquestra. Only then does the LLM come in, reading what happened and deciding whether to continue, fix, or replan.
Lesson 2: One bot to rule them all
Before Orquestra I tried the org-chart route: one bot playing CTO and a "factory" of specialized bots. When I built Orquestra, with a single coordinator that calls the coding agents on demand, I found the setup much better.
Today the team topology depends on the size of the task, which Orquestra classifies as S, M, or L:
- S (small): one dev and one reviewer. That's it.
- M and L (medium and large): a fleet with a team lead, a dev, a reviewer, and a security agent.
Each fleet is made up of Claude Code and Codex playing the roles of whatever workers are needed.
Lesson 3: JP does nothing
This one is simple to explain: I don't approve anything. Orquestra is in charge. It commits, pushes, opens PRs, merges, and deploys on its own. At the end of each task it does a light check: it verifies the acceptance criteria and reads the diff.
Instead of asking for permission, I defined lines it never crosses:
- it never spends money;
- it never sends messages to the outside world;
- it never force-pushes to main;
- it never deletes anything it didn't create itself.
It only calls me when there's a problem or when it's done.
Lesson 4: We like subsidies
Orquestra doesn't use a fixed model. It drives my Claude Code and Codex subscriptions and picks the model for each role:
- the team lead is always the most capable model I have (Fable or Astra);
- the other roles get whatever model the task's complexity calls for;
- review is cross-review: Codex reviews what Claude did and vice versa.
I recommend that last one to everybody: different models make mistakes in different ways, and they almost always get into arguments over who's right.
Lesson 5: Bots don't like autonomy
Even with me reinforcing its autonomy, Orquestra kept stopping. I even asked another bot to investigate why it kept stopping all the time. The diagnosis had two parts: its instructions and a few missing safeguards in the backend.
The first part is the most interesting one. There was a 45-minute limit for small tasks, technical and legitimate. But the idea that this deadline "couldn't be renewed" wasn't written anywhere. Orquestra itself invented that rule in its plan and started obeying it. And when a plan came back BLOCKED, it treated it as if the entire mandate was over, not just that plan.
Nobody told it to stop. It created limits that didn't exist and trapped itself in them. Kind of like you being afraid of becoming senior at work.
The fix was to write explicit continuation rules. A turn only ends in four situations:
1. an accepted delivery;
1. a worker confirmed as active and working;
1. a pause I asked for;
1. a real blocker, with evidence.
The harness lesson here: telling the agent what it can do isn't enough. You have to tell it when it should keep going. Otherwise it fills the silence with limits it made up itself.
Lesson 6: Self-improvement
Finally, Orquestra has two improvement loops.
The first one is per project. Each project has a aprendizados.md (lessons learned) file with test commands, pitfalls, and conventions, handed to that project's devs.
The second is generic and improves Orquestra itself: the fleet skill, the instructions, the model choices. If the same lesson shows up in two projects, it gets promoted to the generic level.
At first I thought about doing a retrospective after every run. Terrible idea: every failure becomes a new rule, the rules fight each other, and the instructions get bloated. Today a routine runs every two days, looks at all the runs together, and applies at most three rules per project. Orquestra approves them itself, everything is logged, and it can all be reverted. The same routine also cleans up the skills.
And to know whether a new rule actually fixes things, the obvious move would be to reproduce the failure. But that means running an entire task again for every rule, which means more tokens (them again). So I built a cheap judge. It reads the failed case and decides, in a single call, whether the rule fixes it without breaking what had been working. It remembers previous decisions so it doesn't get stuck in a loop. It's not infallible, but since everything is reversible, a mistake on its part is cheap.
Final Design
My hyperfocus on this lasted two weeks, and I had a lot of fun building this little guy and learning more about harnesses. I tested plenty of other alternatives, which aren't worth including in this post, and ended up deciding to build my own solution. That's the beauty of vibe coding: if you don't like something, just go build your own solution. It probably won't be better, but at least it's yours.
The final design looks like this:
P.S.: after SpaceXSI decided to swap Grok's model and obliterate the quotas even further, all of this went down the drain :D