Skip to content
View original post on X: Minglei
Published
40/100

Claude Code fixes 13 of 18 bugs when given evidence, not just error messages

AIA test by an independent developer found Claude Code fixed the root cause in 6 of 18 runs given only an error message, rising to 13 of 18 when given evidence showing where the bad value came from. The author describes the experiment as small and not a universal benchmark, and argues that agents often add guards that make errors disappear rather than fixing the underlying bug.
MingleiVerified on X
@rae1101x
View on X

Everyone's trying to make AI agents smarter.

My agent went from 6/18 to 13/18 real bug fixes without changing the model.

The missing piece wasn't intelligence. https://x.com/i/article/2114939319656087552

How to fix your AI agent in 1 prompt

The problem isn't always the model. It's what the agent thinks success looks like.

Claude Code fixed the crash.

The customer still didn't get a receipt.

The agent found a null pointer error, added a check, and made the exception go away. A perfectly reasonable fix.

Except the actual bug was three files away.

I tested this with 18 runs. Given only the error message, Claude Code fixed the root cause 6 times. Given more evidence showing where the bad value came from, it fixed it 13 times.

Same coding agent. Same bug. Different evidence.

It's a small experiment, not a universal benchmark. But it made me rethink something.

We spend enormous amounts of time trying to make AI agents smarter: better models, longer prompts, more elaborate instructions, whole teams of agents supervising other agents.

And yet an agent can write perfectly good code and still make the problem worse.

What if intelligence isn't the part that's missing?

What if we've been teaching agents to win the wrong game?

Below I'll explain why this happens and why your instructions probably aren't solving it, then give you one prompt you can try today.

To see why the prompt works, you first need to see what your agent is actually doing.

I. Your agent isn't broken. It's doing exactly what you asked.

Imagine an online store crashes while sending a receipt.

The error says:

'NoneType' object has no attribute 'name'

Your agent investigates, finds the line, and adds:

> if customer is None:
    return

Run the program again. No exception.

The agent proudly announces that the issue is resolved.

There's just one problem: the customer still doesn't receive the receipt.

The actual bug: the API sent customerId, but another file tried to read customer_id.

One mismatched field name.

The crash happened in the email code. The bug was in how the customer data was read, three files away.

In my experiment, I made this even harder by putting a misleading comment next to the crash. The comment suggested missing customers were normal.

The agent saw the comment, believed it, and added a guard.

From its perspective, everything made sense. The error disappeared. The code looked reasonable. The comment supported the decision.

It wasn't being stupid. It was solving the problem right in front of it.

That's the first thing to understand about agent debugging:

The easiest way to make an error disappear is not always the right way to fix it.

And when your agent gets rewarded for making the error disappear, don't be surprised when that's what it does.

II. You're optimizing for the wrong goal

Think about the last time you asked an AI agent to fix something.

What did you actually type?

"Fix this bug."
"Make the tests pass."
"Get the build green."

Now think about how you knew it was finished.

Maybe the tests passed. Maybe the error disappeared. Maybe the agent wrote a convincing summary of what it changed.

You saw a green checkmark and moved on.

I get it. That's the whole reason we use agents. We want to spend less time babysitting our software.

But there's an uncomfortable difference between these two goals:

Goal A: Make the error stop happening.
Goal B: Restore the correct behavior without introducing another problem.

The first is easy to measure. The second is what you actually want.

In the receipt example, the null check achieved Goal A. It failed Goal B.

This isn't necessarily an intelligence problem. It's a measurement problem.

If success means "the test passed," the agent can optimize for passing the test.

If success means "the feature still works and the original failure is gone," the agent has to do something harder. It has to understand what the software was supposed to do.

A better model may improve its chances. But even a very capable model can pursue the wrong objective.

You don't get a reliable agent just by making it better at completing tasks. You have to get better at defining what completion means.

Which brings us to something most of us don't want to admit.

III. You don't actually want your agent to slow down

There's a reason we keep accepting the green checkmark.

Verification feels expensive.

You ask an agent to fix a bug. It takes 30 seconds and says it's done. Great.

Now imagine it says:

"I have a possible fix, but I need to reproduce the failure, trace the input, run the relevant tests, and verify the original behavior."

Suddenly the agent doesn't feel so magical. You wanted a fix, not another debugging session. So you skip the extra checks.

I understand the temptation. I've spent years dealing with production incidents. When something is broken, the urge to turn the red dashboard green is powerful.

But here's what we forget:

The fastest fix isn't the one that finishes first. It's the one you don't have to fix again.

A bug that comes back costs more than another minute of investigation. Now you're debugging the original problem, the agent's workaround, and whatever new behavior that workaround introduced.

And the second time around, the evidence might be gone, because the first fix made the failure silent.

That's a particularly nasty failure mode. A loud crash becomes a quiet business problem.

Your monitoring looks healthier. Your customers are less happy.

The real cost of agent speed isn't the time it takes to generate a fix. It's the time you spend finding out whether that fix was wrong.

This doesn't mean every change needs a massive validation pipeline. It means you need to stop treating verification as optional friction.

The next question is how to help an agent find the right cause in the first place.

IV. The four levels of debugging evidence

Not every bug needs the same investigation. But there's a useful way to think about the evidence available to an agent.

Level 1: The error

The agent reads the exception or stack trace. It knows where the program failed.

For simple bugs, that might be enough. For harder failures, it has only found the smoke.

Level 2: The code and the data flow

The agent follows the failing value backward.

Where was it created? Who passed it here? Was it transformed along the way? Did one service send customerId while another expected customer_id?

Now the agent can start telling the crash site apart from the cause.

Level 3: What actually happened at runtime

The agent examines concrete artifacts of the failure: a core dump, a binary, a recorded request, a sanitizer report, a trace showing what value was actually present.

This is where a plausible story collides with reality. The source code suggests what should have happened. Runtime evidence establishes what did happen.

Level 4: What happens after the fix

The agent reproduces the failure when possible, applies the change, and checks again.

Not just for the disappearance of the error. It checks that the original behavior still works. In our example: did the receipt actually get sent?

These aren't rigid stages. Different failures need different evidence, and some production incidents can't be reproduced safely.

The point is to move past the first explanation that sounds reasonable.

This is part of why I built CrashLens. It gives coding agents more information about where a failure originated, instead of leaving them staring at the final exception.

In the receipt experiment:

- Error message alone: 6/18 root-cause fixes

- Error plus CrashLens report: 13/18 root-cause fixes

Same model, more useful evidence.

That was an improvement, but it still left five failed attempts. Which reveals another problem:

Even with useful evidence, an agent can reach the wrong conclusion.

So what happens when the agent needs to question its own answer?

V. Debugging isn't an answer. It's a feedback loop.

A lot of people judge AI agents by what they produce.

How quickly did it generate code? How many steps did it complete? Did it finish without asking questions?

But debugging doesn't work like writing an essay. You don't just think hard, produce an answer, and declare victory.

You form a hypothesis. You test it. You observe what happened. You update your understanding.

Observe → Hypothesize → Test → Update → Verify

Take the receipt bug again.

The agent observes a null customer. Its first hypothesis: the customer can legitimately be missing.

Instead of immediately adding a guard, it tests that assumption. Why is the customer missing?

It traces the customer reference back to the incoming request and finds the mismatched field name.

The original hypothesis no longer explains the evidence. So the agent changes its mind.

That's what good debugging looks like. Not having the right answer immediately, but being able to discover when your first answer is wrong.

One critical part of that loop is often missing from agent workflows:

The agent needs to look for evidence against its own explanation.

If it believes the null check is correct, what would prove otherwise?

- A test showing that the customer should exist

- A payload containing the customer ID under a different key

- A successful lookup after correcting the field mapping

Those are observations that can actually change the investigation.

"Looks good to me" isn't an observation. Neither is "I carefully reviewed my changes."

Those are opinions, even when an AI writes them confidently.

Now we can finally talk about the prompt.

VI. How to fix your agent in 1 prompt

You don't need to start with a giant system prompt.

You don't need five subagents debating the root cause.

And you definitely don't need to tell the model it's a genius programmer with 30 years of experience.

Start by changing what the agent is trying to accomplish.

The next time Claude Code, Codex, Cursor, or any other coding agent is debugging a failure, try this:

Save it somewhere. Try it the next time your agent reaches for a quick null check or starts patching the first line of a stack trace.

Notice what this prompt doesn't do.

It doesn't tell the agent to think harder. It tells the agent what evidence to collect, what assumptions to question, and what must be checked before a fix counts as done.

The most important instruction might be the simplest one:

Show me what could prove you wrong.

That's how you get the agent to investigate rather than defend its first guess.

To be clear: I haven't measured the improvement from this exact prompt. My 6/18 vs. 13/18 experiment tested extra debugging evidence, not these instructions.

And a prompt has limits. It can't create missing logs. It can't guarantee a test covers the right behavior. It can't force an agent to follow every instruction.

But it can give the agent a better process to follow. If you use agents every day, that's a useful place to start.

And then you can go one step further.

VII. Put your agent on a scoreboard

Here's an experiment.

For the next ten bugs you give your coding agent, stop measuring only whether it says the task is complete. Track four things:

1. Attempts: how many fixes did the agent produce?

1. Verified fixes: how many passed checks that actually exercised the required behavior, rather than just removing the error?

1. Unverified fixes: how many looked correct but lacked the evidence to show they worked?

1. Regressions or contradicted fixes: how many failed independent checks or broke something that used to work?

You don't need a fancy dashboard. A small table is enough.

The important part: the agent doesn't get to award itself a verified result just because it wrote a convincing explanation.

And zero observed wrong fixes doesn't automatically mean an agent is safe. It might mean you haven't tested it hard enough.

The scoreboard's job is to make the gap between claimed success and demonstrated success visible.

Once you record that gap, you can actually improve the workflow:

- If your agent keeps fixing symptoms, give it better data-flow evidence.

- If it keeps trusting misleading comments, make it check comments against runtime behavior.

- If tests pass while features break, improve what the tests actually verify.

- If it can't reproduce a failure, require it to report what remains uncertain instead of pretending the investigation is complete.

Now you're not just collecting green checkmarks. You're learning where the agent fails and what actually helps.

That's far more useful than endlessly rewriting a system prompt and hoping the next version is smarter.

And it lays the foundation for something bigger: agents whose work can be checked by tools and processes outside the agent itself.

There's a strange thing about AI coding agents.

They can write a convincing explanation for a broken fix.

They can pass a test that never checked what the customer actually needed.

They can confidently report success while the original problem is still there.

And if you don't look closely, you'll believe them.

One prompt won't eliminate that. But it can change the first question your agent asks.

Not "How do I make this error disappear?"

But "What caused this, and how would I know if my fix is wrong?"

That's a much better place to start.

Because the next time your agent says "Fixed," you should have one more question:

What would prove it wrong?

Source details

Minglei · x.comPublished · added here

Original title: Everyone's trying to make AI agents smarter.