In this article, I break down how I built OSA, the architecture behind it, the tools I use, and the lessons I’ve learned along the way. https://x.com/i/article/2115771377152135168
How I Built My AI Bug Bounty Hunting System (And How You Can Build Yours)
Hello everyone!
I love AI, but at the same time, I'm kind of tired of it. It's genuinely great technology, and I've been using it a lot in my work. Looking back, I think I've gotten better at using it, or maybe just found more effective ways to make it work for me. I'm still figuring that part out
The funny thing is, I built my own AI agent with practically zero prior experience in AI, LLMs, or software engineering. I did most of it by using AI itself.
After four years in bug bounty, I've recently started feeling like I'm falling behind. Not because I've lost my skills, but because of how quickly people are integrating AI into their workflows. It feels like everyone's moving fast, and I didn't want to be the person who ignored the shift.
So, in late August, I decided to take a step back and actually learn this technology: how it works, what it's capable of, and, more importantly, how someone with no prior experience in AI can use it to become better at their work.
Abstract
One of the biggest problems with AI is that it tries its best to help you, even when the information it provides is wrong.
I don't think AI will replace human work. I think it will help people deliver higher-quality work in less time. And in bug bounty, a few seconds can make a difference. They can be the difference between submitting a duplicate and finding a valid bug before someone else does.
If I had to explain why I use AI in my bug bounty workflow, my first answer would be simple: to make my workflow faster. If AI isn't making your workflow faster, you're probably not using it effectively or taking advantage of what it was built for.
That's the idea behind building an AI agent for bug bounty. It should speed up your workflow, handle repetitive tasks, and help validate your findings. But it shouldn't replace your judgment or your technical skills. I'd want it to handle roughly 80% of the workflow, leaving the remaining 20% to my own analysis and decision-making. Those numbers are just a way to describe the idea, not a measured result.
If you try to build an agent without having the skills to understand the work it's doing, you'll end up with unrealistic expectations that the agent can't meet. Instead of becoming someone who uses AI to work faster and deliver better results, you risk becoming someone who blindly chases every possible lead in the hope of finding a valid bug, without really understanding what they're doing.
The Stack
I want to share my current stack so you can see what I'm using, why I chose it, and what I've learned along the way.
I used to run OpenCode, but I've since moved to OMP. I explained why I made that transition in this tweet if you want to read more about it:
https://x.com/sl4x0/status/2114274318897041898
My current stack consists of OMP as the runtime, DeepSeek's direct API with pay-as-you-go usage, JavaScript for OMP extensions and plugins, and eight configured MCP servers that give the agent access to the tools it needs depending on the task.
On top of that, I've been building OSA, my bug bounty automation framework, around OMP. It's based on an existing bug bounty AI framework that I've rebuilt and extended for this runtime, rather than something written entirely from scratch.
The important distinction is that OMP is the runtime, while OSA is the control plane around it. OMP handles the model turn loop, tool dispatch, sessions, and context compaction. OSA manages the bug bounty workflow, persistent state, evidence, scope checks, validation, and reporting.
I spent a lot of time researching how engineering teams design AI systems, using ChatGPT to help me understand the concepts and explore different approaches. Based on that research and my own experiments, I ended up organizing the system around several layers.
I'm not claiming this is the definitive way to build an AI agent. It's simply the architecture that made sense for my workflow, and I'm still learning how to improve it.
1. Instructions
The instruction layer is everything that tells the agent what to do, how to approach a task, and what rules it must follow.
In my setup, this includes AGENTS.md, rules, agent definitions, skills, prompts, and configuration files.
One thing I've learned is that putting everything into a single massive prompt isn't the best approach. Different tasks need different instructions, and loading every skill and methodology upfront wastes context.
Instead, I use a layered approach: core instructions establish the overall behavior, while specialized roles and skills provide additional guidance when needed.
The goal is to give the agent the right methodology for the task without overwhelming it with unnecessary information.
2. Agent loop
This is where things get interesting.
An AI model responding to a prompt isn't the same thing as a complete agent workflow. You need to define what happens after the model responds, how tools are invoked, how results are evaluated, and what determines whether the task should continue or stop.
In OSA, there are three distinct mechanisms:
- Model turn loop: OMP handles the model, tool calls, observations, and subsequent model turns.
- Hunt continuation loop: OSA's supervisor determines whether the workflow should continue or finish.
- Hunt state machine: Python manages the hunt's phases, hypotheses, budget, validation, and reporting state.
The bug bounty workflow follows a sequence of phases:
Recon → Map → Hypothesize → Test → Validate → Report
The model can propose hypotheses and perform tests, but it doesn't get to decide on its own that the entire hunt is finished. The supervisor checks outstanding obligations before allowing the workflow to stop.
This distinction matters because a model can prematurely conclude that it has finished, overlook an unresolved hypothesis, or summarize incomplete work as a final result.
I want the system to make those mistakes harder to get away with.
3. Tool execution
The tool execution layer determines what the agent can actually do.
This includes the tools available through OMP, MCP servers, browser access, shell commands, JavaScript extensions, and plugins.
But giving an agent access to tools is only part of the problem. You also need to control how those tools are invoked, inspect their arguments, handle their outputs, and record the results.
In OSA, the JavaScript extension acts as an enforcement adapter around OMP's tool execution events. It connects those events to the guard chain, while Python handles authoritative state, evidence recording, validation, and reporting.
For example, the agent might describe an HTTP request and claim that it confirmed a vulnerability. That description alone isn't evidence. The actual request must execute, its response must be recorded, and the resulting evidence must be evaluated.
The same principle applies to MCP calls: they need to be inspected and checked against the relevant scope rules rather than automatically trusted because they come through an integrated tool.
The idea is simple: the model can propose an action, but the execution layer determines what actually happens and what gets recorded.
4. Context management
Context management is one of the more challenging parts of building an agent.
You can't keep feeding the model every observation, request, response, failed hypothesis, and previous conversation indefinitely. Eventually, the context becomes too large, important information gets buried, or compaction removes details the agent still needs.
OSA treats context as a selected collection of relevant information rather than simply replaying the entire conversation.
For example, before continuing a hunt, the system can assemble a context pack containing relevant assets, observations, leads, hypotheses, previous failures, methodology, and required evidence.
The agent gets the information needed for its next decision without having to process the entire history of the hunt.
I also separate the context used by the model from the durable handoff information needed to resume work. When a session is compacted or a task moves between agents, important information needs to survive beyond the current context window.
The goal is to keep the agent focused while preserving the information required for reliable continuation.
5. State management
Context and state are related, but they aren't the same thing.
Context is the information the model sees. State is the information the system persistently maintains about what has happened and what needs to happen next.
For a long-running bug bounty workflow, I need to track things like:
- The current hunt phase and remaining budget.
- Discovered assets and attack surfaces.
- Open, tested, and rejected hypotheses.
- Evidence collected during testing.
- Pending tasks and agent dispatches.
- Validation status and reporting decisions.
OSA stores this information in structured files, including hunt.json, ledger.json, and scope.json, alongside finding records, validation assessments, and handoff data.
The important design principle is that each type of information has a clear source of truth. The hunt state tracks the workflow, while the evidence ledger records operations and their provenance.
This matters because long-running agents can repeat work, lose track of previous results, or produce inconsistent conclusions if their state isn't managed properly.
I don't want the agent to rely on remembering everything from previous model turns. I want it to be able to recover its position from persistent state.
6. Guardrails and approval gates
This is one of the most important layers in a security research agent.
A model can misunderstand scope, execute an inappropriate action, misinterpret a response, or report a vulnerability that hasn't actually been demonstrated. Instructions alone aren't enough to prevent every one of these problems.
OSA uses several controls for different purposes:
- Tool guard chain: Checks actions against rules covering scope, state integrity, evidence, dispatch conditions, and other workflow requirements.
- Scope gate: Evaluates targets and destinations against the configured scope rules.
- Validation pipeline: Evaluates whether a finding has sufficient evidence and reproduction support.
- Reportability gate: Separates technical validity from impact, reachability, deployment relevance, and whether a finding is ready to report.
- Reporter controls: Restrict report generation and approval through a dedicated reporting workflow.
- Kill switch: Provides a mechanism to block tool execution.
- Evaluation controls: Restrict access to tools that could bypass the normal guard chain.
One distinction I've learned to make is that a technically valid finding isn't automatically a reportable finding.
A result might be reproducible but lack evidence of reachability or meaningful impact. The system should be able to distinguish those situations instead of treating an LLM confidence score as proof.
There is also a practical caveat: these controls aren't all equivalent to hard security boundaries. My current configuration uses permissive settings in some areas, including advisory scope checks and automatic tool execution without per-tool approval prompts. The enforcement behavior depends on the configured mode, and OMP extensions themselves are not a security sandbox.
So I wouldn't describe the system as completely autonomous or inherently safe. The objective is to make the workflow more controlled, make unsupported claims harder to promote, and keep the relevant decisions and evidence visible.
7. Observability, tracing, evaluation, and benchmarks
The last layer is about understanding what the system actually does and whether it's improving.
Without observability, an agent can spend hours working through a task, and when it fails, you might have no clear idea why. Did it choose the wrong hypothesis? Did a tool fail? Did it lose important context? Did validation reject the finding? Did the workflow stop too early?
OSA records information across agent-loop logs, session events, dispatch telemetry, the evidence ledger, scope checks, and reportability assessments.
The tracing tools bring those records together so I can reconstruct what happened during a hunt and follow the decisions associated with a particular hypothesis.
Evaluation is a separate part of the process. I use tests, regression checks, lab evaluations, and benchmarks to examine system behavior and catch problems introduced by changes.
The goal isn't just to have logs. It's to understand why the system behaved the way it did and use that information to improve it.
If I change a prompt, modify a guard, adjust the context pack, or change how the agent handles validation, I want to know whether the change actually improves the result rather than relying on how convincing the output looks.
How the pieces fit together
At a high level, the workflow looks like this:
1. I start a hunt with a program and target.
1. OSA initializes the hunt state and its execution context.
1. OMP runs the model and handles tool dispatch.
1. OSA's execution controls inspect actions before they reach the tools.
1. Executed operations and their evidence are recorded.
1. The agent continues through reconnaissance, mapping, hypothesis generation, testing, and validation.
1. The validation pipeline evaluates potential findings.
1. The reportability gate determines whether the evidence supports reporting.
1. The reporting workflow prepares the result for human review.
1. The supervisor checks outstanding obligations and decides whether the hunt can finish or needs to continue.
Throughout this process, context management, persistent state, guardrails, and telemetry support the workflow rather than operating as isolated features.
The model is only one component of the system.
The model proposes hypotheses and actions. The runtime executes the model's turns. The state machine tracks the hunt. The evidence ledger records what actually happened. The validation and reporting gates evaluate what the evidence supports. The supervisor determines whether the work is finished.
That's the architecture I've been working toward.
I'm still learning, and I don't expect this design to be perfect. But thinking in terms of these layers has given me a much clearer way to identify weaknesses, test changes, and improve the system.
Memory, Success Techniques, and Skills
I wanted my agent to benefit from previous hunts instead of starting from zero every time. But these are three different things, and I treat them separately.
Memory stores previous wins, dead ends, program notes, and lessons learned. Before a hunt or a new hypothesis, the system retrieves relevant information instead of dumping the entire history into the context window. Not everything is trusted: an agent-written note is still an unverified observation until it's validated.
Success techniques take this a step further. Instead of remembering that a particular attack worked on one target, the system tries to extract why it worked and turn that into a reusable approach. Target-specific details are stripped out, and a technique needs evidence from multiple targets before it becomes a durable cross-program lesson.
Skills are different again. They're structured, human-readable playbooks that teach the agent how to approach a particular class of work. They're loaded when needed rather than all at once. A successful technique can become a candidate for a new skill, but promotion requires review; the agent doesn't automatically turn every finding into a new methodology.
How experienced does your agent get over time?
Does it improve after every day, every hunt, or not at all?
My approach is to separate recording experience from learning reusable lessons. A confirmed result can create a success record, repeated success across different targets can support a reusable technique, and reviewed promotion can turn that technique into a skill.
This isn't automatic intelligence growth, and the agent doesn't retrain its model after every hunt. It's a controlled way to make previous experience available to future work without treating every observation as fact.
The system is still accumulating validated knowledge, so I wouldn't claim it has a massive learning corpus. The architecture is there; the quality and quantity of accumulated experience still matter.
Final Thoughts
Maybe this isn't the best system, and maybe there are better ways to build something like this. I'm still learning, and I don't pretend to have the engineering experience of someone who has spent years building these systems. But if you have experience in AI engineering or security research, your feedback matters. There's always room to improve.
One thing I'm confident about, though: experience makes a difference. Give an experienced security researcher a good normal tool, and they'll approach problems differently from someone who has no background in security with the best tools. The tool might be the same, but the person using it isn't.
You see people sharing their wins on platforms, and it's easy to assume that AI has made security research accessible enough for anyone to start finding bugs without learning the fundamentals. But public submissions don't tell the whole story. You don't see every failed attempt, every invalid report, or how much security knowledge went into finding and validating a vulnerability.
Even if AI helps someone find a valid bug without much prior experience, that doesn't mean they've developed the skills to repeat that success consistently. I don't think that approach will remain as effective as people expect.
What will make the difference is the experience you build along the way: understanding vulnerability classes, learning new techniques, researching how technologies work, and knowing how to validate what you find. That's what allows you to use AI to turn an idea into something useful rather than blindly follow whatever it suggests.
I used to be someone who wasn't particularly interested in programming. Last April, while preparing for my marriage, I built a tool called Dependency Confusion Scanner (depconf). I didn't have extensive Python experience, but I had an idea, understood the security problem I wanted to address, and researched how to implement it as a CLI tool. My security background helped me make decisions about the implementation and work toward the result I wanted.
That's where I think AI is most useful: it helps you turn your existing knowledge and ideas into something tangible, even when you lack experience in a particular programming language or engineering discipline.
But there's a downside we need to talk about. AI makes explanations, research, and implementation easier, which can also make us lazy about learning the underlying concepts. If you let it do everything for you, you might finish more tasks without actually becoming better at your work.
Sometimes, you need to slow down, read a research paper, take notes, understand the implementation, and test what you've learned against your bug bounty targets. You need to experience failures as well as successes. Don't let AI take away the effort that builds your expertise.
The question I still don't have an answer to is: how do we learn effectively in a world where AI can make almost everything easier? How do we use it to improve our work without becoming dependent on it or losing the motivation to understand things ourselves?
I'd genuinely like to hear other people's thoughts on this. Feel free to reach out in DMs if you have experience or ideas to share.
Don't treat AI as a magic solution that guarantees success while you sit back and wait for results. Treat it as a powerful tool that requires knowledge, judgment, and careful use.
Keep learning security. Keep exploring new technologies. Keep developing your own understanding.
Use AI to become a better researcher, not a replacement for becoming one.
Peace.