Loading...
Loading...
Will AI replace penetration testers? Not outright, and not evenly. In a December 2025 Stanford, Carnegie Mellon, and Gray Swan AI study on a live 8,000-host network, the ARTEMIS agent placed second overall and beat nine of ten security professionals. AI absorbed finding. It has not absorbed judging, or the years of finding that produce judgment.
Most answers to this question are written for the person paying for the pentest. This one is written for the person doing it.
Ask "will AI replace penetration testers" in any security forum and the reassuring version arrives within minutes: AI handles breadth, humans handle depth, everybody wins. It is also too coarse to plan a career around. The measured results say something narrower, and the part that should worry you is not the part the headlines pick.
At finding vulnerabilities, yes — AI is now ahead of the median penetration tester. Two results from the past year agree: an academic study on a live enterprise network, and a vendor's own web application benchmark. Both put a purpose-built agent above most professionals at enumerating an attack surface and clearing OWASP Top 10 classes.
The first is academic. In Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing, a Stanford, Carnegie Mellon, and Gray Swan AI team put ten security professionals, six existing AI agents, and their own scaffold called ARTEMIS onto a university network of roughly 8,000 hosts across twelve subnets. ARTEMIS finished second overall with nine valid vulnerabilities at an 82% valid-submission rate, beating nine of the ten humans.
Certain configurations of it ran at about $18 an hour in compute against roughly $60 for a professional tester — a figure that excludes the reviewer the rest of this article is about. The paper does not publish the seniority or time incentives of the ten humans, which is worth holding onto before you read the ranking as a rout.
The second is a vendor benchmark from June 2026, which is a weaker instrument. XBOW, an autonomous offensive-security company, ran five professional pentesters and its own agent against 104 web security benchmarks. The most experienced human, with more than twenty years in the field, solved 85% in forty hours. Every other human landed at 59% or below. XBOW also solved 85%, in twenty-eight minutes. Broken out by difficulty, it cleared the easy and medium challenges better than all five humans, and on the hardest ones it still finished ahead of the staff, senior, and junior testers. Only the twenty-year veteran stayed in front.
XBOW built the agent, chose the challenges, and published the score, so read it for the shape and not the number. Two caveats it does not attach: a benchmark set is not an engagement — there is no scoping call, no credentials nobody will hand over, no production database you must not touch — and twenty-eight minutes of parallel compute is not the same unit as forty serial human hours. What survives both caveats is the ordering. XBOW's own write-up adds the line that matters: most vulnerabilities found in the real world sit at those easier levels.
Read the two together and the shape is clear. This is not a wall the machine is stuck behind. It is a distribution, and the machine has settled above the middle of it. Gunter Ollmann, CTO of the pentest-as-a-service firm Cobalt, put a clock on it in Dark Reading in February 2026: "in the next 18 months to three years, I expect that half the pen testers around the world will not be able to keep up with their AI counterparts in this space."
All three of those sources hold a commercial position — the paper is co-authored by Gray Swan AI staff and acknowledges an unrestricted gift from OpenAI.
None of them is selling doom.
Partly. An agent can scope an attack surface, run OWASP Top 10 baseline coverage, and retest a fix on demand without supervision. What it cannot do unsupervised is decide which of its own results are real. The ARTEMIS paper names two hard limits: AI agents "exhibit higher false-positive rates and struggle with GUI-based tasks."
HackerOne's own analysis, reported alongside that paper, describes the current generation as working across a wide but shallow set of engagements, strong on methodical enumeration across a whole estate and weaker anywhere depth is required.
The same paper carries a result that gets quoted less often. ARTEMIS was purpose-built for the task; the off-the-shelf scaffolds tested beside it, including Codex and CyAgent, underperformed most of the human participants. Agent capability lives in the harness, not the underlying model. That is why two teams pointing the same model at the same network get results a tier apart, and it is the single most useful fact in this whole debate — the winning result required a research team building bespoke tooling, not somebody aiming a chatbot at a /16. If you want the mechanics of how AI penetration testing actually assembles a run, that is the piece.
Four things, and none of them closes with a better model.
Notice what those four have in common.
Each needs context that exists nowhere in the target, and none of them has a ground-truth signal you could train against. That distinction predicts what gets automated next better than any capability list:
| Work | Ahead today | Ground-truth signal exists? |
|---|---|---|
| Enumerating a large attack surface | AI | Yes, a host answers or it does not |
| OWASP Top 10 baseline coverage | AI | Yes, the payload fires or it does not |
| Retesting a fix after remediation | AI | Yes, the exploit still works or it does not |
| Confirming a claimed finding | AI, soon | Yes, re-run the reproduction steps |
| Chaining findings into an attack path | Human | No, the path is novel by definition |
| Judging business-logic abuse | Human | No, intent is not in the traffic |
| Proving a clean result is actually clean | Human | No, absence of evidence has no test |
| Owning scope and signing the report | Human | No, accountability is not a measurement |
Every row in the top half has a verifier. Every row in the bottom half does not. That is the whole forecast.
Every finding an agent produces is a claim until a human confirms it, and the volume of plausible-but-wrong claims rose faster than anyone's capacity to read them. The numbers here need separating, though, because two very different populations keep getting reported as one trend.
Purpose-built agents are clean. ARTEMIS submitted at an 82% valid rate, leaving nearly one submission in six for a human to catch, and still far better than the roughly 15% the curl project used to see from human bug-bounty reporters. Ungoverned tooling is the opposite: HackerOne's ninth Hacker-Powered Security Report describes fully AI-written reports as "often polished but technically shallow," and notes triage teams spot them quickly. Polished is the operative word. The failure mode is competent-looking work that costs a reviewer real time to disprove, not gibberish anyone waves off at a glance.
So the flood traces to ungoverned tooling aimed at a channel that paid per accepted report. Keep those two apart and the curl story below reads correctly. Collapse them and you will conclude, wrongly, that agent output is inherently noisy — a conclusion that held in 2024, when almost nobody was engineering the harness, and does not hold now. Whether a given agent hallucinates is a separate question with a separate answer.
Here is the part that should unsettle you more than the volume.
Re-running a claimed exploit to see whether it fires is a closed loop with a ground-truth signal, which is the exact shape of problem automation eats first. Validation is not a moat. It is the next thing on the menu.
What sits behind it — intent, chaining, knowing when to stop — is durable precisely because no verifier exists for it.
The curl shutdown shows what happens to an intake channel when its confirmed-report rate falls through the floor, and it is the clearest public record of AI-generated volume overwhelming human review. The curl project — the open-source data-transfer library shipped on billions of devices — ran a bug bounty through HackerOne for years, paying out more than $100,000 across 87 confirmed vulnerabilities. Then the rate collapsed. curl's creator and lead maintainer, Daniel Stenberg, documented it in January 2026: "Previous years we have had a rate of somewhere north of 15% of the submissions ending up confirmed vulnerabilities. Starting 2025, the confirmed-rate plummeted to below 5%."
He shut the program down at the end of that month, naming three trends together: "the mind-numbing AI slop, humans doing worse than ever and the apparent will to poke holes rather than to help." curl returned to HackerOne on 1 March 2026 to keep receiving reports, with the reward money gone for good. His summary of the aftermath: "Since we dropped the bounty, the inflow tsunami has dried out substantially."
That story says nothing about whether AI can find bugs. It says what happens when the cost of generating a plausible finding drops to near zero while the cost of evaluating one stays exactly where it was, and a cash incentive sits at the end of the pipe. One of the oldest informal on-ramps into offensive security closed, and it closed on a validity statistic.
Set against that: one practitioner's account of a weekend spent running an agent against open-source projects, reporting around fifteen CVEs, every one accepted and none downgraded by MITRE or the maintainers. In the same experiment, he notes, the agent claimed to have found 98 critical vulnerabilities. He does not say how many of those he sent, and his point is about the operator who would have sent all of them: they would "lose trust and damage his reputation, and have even the valid findings be discarded as slop."
Same class of tooling as the reporters who broke curl. Opposite outcome.
The agent was not the variable. What he refused to send was.
At the junior level, largely yes. The work AI does best right now — methodical enumeration, known vulnerability classes, first-draft reporting, and retesting after remediation — is close to a description of what you were handed to become senior. That is the part most writing on this question steps around.
Practitioners are already saying so. A senior offensive-security engineer at a large technology company, in a thread titled "Is AI killing junior pentesters?", was direct about the trade: a well-engineered AI tool, in his experience, delivers more value than a junior he has to guide. In a separate r/cybersecurity thread, a tester described a colleague who used AI to rescue three trainees' projects in a day and was told by an executive that the trainees were therefore not worth keeping. The trainees went. He now carries every project alone.
The strongest counterargument comes from the same community. Offensive security arguably never had a real entry tier to lose. "Penetration Testing does not have a 'I got into IT 13 months ago and did a two semester program, hire me' sort of entry-level," as one hiring practitioner wrote in a thread on breaking into the field. In the thread named above, more than one commenter met the question with surprise that junior pentest roles existed at all. The way in has always run through something adjacent — systems administration, network engineering, development, a stretch on a SOC console — and then sideways into offense.
Both are true, and together they sharpen the problem. The route in was always a side entrance, and the side entrance ran through exactly the work being automated first. Which exposes the thing nobody wants to say out loud: the judgment everyone agrees is durable is sediment. You can throw out most of a list of claimed criticals in an afternoon because you have found the real version of each of those bug classes with your own hands. Automate the hands and the sediment stops forming.
That works precisely once, for the cohort that already has it.
Yes, on a condition that did not use to apply. Only 12% of security researchers surveyed for HackerOne's ninth Hacker-Powered Security Report, published in 2025, believe AI will replace them, and more than two-thirds already use it. Treat the first number as a confidence reading, not a risk assessment.
It is what every automated workforce has said on the way in. The second number is the useful one. Two-thirds are already holding the tool.
The open question is whether they have noticed which half of their job it took.
Jay Bavisi, founder of EC-Council, splits the profession in two: work limited to repeatable scanning and basic checks will largely be automated, while systems thinking, business logic, attacker intent, validation, and prioritization remain essential. Every item on that second list is a judgment about output.
None is a technique for producing one.
Ollmann's version is starker. He expects the market to stop being about a firm's roster of five hundred or a thousand testers and to concentrate on the top one or two percent, who then supervise and improve the tooling. Whether the concentration lands that hard is an open question, and he sells human pentests, so weigh it accordingly. Planning as though it might costs you very little.
Nobody is advertising these titles yet. This is the shape the work is taking, read off what the firms above are actually paying for, and each role is defined by a decision the tooling has no standing to make.
Our own bet as a vendor sits on that last point. Simbian's AI Pentest Agent ships every finding with its Thought Traces — the reasoning trail showing what the agent tried, what it observed, and how it got there — alongside deterministic reproduction steps. That transparency is the point. A finding without a trace is a claim you have to re-derive from scratch. A finding with one is a claim you can confirm or kill without rebuilding the path.
That is the whole economics of the supervisor role, and it cuts both ways: the same trace that lets you review faster is what a future system would need to review itself. It is also what makes continuous testing survivable — running on every release only helps if somebody can clear the output, and the window of exposure closes at fix-verified, not at discovery.
Q: Will AI replace penetration testers? Not outright. In the Stanford, Carnegie Mellon, and Gray Swan AI study published in December 2025, the strongest agent placed second on a live 8,000-host network and beat nine of ten professionals, but a human still finished first and other agents scored below most humans. AI has overtaken the median tester at finding vulnerabilities, while chaining, intent, scoping, and accountability remain human work.
Q: Will AI replace junior penetration testers? Largely, at the task level. AI is displacing enumeration, known vulnerability classes, first-draft reports, and retesting, which is the work juniors were traditionally handed. Practitioners report hiring managers already choosing tooling over trainees. The structural risk is not unemployment but apprenticeship: that work is how senior judgment was formed.
Q: Do compliance auditors accept AI-generated penetration tests? In practice what frameworks require is a qualified, accountable party and an attributable report, not that the testing be done by hand. That means an agent-generated finding set is workable when a named human has scoped the engagement, validated the findings, and signed the report. Confirm the specific requirement with your assessor before relying on it, because interpretations vary by framework and by auditor.
Q: Is it too late to become a penetration tester? No, but the entry path has changed. Offensive security never had a true entry tier — the route in has always run sideways from systems administration, network engineering, development, or a SOC console. What is new is that the first rung is agent supervision: reading findings, killing false positives, and owning the validity rate on client-facing work. Anyone asking how to become a penetration tester in 2026 should assume that rung is where they start.
Q: Do AI-generated pentest findings hallucinate? Purpose-built agents produce false positives at a measurably higher rate than human testers, and ungoverned AI reports are frequently polished but technically shallow. Validity rate matters more than finding count. curl ended its bug bounty after its confirmed-report rate fell from above 15% to below 5%. The under-discussed risk is the opposite error: findings the agent never surfaced at all.
Q: What should a penetration tester learn to stay relevant? Three things compound. Build and tune your own agent harnesses, since that is where capability lives. Get fast and defensible at rejecting findings. And go deep on the classes where correctness turns on intent — authorization boundaries, business logic, and multi-step chains — because those are the ones with no verifier to automate them.
Every answer to this question that survives contact with the data, including the ones written by vendors selling the agents, lands in the same place: the machine got very good at producing candidates, and judging them is now the job.
The uncomfortable version is that judging is also the part you learned by doing the work that is being automated. Nobody is going to hand you that skill, and the only place left to build it is on the output of whatever agent your firm has already bought.
