Research
Backbone or Backbone-Architecture? A Controlled Study of LLM Agents on Web Penetration-Testing CTFs
The model is not the agent.
About the study
A controlled study, by Cyphlon CEO Zeeshan Sultan, of what drives LLM agent performance in web penetration testing: the model, or the scaffold around it. Across the 104 single-flag CTFs of the XBOW validation suite, 9 agent scaffolds and 10 backbone models, more than 2,700 runs were scored on exact flag retrieval and every solve was audited. Holding the model fixed and changing only the scaffold moved the same backbone from 0 to 49 solves.
Full paper
Backbone or Backbone-Architecture? A Controlled Study of LLM Agents on Web Penetration-Testing CTFs
Zeeshan Sultan · June 2026, working draft · PDF
Abstract
We evaluate LLM penetration-testing agents on the XBOW validation benchmark (104 single-flag web-exploitation CTFs) across nine agent scaffolds and ten backbone models, including five 2026 frontier models not previously measured on this benchmark. Three results stand out. (1) GLM-5.2 averages 87.7 ± 4.9/104 over three trials (best 91, ≈84%), clearing 4–7/8 on the hardest tier, within reach of Claude Opus (101–102/104) and the only non-Claude model to clear the hard tier at scale. (2) The agent backend is a first-order variable: a deliberately lean agent on the Claude Agent SDK reaches scores comparable to a published "v2" agent with a bespoke planner and memory subsystem. (3) In a controlled ablation (same model, same targets, same backend; only the scaffold architecture differs), the bespoke v2 architecture underperforms the lean agent, 72 vs. 90/104, with the entire gap on the medium tier: direct evidence that planning machinery tuned for multi-host campaigns can hurt on single-flag CTF. We contribute a reusable benchmark-integrity protocol for deterministic-flag suites (142 solved transcripts audited clean) and a reproducible recipe for running reasoning models through the Claude Agent SDK. We release all code, the harness, and the consolidated score matrices.1 We are explicit about the main limitation: most cells are single-trial (GLM-5.2 is replicated n = 3).
1. Introduction
LLM agents for penetration testing improve rapidly, yet reported success rates swing widely with the scaffold (orchestration + tools), the backbone model, and the benchmark. Deng et al. [1] usefully separate Type-A failures (capability gaps, fixable by tooling) from Type-B failures (planning / state-management limits that persist regardless of model), and present a "v2" agent (codename Excalibur) with a Task-Difficulty-Assessment planner over an Evidence-Guided Attack-Tree Search (TDA-EGATS; an attack-tree analogue of Monte-Carlo Tree Search [4]), a memory subsystem, and a 38-tool typed Tool/Skill Layer, reaching 91% on XBOW with Claude Opus 4.5.
This paper complements that work on three axes the literature has not isolated. New models: we report the first XBOW numbers for five 2026 frontier models. Backend substrate: we separate the contribution of the agent backend (the Claude Agent SDK, which itself supplies tools and planning) from architectural modules layered on top. A controlled ablation: we run the published v2 codebase and a lean v1-class agent on the same model, same targets, same backend, leaving scaffold architecture as the sole variable, a comparison prior work cannot make because each system ships one fixed scaffold.
Contributions.
- First XBOW benchmark of GLM-5.2, Kimi-K2.7, DeepSeek-V4-Pro, and two Qwen3.6 variants; GLM-5.2 ≈ Claude Opus on web exploitation, including the hard tier.
- Evidence that backend substrate ≥ bespoke architecture for this task.
- A controlled v1-class-vs-v2 result in which the bespoke architecture loses.
- A benchmark-integrity protocol (flag = sha256(id)) and an audit of 142 solves verifying no flag-computation shortcut.
- A reproducible fix for reasoning models on the Claude Agent SDK / router bridge.
1.1 Related work
PentestGPT [2] established the LLM-driven CTF/HTB agent; AutoPT, VulnBot, PentestAgent, strix [7], and the HexStrike [8] line vary the tool layer and orchestration (surveyed in [1]). Deng et al. [1] contribute the Type-A/Type-B taxonomy and the v2 agent that motivates our ablation; we run their v2 code, not a reimplementation. Benchmarks probe different skills: XBOW [3] (web CTF, used here), HTB/VulnHub machines, and GOAD (multi-host Active Directory). Prior XBOW numbers are reported for Claude/GPT/Gemini-class models on each system's own scaffold; we add 2026 non-Claude frontier models, an explicit backend-vs-architecture isolation, and a same-everything-but-architecture ablation.
2. Method
Benchmark. XBOW: 104 web-exploitation CTFs in three tiers: L1 (easy, 45),
L2 (medium, 51), L3 (hard, 8). Each flag is FLAG{sha256(XBEN-ID)}, baked
into the target at build time; the agent must exploit the target to retrieve it. We vendor the exact 104
challenge sources in the artifact (XBOW Validation Benchmarks, © XBOW USA Inc.; the
benchmark.yaml autobuild snapshot, 2025-06-16), so the target build ships with the code rather
than depending on an external pin.
Harness. Our benchrunner starts one target at a time (or N in parallel), hands
the agent only the target URL and a generic instruction, captures the full transcript, and scores
exact-flag matches: a solve requires the precise 64-hex flag to appear in agent output; any other
flag-shaped string is a false positive. Per-cell wall-clock cap 30 min; setup cap 15 min.
Scaffolds. pentestgpt (an agentic build on the Claude Agent SDK backend,
§3.4), excalibur (the published v2), strix,
hexstrike-ai/-modified, pentestagent, pentest-swarm-ai,
decepticon, and hexstrike-mod-mcp: the modified HexStrike MCP server (192
tools) driven by the Claude CLI, running on the host with shell + docker access (grey-box; the only
non-sandboxed configuration).
Models. Claude Opus 4.8; 2026 frontier models GLM-5.2, Kimi-K2.7, DeepSeek-V4-Pro,
Qwen3.6-40B-Heretic, Qwen3.6-35B-A3B and its -thinking variant, Mimo-v2.5, Qwen3.6-Uncensored;
plus Qwen3.5 (ten distinct backbones). The non-Claude models were served through an OpenAI-compatible
gateway;2 Opus ran through the Claude Code CLI on an Anthropic subscription.
Provenance (important). All scaffolds except one are blind: container-sandboxed, no
host or docker access. The sole exception is hexstrike-mod-mcp/opus, the modified
HexStrike MCP server run on the host with shell + docker access, which is grey-box; up to 14 of its
solves used docker exec/file reads on the target rather than pure HTTP, so its strict-black-box
range is [88, 102]. We keep it in the table for completeness but treat the blind rows as the trustworthy axis
and never compare grey-box to blind numbers directly.
3. Results
3.1 Leaderboard
Figure 1 and Table 1 rank the main configurations; all 26 full-suite configurations are in results-archive/score_matrix_20260618-consolidated.csv. Tiers are
sharp: Opus (101–102), then GLM-5.2 at 90, then a cluster of open/Chinese models at
47–52, then a long tail. The top row (102) is the modified HexStrike MCP server with host
shell + docker access: grey-box, not black-box, and not directly comparable to the sandboxed rows. The
canonical blind leader is pentestgpt/opus at 101.
| Agent | Model | Pass | L1 | L2 | L3 |
|---|---|---|---|---|---|
| hexstrike-mod-mcp† | opus | 102 | 44 | 50 | 8 |
| pentestgpt | opus | 101 | 44 | 49 | 8 |
| pentestgpt | GLM-5.2 | 90 | 42 | 41 | 7 |
| excalibur (v2) | GLM-5.2 | 72 | 42 | 24 | 6 |
| pentestgpt | Qwen3.6-35B-think | 52 | 27 | 23 | 2 |
| pentestgpt | Qwen3.6-35B | 49 | 29 | 19 | 1 |
| pentestgpt | Kimi-K2.7 | 49 | 31 | 18 | 0 |
| pentestgpt | Mimo-v2.5 | 47 | 28 | 17 | 2 |
| strix | Qwen3.6-35B | 38 | 26 | 12 | 0 |
| strix | Mimo-v2.5 | 33 | 23 | 10 | 0 |
| pentestgpt | Heretic-40B | 32 | 21 | 11 | 0 |
| hexstrike-ai | Mimo-v2.5 | 30 | 17 | 13 | 0 |
| pentestgpt | DeepSeek-V4-Pro | 30 | 24 | 6 | 0 |
| pentestagent | Mimo-v2.5 | 26 | 19 | 7 | 0 |
| pentestgpt | Qwen3.6-Uncens. | 24 | 15 | 9 | 0 |
| hexstrike-ai | Qwen3.6-35B | 9 | 8 | 1 | 0 |
| pentest-swarm | Qwen3.5-35B | 0 | 0 | 0 | 0 |
3.2 GLM-5.2 ≈ Claude Opus
GLM-5.2 on pentestgpt averages 87.7 ± 4.9/104 over three trials (individual runs
90, 82, 91; Table 2), third overall behind only Opus, and is the only non-Claude
model to clear the hard tier at scale (4–7/8); every other non-Claude config scores 0–2 on L3
(Fig. 2). It is also the fastest of the strong solvers (3.5 min/solve, faster than both Opus configs) at a
modest 9.6 K tokens/cell, with zero false positives. This is a blind result and was
cheat-audited (§4). The leaderboard (Table 1,
Fig. 1) shows its first run (90) for comparability with the single-trial rows.
| Run | Total | L1 (45) | L2 (51) | L3 (8) |
|---|---|---|---|---|
| 1 | 90 | 42 | 41 | 7 |
| 2 | 82 | 41 | 37 | 4 |
| 3 | 91 | 42 | 42 | 7 |
| mean ± sd | 87.7 ± 4.9 | 41.7 ± 0.6 | 40.0 ± 2.6 | 6.0 ± 1.7 |
The variance is real but does not move the qualitative claims: across all three runs GLM-5.2 stays third overall (the best non-Claude model), well clear of the 47–52 cluster, and is the only non-Claude model to solve more than 2 of 8 on L3 (4–7 of 8). Easy cells (L1) are essentially deterministic (±0.6); the medium and hard tiers carry the variance, as expected where a single flag can hinge on one exploitation step landing within the time budget.
3.3 Scaffold is a first-order variable
Before isolating the backend, note the raw effect of the scaffold. Holding the backbone fixed at
Qwen3.6-35B-A3B and varying only the scaffold (Fig. 3), solves range from 0 to
49, a 49-solve swing on an identical model, wider than the spread across most backbone
models. Two scaffolds (pentest-swarm, decepticon) score 0; pentestgpt
scores 49. The scaffold is not a second-order detail.
3.4 Backend substrate vs. the paper's tiers
Our pentestgpt scaffold is a lean agent on the Claude Agent SDK
[5] backend. On Opus it reaches 97% (101/104), near
Deng et al.'s v2 (91%, Opus 4.5) and far above their v1 baselines (47–60%). Yet a source diff
shows this scaffold has none of v2's planner/memory/tool-layer modules: its
core/backend.py is identical to Excalibur's apart from its environment-variable prefix, but planner/, memory/,
and tools/{categories,skills} are absent. The implication: the Claude Agent SDK backend
itself supplies Type-A tooling and partial Type-B planning (Bash, file ops, task planning, sub-agents),
reaching v2-class scores without v2's explicit architecture. The backend substrate is a first-order variable.
3.5 Controlled ablation: v1-class vs. true v2
We grafted the published v2 (Excalibur) into the harness and ran GLM-5.2 on it. Same model, same targets, same backend; the only difference is scaffold architecture (Fig. 4). v2 scores 72/104 vs. the lean agent's 90, a loss of 18 solves, and the entire gap is the medium tier (L2: 24 vs. 41). v2 ties on easy cells (L1 42–42) and is within one on hard cells (L3 6–7). The mechanism is visible in transcripts: v2's planner keeps exploring after the flag is in hand (controller log: "Flags found, continuing to verify completeness"), so every v2 cell runs to the 30-min timeout (~1800 s vs. the lean agent's ~212 s). On easy cells the flag is captured fast enough that over-exploration is harmless; on medium cells the wandering exhausts the budget before convergence. The planner is designed for multi-host AD campaigns; applied to single-flag web CTF it is a mismatch. This is a controlled corollary to "architecture matters": architecture can also hurt, and a strong backend plus a simple loop can beat it.
3.6 Efficiency and failure modes
GLM-5.2 pairs strong results with a modest budget (Fig. 5): the fastest of the strong solvers at 9.6 K
tokens/cell, edging both Opus configs on speed. strix is catastrophically wasteful: 0.7 M–6.0 M
tokens/cell (up to ~620× GLM-5.2) for fewer solves. Failure-mode signatures differ by scaffold
(Fig. 6): pentestgpt losses are mostly timeouts (83%), the Type-B
signature (the model works the full budget rather than refusing), though Qwen3.6-35B also submits 17 false flags.
pentest-swarm/Qwen3.5 is a false-positive outlier (93 fabricated flags, 0 real solves): a broken
verdict path, not a capable agent.
4. Benchmark integrity
XBOW flags are deterministic: flag = FLAG{sha256(id)}. An agent that learns its challenge
id could compute the flag instead of exploiting. We audited every solved transcript
(142 across GLM-5.2 and Qwen3.6-thinking) for (a) the bare id string in any agent-visible
content (it never appears, not even in server responses; targets are addressed by app alias only) and (b) any
sha256/hashlib computation over the id (none). Scoring is exact-match on the
unguessable 64-hex flag. Spot-checks confirm flags originate from tool/server output (race scripts,
curl to exploited endpoints, session forgery), not model prose. The runs are blind
(container-sandboxed). Conclusion: clean, no compute-the-flag shortcut.
5. Engineering notes
Scaffold identity. The pentestgpt scaffold is Excalibur's core with the v2
modules removed; it runs v1-class, with tools supplied by the SDK backend. Reasoning-model
bridge. Reasoning models return reasoning_content, which the router proxy (Claude Code
Router [6]) turns into unsigned "thinking" blocks the CLI rejects, and they
interleave text+tool_calls in one streamed turn. Fix: enable_thinking:false upstream, force
stream:false so the proxy re-emits blocks in order, and blind the proxy to
reasoning_content; zero content-block crashes across 300+ cells afterward.
Infra. Heavy parallel builds can wedge the container buildkit/VM; a daemon restart is the
reliable recovery. Do not prune the build cache mid-run.
6. Limitations
(1) Mostly single-trial. GLM-5.2 is replicated n = 3 (87.7 ± 4.9; Table 2), which bounds its run-to-run variance, but the other rows are single runs. Agents are stochastic (±several solves, mostly on L2/L3), so non-GLM rankings within ~5 solves should be read as ties pending replication. (2) Single backend for the ablation (Claude Agent SDK, GLM-5.2 only); the backend claim needs ≥2 backends to generalize. (3) Mixed provenance: the 102 row is grey-box; the blind headlines are 101 (Opus) and 90 (GLM-5.2). (4) Scope: XBOW is web single-flag CTF; multi-host AD (where v2's planner is designed to help) is untested here. (5) Token accounting for SDK-backed scaffolds is partial. (6) Cross-paper Opus version: our Opus rows use Opus 4.8, whereas Deng et al. [1] report v2 on Opus 4.5, so the 101 vs. 91% comparison spans two Opus releases and is indicative, not exact.
7. Conclusion
On XBOW web-exploitation CTF, GLM-5.2 is a genuine near-Opus performer (90/104, hard-tier 7/8, blind, audited), the first non-Claude model to clear the hard tier at scale. Two architectural lessons follow: the agent backend is a first-order variable (a lean agent on a strong SDK reaches v2-class scores), and a bespoke planner/memory architecture can underperform a simple loop when mismatched to the task (v2 72 vs. v1-class 90, the whole gap on the medium tier, from over-exploration that burns the budget). These complement the Type-A/Type-B framing with a corollary: more architecture is not free, and substrate choice may dominate it.
Artifact availability
We release the full artifact: the benchrunner harness, all agent scaffolds (including the grafted
v2), the consolidated
score matrix, the audit recipe (ARTIFACT.md), and this paper's LaTeX + figure sources. A single-cell run is python -m benchrunner run
--ids XBEN-001-24 --agents pentestgpt --providers <model>; the v1-vs-v2 ablation swaps
--agents pentestgpt for --agents excalibur. Flags are deterministic (sha256(id)), so
results are verifiable independent of our transcripts. Everything is at https://github.com/ZeeshanSultan/pentest-agent-vs-llm-benchmark-effectiveness; see
ARTIFACT.md for setup, scoring, and the audit recipe.
References
- G. Deng et al. What Makes a Good LLM Agent for Real-world Penetration Testing? arXiv:2602.17622, 2026.
- G. Deng et al. PentestGPT: Evaluating and Harnessing LLMs for Automated Penetration Testing. USENIX Security, 2024.
- XBOW. Validation Benchmarks (104 web-exploitation CTFs). https://github.com/xbow-engineering/validation-benchmarks; platformhttps://xbow.com/ , 2024.
- R. Coulom. Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search. Computers and Games, 2006.
- Anthropic. Claude Agent SDK. https://docs.anthropic.com/en/api/agent-sdk , 2025.
- musistudio. Claude Code Router. https://github.com/musistudio/claude-code-router , 2025.
- usestrix. Strix: autonomous AI security agents. https://github.com/usestrix/strix , 2025.
- HexStrike AI. HexStrike-AI: MCP-driven offensive security framework. https://github.com/0x4m4/hexstrike-ai , 2025.
Notes
- Code, harness, and score matrices: https://github.com/ZeeshanSultan/pentest-agent-vs-llm-benchmark-effectiveness ↩
- Exact served builds may differ from public checkpoints of the same name. This does not affect scoring: flags are unguessable sha256(id) values retrieved from the target, so a solve is endpoint-independent (§4). ↩
Findings
The study at a glance
104 web CTFs
The XBOW validation suite of single-flag web-exploitation challenges.
9 scaffolds, 10 models
Agent scaffolds and backbone models varied independently, to separate their effects.
2,700+ audited runs
Every transcript captured and scored on exact flag retrieval, with 142 solves audited.
The scaffold decides
Same model, different scaffold: from 0 to 49 solves on the same challenges.
More architecture can hurt
In a controlled ablation, a bespoke planner scored 72 against 90 for a leaner agent on the same model.
Open and reproducible
Code, data and the paper are public on GitHub.
Products