Research

Backbone or Backbone-Architecture? A Controlled Study of LLM Agents on Web Penetration-Testing CTFs

The model is not the agent.

About the study

A controlled study, by Cyphlon CEO Zeeshan Sultan, of what drives LLM agent performance in web penetration testing: the model, or the scaffold around it. Across the 104 single-flag CTFs of the XBOW validation suite, 9 agent scaffolds and 10 backbone models, more than 2,700 runs were scored on exact flag retrieval and every solve was audited. Holding the model fixed and changing only the scaffold moved the same backbone from 0 to 49 solves.

Full paper

Backbone or Backbone-Architecture? A Controlled Study of LLM Agents on Web Penetration-Testing CTFs

Zeeshan Sultan · June 2026, working draft · PDF

Abstract

We evaluate LLM penetration-testing agents on the XBOW validation benchmark (104 single-flag web-exploitation CTFs) across nine agent scaffolds and ten backbone models, including five 2026 frontier models not previously measured on this benchmark. Three results stand out. (1) GLM-5.2 averages 87.7 ± 4.9/104 over three trials (best 91, ≈84%), clearing 4–7/8 on the hardest tier, within reach of Claude Opus (101–102/104) and the only non-Claude model to clear the hard tier at scale. (2) The agent backend is a first-order variable: a deliberately lean agent on the Claude Agent SDK reaches scores comparable to a published "v2" agent with a bespoke planner and memory subsystem. (3) In a controlled ablation (same model, same targets, same backend; only the scaffold architecture differs), the bespoke v2 architecture underperforms the lean agent, 72 vs. 90/104, with the entire gap on the medium tier: direct evidence that planning machinery tuned for multi-host campaigns can hurt on single-flag CTF. We contribute a reusable benchmark-integrity protocol for deterministic-flag suites (142 solved transcripts audited clean) and a reproducible recipe for running reasoning models through the Claude Agent SDK. We release all code, the harness, and the consolidated score matrices.1 We are explicit about the main limitation: most cells are single-trial (GLM-5.2 is replicated n = 3).

1. Introduction

LLM agents for penetration testing improve rapidly, yet reported success rates swing widely with the scaffold (orchestration + tools), the backbone model, and the benchmark. Deng et al. [1] usefully separate Type-A failures (capability gaps, fixable by tooling) from Type-B failures (planning / state-management limits that persist regardless of model), and present a "v2" agent (codename Excalibur) with a Task-Difficulty-Assessment planner over an Evidence-Guided Attack-Tree Search (TDA-EGATS; an attack-tree analogue of Monte-Carlo Tree Search [4]), a memory subsystem, and a 38-tool typed Tool/Skill Layer, reaching 91% on XBOW with Claude Opus 4.5.

This paper complements that work on three axes the literature has not isolated. New models: we report the first XBOW numbers for five 2026 frontier models. Backend substrate: we separate the contribution of the agent backend (the Claude Agent SDK, which itself supplies tools and planning) from architectural modules layered on top. A controlled ablation: we run the published v2 codebase and a lean v1-class agent on the same model, same targets, same backend, leaving scaffold architecture as the sole variable, a comparison prior work cannot make because each system ships one fixed scaffold.

Contributions.

  • First XBOW benchmark of GLM-5.2, Kimi-K2.7, DeepSeek-V4-Pro, and two Qwen3.6 variants; GLM-5.2 ≈ Claude Opus on web exploitation, including the hard tier.
  • Evidence that backend substrate ≥ bespoke architecture for this task.
  • A controlled v1-class-vs-v2 result in which the bespoke architecture loses.
  • A benchmark-integrity protocol (flag = sha256(id)) and an audit of 142 solves verifying no flag-computation shortcut.
  • A reproducible fix for reasoning models on the Claude Agent SDK / router bridge.

PentestGPT [2] established the LLM-driven CTF/HTB agent; AutoPT, VulnBot, PentestAgent, strix [7], and the HexStrike [8] line vary the tool layer and orchestration (surveyed in [1]). Deng et al. [1] contribute the Type-A/Type-B taxonomy and the v2 agent that motivates our ablation; we run their v2 code, not a reimplementation. Benchmarks probe different skills: XBOW [3] (web CTF, used here), HTB/VulnHub machines, and GOAD (multi-host Active Directory). Prior XBOW numbers are reported for Claude/GPT/Gemini-class models on each system's own scaffold; we add 2026 non-Claude frontier models, an explicit backend-vs-architecture isolation, and a same-everything-but-architecture ablation.

2. Method

Benchmark. XBOW: 104 web-exploitation CTFs in three tiers: L1 (easy, 45), L2 (medium, 51), L3 (hard, 8). Each flag is FLAG{sha256(XBEN-ID)}, baked into the target at build time; the agent must exploit the target to retrieve it. We vendor the exact 104 challenge sources in the artifact (XBOW Validation Benchmarks, © XBOW USA Inc.; the benchmark.yaml autobuild snapshot, 2025-06-16), so the target build ships with the code rather than depending on an external pin.

Harness. Our benchrunner starts one target at a time (or N in parallel), hands the agent only the target URL and a generic instruction, captures the full transcript, and scores exact-flag matches: a solve requires the precise 64-hex flag to appear in agent output; any other flag-shaped string is a false positive. Per-cell wall-clock cap 30 min; setup cap 15 min.

Scaffolds. pentestgpt (an agentic build on the Claude Agent SDK backend, §3.4), excalibur (the published v2), strix, hexstrike-ai/-modified, pentestagent, pentest-swarm-ai, decepticon, and hexstrike-mod-mcp: the modified HexStrike MCP server (192 tools) driven by the Claude CLI, running on the host with shell + docker access (grey-box; the only non-sandboxed configuration).

Models. Claude Opus 4.8; 2026 frontier models GLM-5.2, Kimi-K2.7, DeepSeek-V4-Pro, Qwen3.6-40B-Heretic, Qwen3.6-35B-A3B and its -thinking variant, Mimo-v2.5, Qwen3.6-Uncensored; plus Qwen3.5 (ten distinct backbones). The non-Claude models were served through an OpenAI-compatible gateway;2 Opus ran through the Claude Code CLI on an Anthropic subscription.

Provenance (important). All scaffolds except one are blind: container-sandboxed, no host or docker access. The sole exception is hexstrike-mod-mcp/opus, the modified HexStrike MCP server run on the host with shell + docker access, which is grey-box; up to 14 of its solves used docker exec/file reads on the target rather than pure HTTP, so its strict-black-box range is [88, 102]. We keep it in the table for completeness but treat the blind rows as the trustworthy axis and never compare grey-box to blind numbers directly.

3. Results

Bar chart of XBOW-104 solves by tier for every agent and model configuration.
Figure 1. XBOW-104 solves by tier. The hard-tier (red) segment is large only for the two Opus configs, GLM-5.2, and v2/GLM-5.2; Qwen3.6-35B, its thinking variant and Mimo-v2.5 clear 1–2 of 8.

3.1 Leaderboard

Figure 1 and Table 1 rank the main configurations; all 26 full-suite configurations are in results-archive/score_matrix_20260618-consolidated.csv. Tiers are sharp: Opus (101–102), then GLM-5.2 at 90, then a cluster of open/Chinese models at 47–52, then a long tail. The top row (102) is the modified HexStrike MCP server with host shell + docker access: grey-box, not black-box, and not directly comparable to the sandboxed rows. The canonical blind leader is pentestgpt/opus at 101.

Table 1. XBOW-104, single trial. † grey-box: the modified HexStrike MCP server run on the host with shell + docker access, not black-box; all other rows are container-sandboxed (blind). Mimo-v2.5 rows on strix, hexstrike-ai and pentestagent include cells that errored while the model backend was down (70 cells across the Mimo runs, per the score matrix), so they may understate those agents.
AgentModelPassL1L2L3
hexstrike-mod-mcp† opus 102 44 50 8
pentestgpt opus 101 44 49 8
pentestgpt GLM-5.2 90 42 41 7
excalibur (v2) GLM-5.2 72 42 24 6
pentestgpt Qwen3.6-35B-think 52 27 23 2
pentestgpt Qwen3.6-35B 49 29 19 1
pentestgpt Kimi-K2.7 49 31 18 0
pentestgpt Mimo-v2.5 47 28 17 2
strix Qwen3.6-35B 38 26 12 0
strix Mimo-v2.5 33 23 10 0
pentestgpt Heretic-40B 32 21 11 0
hexstrike-ai Mimo-v2.5 30 17 13 0
pentestgpt DeepSeek-V4-Pro 30 24 6 0
pentestagent Mimo-v2.5 26 19 7 0
pentestgpt Qwen3.6-Uncens. 24 15 9 0
hexstrike-ai Qwen3.6-35B 9 8 1 0
pentest-swarm Qwen3.5-35B 0 0 0 0

3.2 GLM-5.2 ≈ Claude Opus

GLM-5.2 on pentestgpt averages 87.7 ± 4.9/104 over three trials (individual runs 90, 82, 91; Table 2), third overall behind only Opus, and is the only non-Claude model to clear the hard tier at scale (4–7/8); every other non-Claude config scores 0–2 on L3 (Fig. 2). It is also the fastest of the strong solvers (3.5 min/solve, faster than both Opus configs) at a modest 9.6 K tokens/cell, with zero false positives. This is a blind result and was cheat-audited (§4). The leaderboard (Table 1, Fig. 1) shows its first run (90) for comparability with the single-trial rows.

Table 2. GLM-5.2 variance over three independent full-104 runs (PentestGPT/SDK, blind). L1 is near-deterministic; the hard tier is the most variable in relative terms but never collapses.
RunTotalL1 (45)L2 (51)L3 (8)
19042417
28241374
39142427
mean ± sd87.7 ± 4.941.7 ± 0.640.0 ± 2.66.0 ± 1.7

The variance is real but does not move the qualitative claims: across all three runs GLM-5.2 stays third overall (the best non-Claude model), well clear of the 47–52 cluster, and is the only non-Claude model to solve more than 2 of 8 on L3 (4–7 of 8). Easy cells (L1) are essentially deterministic (±0.6); the medium and hard tiers carry the variance, as expected where a single flag can hinge on one exploitation step landing within the time budget.

Chart of hard-tier (L3) solves out of 8 for each configuration.
Figure 2. The hard tier (L3, 8 challenges): only Claude Opus and GLM-5.2 solve it at scale.
Chart of solves for one backbone model across seven scaffolds, from 0 to 49.
Figure 3. One backbone (Qwen3.6-35B-A3B), seven scaffolds: 0 → 49 solves. Scaffold and backend choice swings the same model by 49 solves, larger than most model-to-model gaps.

3.3 Scaffold is a first-order variable

Before isolating the backend, note the raw effect of the scaffold. Holding the backbone fixed at Qwen3.6-35B-A3B and varying only the scaffold (Fig. 3), solves range from 0 to 49, a 49-solve swing on an identical model, wider than the spread across most backbone models. Two scaffolds (pentest-swarm, decepticon) score 0; pentestgpt scores 49. The scaffold is not a second-order detail.

3.4 Backend substrate vs. the paper's tiers

Our pentestgpt scaffold is a lean agent on the Claude Agent SDK [5] backend. On Opus it reaches 97% (101/104), near Deng et al.'s v2 (91%, Opus 4.5) and far above their v1 baselines (47–60%). Yet a source diff shows this scaffold has none of v2's planner/memory/tool-layer modules: its core/backend.py is identical to Excalibur's apart from its environment-variable prefix, but planner/, memory/, and tools/{categories,skills} are absent. The implication: the Claude Agent SDK backend itself supplies Type-A tooling and partial Type-B planning (Bash, file ops, task planning, sub-agents), reaching v2-class scores without v2's explicit architecture. The backend substrate is a first-order variable.

3.5 Controlled ablation: v1-class vs. true v2

We grafted the published v2 (Excalibur) into the harness and ran GLM-5.2 on it. Same model, same targets, same backend; the only difference is scaffold architecture (Fig. 4). v2 scores 72/104 vs. the lean agent's 90, a loss of 18 solves, and the entire gap is the medium tier (L2: 24 vs. 41). v2 ties on easy cells (L1 42–42) and is within one on hard cells (L3 6–7). The mechanism is visible in transcripts: v2's planner keeps exploring after the flag is in hand (controller log: "Flags found, continuing to verify completeness"), so every v2 cell runs to the 30-min timeout (~1800 s vs. the lean agent's ~212 s). On easy cells the flag is captured fast enough that over-exploration is harmless; on medium cells the wandering exhausts the budget before convergence. The planner is designed for multi-host AD campaigns; applied to single-flag web CTF it is a mismatch. This is a controlled corollary to "architecture matters": architecture can also hurt, and a strong backend plus a simple loop can beat it.

Chart comparing v2 and the lean agent on GLM-5.2 by tier.
Figure 4. Controlled ablation. Same model (GLM-5.2), targets, and backend; only the scaffold architecture differs. v2 ties on L1 and loses the whole gap on L2.

3.6 Efficiency and failure modes

GLM-5.2 pairs strong results with a modest budget (Fig. 5): the fastest of the strong solvers at 9.6 K tokens/cell, edging both Opus configs on speed. strix is catastrophically wasteful: 0.7 M–6.0 M tokens/cell (up to ~620× GLM-5.2) for fewer solves. Failure-mode signatures differ by scaffold (Fig. 6): pentestgpt losses are mostly timeouts (83%), the Type-B signature (the model works the full budget rather than refusing), though Qwen3.6-35B also submits 17 false flags. pentest-swarm/Qwen3.5 is a false-positive outlier (93 fabricated flags, 0 real solves): a broken verdict path, not a capable agent.

Scatter plot of solves against mean tokens per challenge on a log scale, one point per model.
Figure 5. Solves vs. mean tokens/challenge (log x) on a single scaffold (PentestGPT/SDK, one point per model). GLM-5.2 (green) comes closest to Opus-tier solves at 9.6 K tokens/cell, well below the Qwen3.6-35B variants (24–26 K). (Cross-scaffold, strix reaches 0.7–6.0 M tokens/cell, off this chart's scale.)
Stacked bar chart of outcomes per model: solves, timeouts, errors and false flags.
Figure 6. Outcome composition on the PentestGPT/SDK scaffold. Most non-solves are timeouts (Type-B, 83% across the PentestGPT rows); Qwen3.6-35B also submits 17 false flags.

4. Benchmark integrity

XBOW flags are deterministic: flag = FLAG{sha256(id)}. An agent that learns its challenge id could compute the flag instead of exploiting. We audited every solved transcript (142 across GLM-5.2 and Qwen3.6-thinking) for (a) the bare id string in any agent-visible content (it never appears, not even in server responses; targets are addressed by app alias only) and (b) any sha256/hashlib computation over the id (none). Scoring is exact-match on the unguessable 64-hex flag. Spot-checks confirm flags originate from tool/server output (race scripts, curl to exploited endpoints, session forgery), not model prose. The runs are blind (container-sandboxed). Conclusion: clean, no compute-the-flag shortcut.

5. Engineering notes

Scaffold identity. The pentestgpt scaffold is Excalibur's core with the v2 modules removed; it runs v1-class, with tools supplied by the SDK backend. Reasoning-model bridge. Reasoning models return reasoning_content, which the router proxy (Claude Code Router [6]) turns into unsigned "thinking" blocks the CLI rejects, and they interleave text+tool_calls in one streamed turn. Fix: enable_thinking:false upstream, force stream:false so the proxy re-emits blocks in order, and blind the proxy to reasoning_content; zero content-block crashes across 300+ cells afterward. Infra. Heavy parallel builds can wedge the container buildkit/VM; a daemon restart is the reliable recovery. Do not prune the build cache mid-run.

6. Limitations

(1) Mostly single-trial. GLM-5.2 is replicated n = 3 (87.7 ± 4.9; Table 2), which bounds its run-to-run variance, but the other rows are single runs. Agents are stochastic (±several solves, mostly on L2/L3), so non-GLM rankings within ~5 solves should be read as ties pending replication. (2) Single backend for the ablation (Claude Agent SDK, GLM-5.2 only); the backend claim needs ≥2 backends to generalize. (3) Mixed provenance: the 102 row is grey-box; the blind headlines are 101 (Opus) and 90 (GLM-5.2). (4) Scope: XBOW is web single-flag CTF; multi-host AD (where v2's planner is designed to help) is untested here. (5) Token accounting for SDK-backed scaffolds is partial. (6) Cross-paper Opus version: our Opus rows use Opus 4.8, whereas Deng et al. [1] report v2 on Opus 4.5, so the 101 vs. 91% comparison spans two Opus releases and is indicative, not exact.

7. Conclusion

On XBOW web-exploitation CTF, GLM-5.2 is a genuine near-Opus performer (90/104, hard-tier 7/8, blind, audited), the first non-Claude model to clear the hard tier at scale. Two architectural lessons follow: the agent backend is a first-order variable (a lean agent on a strong SDK reaches v2-class scores), and a bespoke planner/memory architecture can underperform a simple loop when mismatched to the task (v2 72 vs. v1-class 90, the whole gap on the medium tier, from over-exploration that burns the budget). These complement the Type-A/Type-B framing with a corollary: more architecture is not free, and substrate choice may dominate it.

Artifact availability

We release the full artifact: the benchrunner harness, all agent scaffolds (including the grafted v2), the consolidated score matrix, the audit recipe (ARTIFACT.md), and this paper's LaTeX + figure sources. A single-cell run is python -m benchrunner run --ids XBEN-001-24 --agents pentestgpt --providers <model>; the v1-vs-v2 ablation swaps --agents pentestgpt for --agents excalibur. Flags are deterministic (sha256(id)), so results are verifiable independent of our transcripts. Everything is at https://github.com/ZeeshanSultan/pentest-agent-vs-llm-benchmark-effectiveness; see ARTIFACT.md for setup, scoring, and the audit recipe.

References

  1. G. Deng et al. What Makes a Good LLM Agent for Real-world Penetration Testing? arXiv:2602.17622, 2026.
  2. G. Deng et al. PentestGPT: Evaluating and Harnessing LLMs for Automated Penetration Testing. USENIX Security, 2024.
  3. XBOW. Validation Benchmarks (104 web-exploitation CTFs). https://github.com/xbow-engineering/validation-benchmarks; platformhttps://xbow.com/ , 2024.
  4. R. Coulom. Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search. Computers and Games, 2006.
  5. Anthropic. Claude Agent SDK. https://docs.anthropic.com/en/api/agent-sdk , 2025.
  6. musistudio. Claude Code Router. https://github.com/musistudio/claude-code-router , 2025.
  7. usestrix. Strix: autonomous AI security agents. https://github.com/usestrix/strix , 2025.
  8. HexStrike AI. HexStrike-AI: MCP-driven offensive security framework. https://github.com/0x4m4/hexstrike-ai , 2025.

Notes

  1. Code, harness, and score matrices: https://github.com/ZeeshanSultan/pentest-agent-vs-llm-benchmark-effectiveness ↩
  2. Exact served builds may differ from public checkpoints of the same name. This does not affect scoring: flags are unguessable sha256(id) values retrieved from the target, so a solve is endpoint-independent (§4). ↩

Findings

The study at a glance

01

104 web CTFs

The XBOW validation suite of single-flag web-exploitation challenges.

02

9 scaffolds, 10 models

Agent scaffolds and backbone models varied independently, to separate their effects.

03

2,700+ audited runs

Every transcript captured and scored on exact flag retrieval, with 142 solves audited.

04

The scaffold decides

Same model, different scaffold: from 0 to 49 solves on the same challenges.

05

More architecture can hurt

In a controlled ablation, a bespoke planner scored 72 against 90 for a leaner agent on the same model.

06

Open and reproducible

Code, data and the paper are public on GitHub.

Products

More from Cyphlon

See all products →

Book a call

Book a 30-minute call

Pick a time that suits you. We'll talk through what you're building and where your security stands, then suggest where to start.

Cyphlon

1:1 Consultation Call

30 min Google Meet

An introductory call with our engineers about any engagement: a pentest, continuous testing, managed AppSec, security engineering, agentic AI security or a fractional CISO. We'll ask about your stack, your deadlines and what your customers or auditors expect, and tell you plainly if we're not the right fit.