Research we publish, tools we open-source

The model is not the agent.

Backbone or Backbone-Architecture? A Controlled Study of LLM Agents on Web Penetration-Testing CTFs

A controlled study, by Cyphlon CEO Zeeshan Sultan, of what drives LLM agent performance in web penetration testing: the model, or the scaffold around it. Across the 104 single-flag CTFs of the XBOW validation suite, 9 agent scaffolds and 10 backbone models, more than 2,700 runs were scored on exact flag retrieval and every solve was audited. Holding the model fixed and changing only the scaffold moved the same backbone from 0 to 49 solves.