AI agents have found real weaknesses under controlled conditions on a live university network. For a business application, the buying decision comes down to who owns the coverage, which workflows get examined, and what evidence engineering can use.
Imagine you run a payments platform and two proposals land before a major release. One promises broad automated coverage of your public application and API. The other spends more time on the application’s approval and payout workflows. Both cover the same domains. Only one proposal says how it will assess whether a payout can occur without the required independent approval. That difference is worth more to your release decision than a comparison of how many endpoints each provider can visit.
What the university study tells a buyer
In a paper published at ICLR 2026, Justin W. Lin and colleagues compared ten security professionals, six existing AI agents, and the researchers’ own agent system, ARTEMIS, on a live university network. The scope spanned roughly 8,000 hosts across 12 subnets. The higher-scoring ARTEMIS configuration ranked second overall under the study’s scoring system, which weighted the complexity and criticality of findings. It had nine valid findings, and 82 percent of its submissions were judged valid. The other agents varied in performance, and the authors reported weaknesses with graphical interfaces. Lin and colleagues’ ICLR paper, Table 1 and sections 3 and 6
The design matters. This was one network, with a small participant pool, up to ten hours of active engagement and four days of access. University IT knew about the exercise and manually approved some flagged activity that would otherwise have been blocked. The study gives buyers evidence that AI can contribute to serious assessment work in a live environment. It leaves open how a given provider will cover the rules that govern your specific product. The ranking describes performance in that study and depends on what the researchers measured. Study limitations and methodology
Hexkey’s view is that the division of work should follow what you need to learn. Automation is a good candidate for broad, repeatable coverage across similar surfaces. Workflows with consequential exceptions deserve a practitioner who understands the business rule, makes the judgment about evidence, and owns the result. The study’s uneven performance across agents is a good reason to ask what the vendor’s actual process covers.
Put business rules in the scope of work
A scope that names the domains, test accounts, and dates gives the tester permission to work. Add the decisions you expect the report to inform. For the fictional payments platform, start with payout approval, changes to settlement details, and delegated support access. Describe who should be able to act and under what conditions. Include legitimate exceptions, such as a finance supervisor’s ability to release a payment after an approval is reissued. The engineering team should be able to distinguish a violation from a supported workflow without having to infer intent from a screenshot.
Ask the provider to explain which access it needs to assess those rules. A browser interface, an API used by a mobile app, and a scheduled worker can all participate in the same transaction. Source access can help establish where a control lives and which interfaces share it. Accounts representing meaningful roles and realistic test records allow the practitioner to assess how it behaves. OWASP’s Web Security Testing Guide treats business logic as a distinct testing area. Its Application Security Verification Standard gives buyers a way to name relevant control requirements in the scope. OWASP guidance on business logic testing · OWASP Application Security Verification Standard
Decide where the assessment can run. A representative environment with synthetic transactions may be a better place to examine payment state changes. A carefully bounded production check may answer a different question about the configuration customers actually use. Agree on test windows, transaction limits, and who can stop activity before testing starts. Where test and production differ in a material way, ask the report to identify that gap so a release owner knows how far the result extends.
If the scope is too large for the window, make the tradeoff explicit. Ask the vendor to estimate what it can cover in a broad pass and to identify the handful of workflows that warrant deeper review. A second client or deployment environment may add little if it uses the same controls and configuration. It may deserve its own coverage if it handles a different role or enforces authorization independently. That is a coverage decision the vendor and the product owner can make together before the clock starts.
Ask to see the handoff before you sign
Request a redacted sample finding and a sample coverage record. One should show the business rule, observed behavior, affected versions or surfaces, and the consequences. The other should show which workflows were exercised, which roles were available, and where time or access limited the assessment. Give both samples to the engineering lead who would be asked to act on them. A report can be technically accurate and still leave that person without enough information to choose a change or estimate its scope.
The useful question about AI is who checks the claims it helps produce. Ask for a named practitioner who can explain the evidence behind any reported issue and can defend the priority assigned to it. Agree on who will review the highest-impact fixes and how a retest will account for legitimate use. Set the expectation that some controls will hold under the tested conditions and that untested parts of the system will be visible in the record. At Hexkey, a named tester of record is responsible for the report. Hexkey’s penetration testing methodology
Once proposals come back, compare them against the buyer decision. If the test is needed for a release, ask whether the expected evidence will let you approve it, change its scope, or delay a specific feature. If it is needed for planning, ask whether the results will tell you which team owns the most material work. Pay for broad coverage when it buys evidence you can act on, and protect enough practitioner time to examine the workflows that would be most expensive to get wrong.
Sources and further reading
- Justin W. Lin and colleagues, Comparing AI Agents to Cybersecurity Professionals in Real-World Penetration Testing, ICLR 2026 conference paper.
- OWASP Foundation, Web Security Testing Guide v4.2, Business Logic Testing, December 3, 2020.
- OWASP Foundation, Application Security Verification Standard 5.0.0, May 30, 2025.
- Hexkey, Penetration Testing, accessed November 20, 2026.
Want a second look at your own applications?
We find and verify what’s exposed, fix what matters, and modernize what can’t be secured piece by piece.

