Five signs you bought a vulnerability scan, not a penetration test

A penetration test is defined by its scope and method, not its tool. How to tell the difference from the report in your hands, how to buy the real thing, and how to run the automated half continuously.

Praveen Kumar

Praveen Kumar

Founder & Technology Leader

10 min read
On this page

An enterprise prospect asks for your latest penetration test. So does the SOC 2 auditor. You send the ninety-page PDF a vendor delivered last spring, and if the prospect's security team is any good, they will read it in ten minutes and know what it is: scanner output with a cover page. Mostly TLS cipher warnings, missing headers and outdated library versions, sorted by CVSS, with no sign that anyone tried to log in as one customer and read another's data.

That document was sold as a penetration test. It was a vulnerability scan. Both have value, and the difference is not the tool. It is whether a skilled person spent time attacking your application's authorisation and business logic, the two places where automation structurally cannot go and where the breaches that make the news actually happen.

Here are the five signs, what to buy instead, and how to run the automated half every week so the annual manual test can concentrate on what only a human finds. The buyer's kit, from the scoping questionnaire to the attestation letter template and a continuous scanning pipeline, is in our engineering notes repository.

The five signs

1. The findings are tool output. Every finding is a CVE or a configuration weakness with a CVSS score pasted from a database, no reproduction steps, no screenshots, no statement of what an attacker would actually gain in your application. Automated findings are described in the scanner's words, not yours.

2. There is nothing about authorisation or business logic. No horizontal privilege test (can user A read user B's object?), no vertical test (can a basic user reach an admin function?), no workflow bypass, no parameter manipulation on prices or quantities, no race on a one-time action. These are the categories the OWASP Web Security Testing Guide labels ATHZ and BUSL, and they are absent because scanners cannot form the hypothesis "this id belongs to someone else".

3. There is no scope and method section worth the name. A real test names what was in scope, what was out, what credentials and roles were used, which standard it tested against and at what level, and which methodology's phases it followed. A scan says "the target URL".

4. Testing was unauthenticated, or with one role. Most of an application lives behind login. A test that never had credentials for each role never saw it.

5. There is no retest, no attestation, and the whole thing took a day. A penetration test ends with findings fixed and verified, and a one-page letter you can show a customer. A scan ends with a PDF.

If three or more of those apply, you have a vulnerability scan. Keep it; it is useful. Just do not hand it to the enterprise prospect as a penetration test.

Why automation cannot find the things that matter

OWASP's 2021 Top 10 put Broken Access Control first, with the project reporting that 94% of applications tested showed some form of it. It stays at the top in the 2025 revision. Access control flaws are also the category automated scanners are weakest at, for a structural reason: to find one, you need to know which objects a given user should not be able to see, and that knowledge lives in your business rules, not in HTTP responses.

A scanner can see that /api/baskets/4711 returns 200. It cannot know that basket 4711 belongs to a different customer. A tester logged in as two users can, in about a minute. The same holds for business logic: a negative quantity that produces a refund, a coupon applied twice, an invitation link that works after the inviter was removed, a password reset that accepts any user's token. None of these is a vulnerability in a library. All of them are vulnerabilities in your product.

Two different activities compared: continuous vulnerability scanning in the pipeline and the annual manual penetration test, with what each finds and the standards that define them

What a penetration test actually is

Three public standards define it; a proposal should name them.

Scope by assurance level: OWASP ASVS. The Application Security Verification Standard, version 5.0 since May 2025, is about 350 requirements across 17 chapters. Level 1 is the minimum, around 20% of the requirements, the first layer of defence against common attacks. Level 2 is where most applications handling sensitive data should be, about 70% of the standard cumulatively. Level 3 is for the highest-value systems. Choosing the level is the single most important scoping decision, because it tells the tester how deep to go on authentication, session management, authorisation, cryptography and the rest.

Method by test category: OWASP WSTG. The Web Security Testing Guide, currently version 4.2, organises testing into categories: information gathering, configuration, identity management, authentication, authorisation, session management, input validation, error handling, cryptography, business logic, client-side and API testing. Insecure direct object references are WSTG-ATHZ-04; business logic has its own ten tests, from data validation and forged requests to workflow circumvention and payment functionality. A proposal that references these is promising coverage you can check against the report.

Phases by engagement: PTES. The Penetration Testing Execution Standard describes seven phases: pre-engagement, intelligence gathering, threat modelling, vulnerability analysis, exploitation, post-exploitation and reporting. Its value for a buyer is that it defines what happens before and after the attacking, which is where scans skip straight past.

Compliance regimes point the same way. PCI DSS version 4.0.1 separates the two activities explicitly: requirement 11.3 wants internal and external vulnerability scans at least every three months, with external scans by an approved scanning vendor, while requirement 11.4 wants penetration testing at least once every twelve months and after significant changes, following a documented, industry-accepted methodology, covering the application layer, performed by a qualified and organisationally independent tester, with findings corrected and the test repeated to verify. SOC 2 contains no explicit penetration-testing requirement at all; auditors look for vulnerability management under CC7.1 and evaluations under CC4.1, which is why a scan often satisfies the auditor and not the customer. ISO/IEC 27001:2022 reaches it through the controls on technical vulnerability management and security testing.

What the two find: a worked example

To make the difference concrete, the kit targets OWASP Juice Shop, the deliberately vulnerable web shop OWASP maintains for exactly this purpose. It runs from source on Node.js in a few minutes and documents every planted flaw as a challenge.

The comparison below is drawn from Juice Shop's published challenge catalogue and the ZAP rule set, both public, so every row can be checked against its source.

What the automated scan reportsWhat the manual test finds
Missing or weak security headers, cookie flags, information disclosure in error pagesHorizontal privilege escalation: the "View Basket" challenge, where changing the basket id in the request returns another customer's basket
Outdated JavaScript libraries with known CVEsForged content on behalf of another user: the "Forged Feedback" challenge, where the user id in a feedback submission is attacker-controlled
Reflected input and some injection points on obvious parametersToken logic flaws: the JWT challenges, where a token with its signature algorithm set to none or re-signed with a public key is accepted
TLS configuration observationsWorkflow and business-logic abuse: negative quantities, coupon manipulation, and ordering flows that skip a step

The left column is worth fixing and worth automating. The right column is what loses a customer's data, and nothing on the left would have hinted at it.

Buying the real thing

Scope first, then ask for proposals. The questionnaire in the kit covers what is being tested and why, the ASVS level, the testing style (black, grey or white box), the business flows that matter most, constraints, and deliverables. Then put the vendor through twelve questions, of which four do most of the work:

  • Which methodology and standard, at what level? Expect WSTG or PTES and an ASVS level in writing.
  • How many days are manual? A one-day engagement on a real application is a scan.
  • Will you test authorisation and business logic with every role we provide, and how?
  • Show us a redacted finding. It should have reproduction steps, evidence, business impact, severity reasoning and a specific fix.

Sign rules of engagement before anything starts: authorisation, scope, credentials, method, constraints, who to call when something critical is found, how evidence is handled, what is delivered and when. The template in the kit is a page and a half; it is also what makes the testing legal.

A finding should look like the template in the kit: a specific title, severity with the reasoning, a reference to the standard it violates, impact in business terms, affected component, step-by-step reproduction with evidence, root cause, a testable recommendation, and a retest record. An engineer should be able to fix it from the finding alone.

Finish with a retest of everything rated high or above and an attestation letter: scope, method, standard and level, summary counts, confirmation of remediation, limitations, signed by a named tester. That one page is what the prospect and the auditor actually read.

The continuous half

None of this means scanning is beneath you. It means scanning belongs in a pipeline, running every week against staging with credentials, so the humans you pay annually never spend their first day finding missing headers.

OWASP ZAP's Automation Framework describes a scan as a YAML plan: an environment with contexts, URLs and authentication, then jobs in order, typically a spider, an AJAX spider for single-page applications, a wait for passive scanning, an active scan, reports, and an exit-status job that fails the run on findings above a threshold. The plan in the kit authenticates against staging with a test user, caps the active scan's duration, writes an HTML report for humans and a SARIF file for GitHub code scanning, and fails on anything rated High. It runs locally with ZAP's command-line automation mode and in GitHub Actions through ZAP's official action on a weekly schedule.

Two rules keep it useful rather than noisy. Authenticate, or you are scanning the login page. And triage once, properly: mark the false positives in the plan's alert filters so the weekly run only reports change.

When this is wrong

A pre-revenue product with no sensitive data and no customers asking is better served by static analysis plus a ZAP baseline scan in the pipeline until there is something worth a manual test. A regime that defines the method, as PCI DSS does, overrides anything here. And a company with an internal red team already knows all of this and buys external tests for independence, not discovery.

The checklist

Before you accept a report as a penetration test:

  • The scope section names the environments, roles and credentials used.
  • The method section names the standard (ASVS level) and methodology (WSTG or PTES phases).
  • Findings include authorisation and business-logic categories, or an explicit statement that they were tested and nothing was found.
  • Each finding has reproduction steps, evidence, impact and a specific fix.
  • A retest has been done or is scheduled, and an attestation letter follows it.
  • The tester is named, qualified and independent of the people who built the system.
  • Automated scanning runs continuously in your own pipeline, so the test covered what tools cannot.

The scoping questionnaire, rules of engagement, vendor questions, finding and attestation templates, and the ZAP plan with its workflow are in the engineering notes repository.

Praveen Kumar

Written by

Praveen Kumar

Talk to the team

Use this article

Share

Ready to work with us?

Whether you need a dedicated team, a project partner, or strategic technology guidance, we're here to help.