Testing vulnerability scanner products
A scanner is judged on the findings it reports, so a missed or spurious finding in a new release reaches every customer scan at once.
Research desk, cybersecurityqavendors.com. Published 2026-11-11.
Short answer
Build a lab of targets with recorded vulnerable and patched states, scan it with every release and every check update, and compare missed findings, spurious findings, scan duration and resource use with the previous release. Ask a QA vendor for a case where it ran this comparison for a scanner or a similar security product.
Why scanner releases regress
A vulnerability scanner's output is a list of findings, and customers act on each one. A new release can change how a check fingerprints a service version, parses a banner or reads the installed package list. Any of these changes can drop a finding on a vulnerable host or add one on a patched host, and no error is raised.
When a customer feeds scanner output into a CTEM1 programme, findings set the remediation queue. A spurious finding sends an engineer to fix a host that is already patched. A missed finding leaves an exposure off the queue.
Build a target lab
- Known-vulnerable targets. Build hosts, containers and web applications with a recorded list of vulnerable package versions and misconfigurations, one expected finding per item.
- Known-patched twins. Build a patched copy of each target with the same services and banners. Any finding on the twin that is not in its expected-results file is spurious; the file also lists findings the product documents as expected, such as a banner-only check against a backported patch.
- Expected-results file. Store the expected findings per target in version control beside the target build scripts, with an owner who approves every change.
- Spread of targets. Cover each operating system family, service type and network position your customers scan, including targets behind a firewall that drops probes.
- Rebuild from scripts. Rebuild the lab from scripts before each run, so drift in a target is not mistaken for a scanner change.
Compare each release with the last
Scan the lab with the release candidate and with the current release, using the same scan profile. Diff both result sets against the expected-results file. Report each missed finding and each spurious finding by check, target and release.
A check update that starts flagging a patched twin is a false-positive regression2. Block the release until someone decides in writing whether the check, the target or the expected result is wrong. The same decision record applies to rule-based products, as described in detection regression testing.
Record scan duration, scanner CPU and memory use, and the load each scan puts on targets. Set a budget per scan profile and fail the run when a release exceeds it. The method for resource budgets on hosts is in performance testing security agents.
Authenticated scans and check updates
Run both modes on each target, because each mode runs different code and fails in different ways. An authenticated scan logs in and reads installed packages and configuration; an unauthenticated scan sees only open ports, banners and responses.
For authenticated scans, test each credential type the product supports, plus a wrong credential and an expired one. A failed login should appear in results as a failed login, not as a host with no findings.
Check or plugin updates ship more often than scanner releases. Run the lab against every check update, not only against product releases. Automate the scan and the diff in the release pipeline; the split between automated and manual work is in test automation for security products.
What to ask a QA vendor
Request evidence that the vendor built or maintained a target lab with expected results for a scanner or a similar security product. Test-type depth, dimension 3, can credit the lab diff as automation, the scan budgets as performance or load, and the patched-twin checks as detection or false-positive regression. Evidence depth, dimension 1, counts the published case, and one naming the target count and check update cadence shows the lab's scale.
A target lab gains value with every check update it has been run against, so ask how long the vendor kept that client: engagement longevity, dimension 4. Lab images contain deliberately vulnerable hosts, so ask which network segment holds them, with scanner builds and scan results, and who holds access: security posture, dimension 6.
Penetration testing of a client's systems is not scanner QA and is never counted as evidence, as the methodology states. No vulnerability management facet page is published; the facet is below threshold on not published.
- 1 CTEM: A repeating program, not a product, for finding, ranking, testing and fixing the weaknesses that matter most to an organization, in five stages. Source
- 2 false-positive regression: A case where a security tool starts flagging harmless activity as malicious after an update, when it did not do so in an earlier version. Source
Questions buyers ask
How do you measure scanner false positives and false negatives?
Scan a lab of known-vulnerable targets and their patched twins, then diff the results against a stored expected-findings file. A finding that is not in a target's expected-results file is spurious; an expected finding that is absent is missed.
What is expected-findings regression testing for a scanner?
Scanning targets whose vulnerable and patched states are recorded in advance, and comparing each release's findings with that record, check by check and target by target.
How often should a scanner be regression tested?
Run the lab on every release candidate and on every check or plugin update, because check updates ship separately from the scanner and can change results on their own.
Should the test lab cover authenticated and unauthenticated scans?
Yes. The two modes run different code paths. Scan each target in both modes, and include wrong and expired credentials so a failed login is reported as a failure, not as a clean host.
What belongs in a scanner benchmark test plan?
The target list with expected findings, the scan profiles, the previous release as baseline, budgets for scan duration and resource use, and the rule for blocking a release on a new missed or spurious finding.
Does a penetration test count as QA for a scanner product?
No. A penetration test assesses a client's systems and does not compare scanner releases against expected results. The ranking does not count penetration testing as evidence.
What should I ask a QA vendor about scanner testing?
Ask for a case with a target lab and expected results, the check update cadence it tested against, how long the engagement ran, and where lab images and scan results were stored.