Security product QA checklist
What an outsourced QA team should cover before a security product release, grouped by product type and mapped to the test types the ranking scores.
Research desk, cybersecurityqavendors.com. Published 2026-11-11.
Short answer
Cover the supported platform matrix, performance against a baseline, upgrade and rollback, integrations, detection and false-positive regression, and the evidence handed over at release. Agents weight compatibility and performance; detection platforms weight regression; identity products weight protocol and permission tests.
Before testing starts
- Supported operating systems, versions and architectures, copied from your product documentation.
- Performance budgets for CPU, memory, disk and boot time on reference hosts.
- Integrations in scope, with the versions of each connected system.
- Benign and malicious sample sets, versioned, with an owner for each.
- The person who signs off a release and the evidence they need.
By product type
Endpoint agents. An EDR1 or EPP2 agent ships to every OS build your customers run. Test install, upgrade from prior versions, rollback, coexistence with other agents, and resource cost per build. The full cycle is in testing EDR agents.
Detection platforms. A SIEM3 or XDR4 depends on parsers and field names. Send test events through the real collection path and fail the build when a rule stops firing.
Identity products. IAM5 and PAM6 products need login, token and permission flows tested across identity providers, with negative cases for denied access.
Data protection. DLP7 and backup products need restore correctness and policy matching tested on seeded data that contains no customer content.
The release checklist
- Platform matrix run. Suggested practice: every supported OS and version in the top tier passes on each build; lower tiers run on a schedule. This is the compatibility test type in dimension 3.
- Performance against baseline. Each build is compared with the last release on the same hosts and workload.
- Upgrade and rollback. Upgrade from at least two prior versions and roll back on each supported OS.
- Detection and false-positive regression. Known-bad samples still detect; known-clean samples still pass. See detection regression testing.
- Integrations. Each connected system in scope passes a smoke test at the stated version.
- Hand-over. Results, open defects with severity, and the build identifiers tested go to the release owner.
Using the checklist to pick a vendor
Each item maps to a test type in dimension 3 of the methodology. A vendor whose evidence covers none of your top three items has shown no work on what you need, whatever its overall score.
- 1 EDR: Software that watches laptops, servers and phones for suspicious behavior, records what it sees, and alerts administrators so they can investigate and respond. Source
- 2 EPP: Software installed on workstations and laptops to block attacks, typically bundling antivirus, a personal firewall and host-based intrusion prevention. Source
- 3 SIEM: A system that collects log data from many sources into one place so security teams can search it, correlate events and spot attacks. Source
- 4 XDR: A detection approach that pulls data from endpoints, email, cloud workloads and servers into one view, then ranks threats so analysts can respond. Source
- 5 IAM: The practice of managing user identities in an organization and deciding which roles and access privileges each person or account gets. Source
- 6 PAM: The discipline of controlling, monitoring and protecting accounts with elevated rights, such as administrators, because they can reach critical systems and sensitive data. Source
- 7 DLP: Controls that identify, monitor and protect sensitive data in use, in transit and in storage, and block unauthorized use or transmission of it. Source
Questions buyers ask
What should be defined before QA starts on a security product?
Supported platforms and versions, performance budgets for the agent or the pipeline, the integrations in scope, the benign and malicious sample sets, and who signs off a release.
Which environments and OS versions belong in the test scope?
Every OS version and architecture listed as supported in your product documentation, plus the next preview build where vendors publish one. Rank combinations by customer share and run the top tier on every build.
How do we plan false positive regression tests?
Keep a versioned corpus of benign software and activity your customers run, replay it on each build, and fail the build when a known-clean item raises an alert. Record every change to the corpus.
What performance baselines should be recorded for an agent?
CPU, memory, disk I/O and boot time on reference hosts with and without the agent, under idle and under a scripted workload. Compare each build against the previous release, not against a fixed number alone.
How do we test updates and rollbacks safely?
Upgrade from at least two prior versions, roll back, and confirm the host and the agent's protection state after each step. Stage updates to a small host group before a wider rollout.
What evidence should QA hand over at release?
The matrix of platforms tested with pass or fail, performance against baseline, detection and false-positive regression results, open defects with severity, and the build identifiers tested.