What we tested, what we missed, and what this doesn't prove

The six SecScan missed

• An SSRF case (needs an out-of-band collector)
• An API endpoint with no authentication
• Function-level access control on an API
• Cross-account access to another user’s data
• A page-level insecure direct object reference
• Guessable, enumerable identifiers

These are the tests that need two logins, an API description or an out-of-band collector. That is where the gap is, and it is why the coverage page lists broken access control as needing a second account.

The benchmark also caught its own bias. ZAP never requested three pages because nothing on the home page linked to them, while SecScan found them from a built-in path list, which made the first scores look better than they were. Those pages are now linked like real features. Earlier, three rounds of adjusting match patterns moved ZAP's score from 59.5% to 47.3% to 31.1% on identical data, which is why scoring moved to blind judges.

All 16 high findings were checked against the live sites the same day and all 16 were real. The 82 medium and low findings in that run were not individually re-checked; earlier rounds verified the same finding types on the same kinds of sites. Two of the highs deserve a caveat: the archive.org source map is a copy of a public library, so what it exposes is that library and not archive.org's own code, and “no Content-Security-Policy” as a high on a site like Wikipedia is a judgment about severity that reasonable people rate lower.

Rescanned on 30 September, after the fixes

The same thirty sites were scanned again once every check had been read line by line for findings that fire on a healthy site. Six answered with a bot-protection page and are withheld, as before. The run returned 296 findings, 102 above Info, 14 high and no critical. Every difference from the 24 September run was checked against the live site, and the header-based findings were then re-derived independently from a separate fetch. Two more false positives were still in that result set and were found by that second pass, not by the scan; both are Info-level and both are on the false positives page. The 16 highs above are the 24 September figures and were not recounted by hand a second time.

And the sweep found a false positive

The longer history — false positives that shipped, ones caught before release, and findings that look wrong but are right — is on the false positives page. Before the 24 September sweep, 111 runs across 39 hosts were on record, and the round before it, on 19 September, found 80 actionable findings on 20 sites, eight of them high, every one verified true, and none false.

• About 2,500 automated tests pass (2,497 at the last full run, on 30 September), covering both directions: a finding appears when the condition is present and is suppressed when it is not.
• A 19-flaw quick check runs against a local vulnerable site and fails the build if any planted flaw goes undetected. A fuller gate points the real scanner at that site with two signed-in accounts and requires every planted finding to be detected: 82 of 82 on a fresh copy, with server-side request forgery checked separately because it needs a public callback address.
• End-to-end scripts drive the MCP server the way an editor does (28 checks on the tools alone), and confirm that an API token cannot reach any account route.
• Twenty known-good sites are scanned before scanner changes, four of them with recorded finding counts that must not move unexplained.
• Every scan records each check as ran, skipped (with the reason) or failed, so a clean report cannot quietly mean “we did not look”.

• No outside party has reproduced any of these results.
• The benchmark measures known, planted faults. It says nothing about unknown flaws in real applications.
• SecScan is an automated scanner. It cannot reason about business logic, and it is not a penetration test.
• Still open: one hard-coded token can produce three separate high findings; server-side request forgery has not been swept for false positives on real sites; and the subresource-integrity check still lists marketing tags that their vendor edits in place, with advice to self-host or restrict them rather than to add a hash.

SecScan is built by one person, and none of this replaces a review by a security professional. It is meant to catch the common, reliably detectable problems early and say plainly what it did not test.