A missed vulnerability costs you one finding. A false positive costs you the report — once a scanner has sent you chasing something that was never wrong, you have no way to tell which of the other findings to believe. So the checks get pointed at sites whose answers are already known, and every finding that survives is read by hand. The test results, including the misses, are on how we test.
Google, GitHub, Cloudflare and Mozilla run their security properly. On targets like those a correct scan returns few findings and zero false ones, which makes every extra finding a lead. Each one is checked against the live response by hand before anything is changed — including the ones that turn out to be right.
These were live. Each one is fixed, with a regression test, and the reasoning is recorded so the fix is not quietly undone later.
The check confirmed on the payload appearing back in the response body, whatever its Content-Type. The same bytes in JSON or plain text are inert, because a browser will not run them as a page. The test app's own error path returned the payload as text/plain and was counted.
Any endpoint that echoes a query parameter in JSON or plain text. Now it must be served as HTML.
The check flagged every script whose hostname differed from the page's, including the site's own CDN (static.example.com) and scripts such as Stripe.js and Google Tag Manager that the provider updates in place, where a pinned hash would break the page.
Any site with a separate asset host or a payment or tag script. On 20 well-run sites, flagged sites fell from 11 to 4.
A flattened Set-Cookie header was re-split on commas. The comma inside expires=Tue, 09-Mar-2027 matched, shearing the cookie so every attribute after the date became invisible — including the HttpOnly the cookie actually carried.
Any cookie whose expires= precedes its flags. That is most of them.
The check matched any * anywhere in the directive, so ordinary allowlist entries like https://*.onetrust.com tripped it. Nearly every policy that loads analytics has one.
Reported High. The better the policy, the more likely it fired.
A header-absence check. GitHub enforces rate limits at the edge and exposes no rate-limit headers on HTML responses.
Every site that rate-limits without advertising it.
The finding was right — the file is not RFC 9116. The evidence was hardcoded to say "not found" regardless, so anyone who checked saw a file returning HTTP 200 and concluded the scanner was broken.
A correct finding made to look like a bug.
Three never reached anyone, because a new detector gets pointed at known-good targets before it is committed. The third is the one worth reading: it is the detector written to fix the second, making the same class of mistake in a new place.
The probe gated on whether a port was reachable. Cloudflare's MX completes the TCP handshake on 587 and then resets before answering EHLO, so the probe learned nothing — and silence was being scored as a failure.
It fired on some runs and not others, depending on when the reset landed. A finding the reporter cannot reproduce is worse than a consistent one: re-running the scan appears to fix it.
All of it true — of the Cloudflare "Just a moment…" interstitial those sites return to a scanner. None of it true of the sites themselves.
It is not one bad check. When an edge answers instead of the origin, every content-derived check describes the interstitial — CSP, cookies, headers, fingerprint — and the whole report reads as a scan of the customer.
The detector written to fix the problem above made the same mistake: it matched x-datadome by presence, and DataDome sets that header on allowed traffic. The real 1.3 MB homepage was sitting in the response body.
Header signals now match on value, and the pattern is mandatory on every one of them, so the shortcut cannot be taken again.
Half of a false-positive audit is deciding what to leave alone. These were each investigated, confirmed correct, and written down as correct — because the obvious way to verify them produces the wrong answer, and without the note someone eventually "fixes" a working check.
The next run was meant to produce a published baseline, not another audit. It produced both. Every entry was verified against the live site before anything was changed, and the first is the one that would have cost us the most.
GOV.UK publishes v=spf1 -all at gov.uk and p=reject at _dmarc.gov.uk — close to the strongest email authentication a UK domain can have. We looked for both on the wrong name. The registrable domain was derived by counting label lengths: "uk" is two characters and "gov" is three, so it took the .co.uk branch and returned the hostname unchanged, querying www.gov.uk for SPF and _dmarc.www.gov.uk for DMARC. Neither exists. www.bbc.co.uk escaped only by having a fourth label.
One finding, three separate bugs. The severity had also been escalated from Medium to High because the MX lookup counted two CNAME records pointing at Fastly as mail servers — a DNS answer carries the whole resolution chain, not just the type you asked for. And the evidence line read "Status: NOERROR (domain exists)" about a name that does not exist, because that string was hardcoded.
DNSKEY records live at the zone apex, and we asked the scanned hostname. www.nasa.gov answers with a single CNAME and no DNSKEY, while nasa.gov is properly signed.
This one we introduced ourselves, with the fix above. Until answers were matched against the type requested, that CNAME had been counted as a DNSKEY and the check passed for entirely the wrong reason — a hidden false negative that the fix converted into a visible false positive. Across the corpus, DNSSEC findings went from 13 of 30 to 26 of 29 in a single run, which is what made it obvious. The higher figure is the correct one: the old number was under-reporting.
stripe.com sends no rate-limit headers at all and unquestionably rate-limits. Rate limiting lives on login and API routes, not the homepage, and most CDNs throttle silently. The check could see whether rate limiting was advertised, and nothing more.
The infrastructure allowlist behind it had already been patched twice — once for Google server tokens, once for GitHub's edge — which is the shape of a check chasing an unobservable property one vendor at a time. It now says only what it saw, at Info, with no weight in the grade.
A survey widget's apiKey inside Google Tag Manager, with a payload carrying a company id and an issued-at and nothing else. Vendor SDKs ship exactly this kind of token publicly, to identify which account a widget belongs to.
Tokens now split on whether the payload carries a principal claim — sub, user_id, email, role, scope. With one, it is a credential leak. With none, it is an account identifier, reported at Info so the signal survives without the accusation.
JSEncrypt, an open-source RSA library loaded on countless bank login forms, contains one BEGIN RSA PRIVATE KEY literal used to format its output. There is no key material anywhere near it. The pattern matched the bare header, so every site loading the library earned a maximum-severity Critical.
Verified against the real 57 KB minified library: one header occurrence, zero key blocks. The pattern now requires a base64 key body to follow the header, and still matches a key genuinely embedded in source.
The cookie is __Host-js_csrf — Secure, SameSite=None and deliberately not HttpOnly, while every other cookie Dropbox sets carries it. A double-submit CSRF token works precisely because the page can read it and echo it back in a request header.
Marking it HttpOnly hardens nothing; it breaks the defence, because the page can no longer read the value either. We were telling Dropbox to disable their CSRF protection. The exemption applies to HttpOnly alone — a CSRF cookie still needs Secure and SameSite.
When a bot-protection interstitial answers instead of the origin, every check that reads the response describes that page. Etsy's report said the site was missing CSP and served a wildcard CORS policy. Both were true only of Cloudflare's challenge page.
Twelve of eighteen findings described a page Etsy never served. Naming them as untrusted in a coverage note was not enough. Findings are now withheld unless their source never read the intercepted response — DNS, email authentication, mail transport. Etsy goes from 18 findings to 4, all four genuinely about Etsy.
Four of these, and two from the earlier round, are the same error in different places: treating the presence of a response as its meaning. A reachable port read as a STARTTLS answer. A vendor header's presence read as "blocked", when it is set on allowed traffic. A CNAME counted as a mail server. NXDOMAIN read as "unknown", when it is the ordinary way of saying "no record". A header saying a key would go here, read as a key. Twice we committed that error while fixing it.
Six reviewers each took a family of checks and fed the real functions realistic harmless responses, looking for a finding that fires on a healthy site. A full passive sweep of the same thirty sites followed, with every difference from the last one checked against the live site. Then the result set was re-derived independently, and that is where two more turned up. Nothing on this list was found by a customer.
GitLab delivers its policy in a <meta http-equiv="Content-Security-Policy"> tag, and we read response headers only. The report said it had none.
A meta policy is now read and analysed like a header. gitlab.com now reports what is true instead: unsafe-inline and unsafe-eval in script-src, and an open object-src, both Medium. A commented-out tag and a Report-Only tag are not treated as policies, and clickjacking still reads headers only, because a meta tag cannot carry frame-ancestors.
zoom.us redirects to www.zoom.com but keeps its assets on st1.zoom.us. Judged against the final host alone, Zoom's own CDN looked like a stranger's script.
The address you asked us to scan now counts as the site's own. A lookalike such as zoom.us.evil.example is still a stranger, and Zoom's one real third-party script is still reported.
archive.org's map is a vendored copy of a public library, so what it exposes is that library and not archive.org's code. Rated High all the same. On fastly.com we reported only the first map we found.
A map is now rated by whose code it embeds: fewer than three of the site's own files and under 5% of the map, or a map served from a vendor path, is a Low finding that says how many files are the site's. archive.org went from High to Low. fastly.com now shows both of its maps, and the second, which holds the site's own SearchCombobox.astro, is High.
nasa.gov's TXT lookup returns a resolver error about one time in ten. We decided "absent" from the first name, www.nasa.gov, which answered cleanly with no TXT, and ignored that the parent, where the record lives, had errored. One scan in five reported a missing SPF record.
It took ten repeat scans of one site to see it, which is why a single clean run proves little. Any resolver error up the chain now makes the answer inconclusive, and a SERVFAIL is retried once. The same code serves DMARC, so it could have made the same false claim there. Ten of ten scans are clean afterwards.
Cloudflare answered the scanner HTTP 403 on both paths while /.well-known/security.txt is a valid file. A refusal was being read as "the file does not exist".
Only a definite answer supports "missing" now. A refusal, a rate limit, a server error or a timeout means we could not tell, and nothing is reported. A real 404 still is.
The pattern admin matched inside Administratoren, Adminkandidaturen and Requests_for_adminship, which are discussion pages about the role, not an admin panel.
Keywords now match whole path segments. /admin/, /wp-admin/, /administrator/ and */store/admin still do. Found by verifying the sweep, not by the sweep itself: an Info finding that looks harmless is the one nobody checks.
The Server header was BunnyCDN-HOU1-893, a CDN edge-node id. The rule fired on any digit, so it was reported as a leaked software version.
A version now has to look like one (nginx/1.31.3, Apache/2.4.41). The real nginx.org banner is still reported, and verified against the live header.
Not live sites: each of these was reproduced through the real function with a harmless response before it was fixed, and each fix has a test that fails on the old code and a second test that the real problem is still found.
The cost. Every one of these makes the check less eager, and some real problems in unusual shapes can now be missed: an app that renders the same record for any identifier, an unauthenticated DELETE that answers 404 for a missing id, and a storage bucket whose first hundred objects are all static assets. They are recorded in the audit as narrowed on purpose, not hidden.
What we did not verify. The bot-protected sites that answer the scanner with a challenge (on the last full run amazon.co.uk, apnews.com, ebay, etsy, nytimes, reuters and stackoverflow; npmjs.com and apnews.com change from one run to the next) are withheld rather than graded, so nothing here says what they would show. Technology and CVE matches were only spot-checked: nginx.org's version banner, jquery.com's X-Powered-By and python.org's jQuery 1.8.2 with its two CVEs were confirmed; the unclaimed-resource page texts for several takeover providers were written from memory of each provider's page and not fetched.
Fewer findings is the goal here. Every one that came off was something the site had not done wrong. The last column is the same four sites on 30 September; its totals include Info-level notes, which carry no weight in the grade.
| Target | Before | After | 30 Sep (all / above Info) |
|---|---|---|---|
| google.com | 15 | 13 | 15 / 6 |
| github.com | 12 | 11 | 11 / 5 |
| cloudflare.com | 9 | 8 | 7 / 2 |
| mozilla.org | 8 | 7 | 6 / 2 |
The newest checks were then verified across 15 sites — including Wikipedia, the BBC, Amazon, Vercel, Shopify and GOV.UK — with zero findings above Info, both bot-protection interstitials correctly identified, and every real page left alone.
The second round is measured the same way, across the 29 sites common to both runs: actionable findings fell from 168 to 117. GOV.UK went from 9 findings and 3 actionable to 7 and 0, etsy.com from 17 and 10 to 4 and 1, dropbox.com from 13 and 5 to 12 and 4. Not all 51 were deletions: some were findings withheld because they described a bot-protection page rather than the customer, and one was a check reclassified to Info because it cannot observe what it was asserting. That one still appears in every report — it simply no longer counts against the grade.
