How to Tell When Your Scanner Is Lying to You

I wrote a tool to scan websites for technical and compliance problems. It swept 65 sites in 83 seconds and reported six GDPR violations — sites loading analytics that set cookies without asking for consent.

It was wrong about three of them.

I found out because I checked before I acted. This article is about how I checked, why the tool lied so convincingly, and the genuinely uncomfortable pattern in which cases it got wrong. If you use any automated scanner — security, SEO, accessibility, compliance, or a linter — the failure mode described here applies to yours too.

The lie

The scanner's logic looked reasonable. Fetch the page. Look for an analytics script. Look for a cookie banner. If there is a tracker and no banner, that is a consent problem.

Six sites matched. The findings were specific, formatted, and marked CRITICAL.

Then I opened each one in a real browser with a clean profile, captured every network request, and dumped the cookie jar before touching anything. Three of the six collapsed:

• One was running Google Consent Mode in denied state. The analytics script loads, but it is blocked from writing anything. Zero cookies. That site was not violating the rules — it was following them carefully.
• One had an analytics snippet that was never configured. It still carried the placeholder measurement ID from whatever template it was copied out of, commented out. It had never collected a single visit in its life.
• One did have a consent banner. The platform it was hosted on injected one. My scanner simply could not see it.

Three real findings, three inventions. And here is the part that should worry you: the tool reported all six with exactly the same confidence. There was no hedge, no lower score, no "uncertain" flag on the three it made up. CRITICAL, six times.

Why it lied

The root cause is embarrassingly simple.

My scanner never executed JavaScript. That is why it was fast — no browser, no rendering, no waiting. And cookie consent banners are almost always injected by JavaScript.

So when the tool reported "no consent banner found", it was never actually concluding "there is no banner." It was concluding "I cannot perceive banners."

Those two sentences look similar and mean opposite things. One is a finding. The other is a confession. My tool printed the confession and labelled it a finding.

This generalises. Absence of evidence, produced by a tool that is structurally incapable of perceiving that evidence, is not evidence of absence. It is noise in a suit.

The four checks

Here is the checklist I now run before any automated finding is allowed to reach a human being.

1. Can the tool actually see the thing it says is missing?

Before you trust a negative finding — "no banner", "no alt text", "no rate limiting" — ask what the tool would have needed to do in order to see the thing, and whether it did that.

My scanner reported on JavaScript-injected UI without running JavaScript. Once you say that sentence out loud, the bug is obvious. But nobody says it out loud, because the report looks so finished.

Negative findings are the dangerous ones. A tool saying "I found X" is usually reliable — it found something. A tool saying "X is missing" is only as trustworthy as its ability to have found X in the first place.

2. Test it against something whose answer you already know

Before I ran the scanner on 65 unknown sites, I ran it against one site I had audited by hand the week before, where I knew every finding: no HTTPS redirect, zero security headers, no canonical tag, no <h1> on the homepage, compression correctly enabled. The tool reproduced every one, and — importantly — did not flag the compression that was working fine.

That is the cheapest correctness check that exists for any scanner, and it is what let me trust the tool on sites I had never seen.

But notice what it did not save me from. The scanner passed its ground-truth test perfectly and was still structurally blind. The site I validated against had no cookie banner and no trackers, so the blind spot never fired.

Passing a ground-truth test is necessary. It is not sufficient. Your test case has to actually exercise the thing that can break.

3. Verify the mechanism, not the marker

This is the one that produced my worst bug.

I was checking: is there an analytics script on this page?

The law cares about: is a cookie being written to the visitor's device?

Those are not the same test. The first is a proxy for the second — usually correlated, easy to check, and wrong in exactly the cases that matter. I had coded the proxy and labelled it the real thing.

Every heuristic scanner is full of proxies like this, because proxies are what make it cheap. That is fine. What is not fine is forgetting which of your checks are proxies, and then reporting a proxy's result as though it were the mechanism's.

4. Keep a false-positive log

Write down every finding you killed, and why.

Not for tidiness — for calibration. It is the only honest record of how often your tooling is confidently wrong. Mine now says: three out of six, on the finding type I cared about most.

Without that log I would have remembered this as "the scanner works well." With it, I know the true rate, and I know never to let that finding type reach a stranger unverified.

The log cuts both ways, incidentally. The browser pass also caught a third-party font request leaking visitor IP addresses that the fast scan had missed entirely. Verification is not just for killing false positives. It finds what the cheap pass could not see.

Then it accused me

After fixing the six findings down to three, I pointed the scanner at imparlabs.com as a sanity check.

It came back CRITICAL. A GDPR violation. On my own site.

It was wrong, and the reason is worth sitting with.

imparlabs.com runs self-hosted, cookieless analytics. It writes zero cookies to your device. Under ePrivacy there is nothing to consent to, so it requires no cookie banner. That is not a loophole I am exploiting — that is the entire point of building it that way.

My scanner saw "an analytics script is present, and there is no cookie banner" and called it an offence. It was flagging the exact privacy-first architecture that makes the banner unnecessary in the first place.

That is check 3 failing, in its purest form. Consent is triggered by a cookie being written, not by a script existing. I had coded the wrong test, and it condemned every cookieless setup on earth — including the one I had deliberately built to be clean.

The fix was to make the rule key on the mechanism: cookieless analytics (Umami, Plausible, Fathom) are consent-exempt, excluded from the finding, and credited rather than flagged. Matomo deliberately stays in the consent-required column, because from static HTML you cannot tell whether it is self-hosted in cookieless mode. When the tool cannot know, it must not guess — it must escalate to a human.

If you take one practical thing from this article, take this: if you run analytics in the EU, the question that decides whether you need a consent banner is not "which vendor is this?" It is "does this write to my visitor's device?" Cookieless, self-hosted analytics answers no — and the banner requirement disappears along with it.

The uncomfortable pattern

Now look at which three findings were wrong.

• The site running Consent Mode in denied state — doing it correctly.
• The site whose analytics was never wired up — collecting nothing.
• My own cookieless setup — built specifically to be clean.

Meanwhile the straightforward offenders — script loaded, cookies written, no banner anywhere — were caught correctly and easily.

A cheap scanner's false positives do not spread evenly across your list. They concentrate on the sophisticated cases.

This makes sense once you see it. A site that has thought carefully about consent produces a subtle signature: the script is still there, but its behaviour is constrained in a way that only shows up at runtime. Through a static scan, careful and careless look identical. The naive violator, by contrast, is trivially detectable — they did nothing at all.

So your tool is most wrong about precisely the people who will know instantly that you are wrong. Which is why this matters far beyond tidy engineering:

A false positive caught internally costs you a minute. The same false positive sent to a stranger costs you the relationship — and it poisons every finding in that report that was real.

That asymmetry is the whole argument for the gate.

The rule

A scanner produces candidates, never claims. Nothing reaches a person until something that can actually see the truth has confirmed it.

This is not an argument against fast tooling. The 83-second sweep was genuinely valuable — it took 65 sites down to a handful worth looking at, which is exactly what it should do. The speed comes from the approximation. Making the scanner slow and thorough would destroy the thing that made it useful.

The answer is not a better scanner. It is knowing which side of the trust boundary the scanner sits on.

Use the fast pass to decide where to look. Use a browser — something that can actually observe the behaviour — to decide what is true. And never, ever let the first one speak directly to another human being on your behalf.

Because a scanner will lie to you. It will do it in a well-formatted table, marked CRITICAL, with total confidence, and it will not tell you which of its findings it invented.

That part is your job.