The Web’s Measurement Problem

Executive Summary

The web’s measurement infrastructure is not merely inaccurate. It is structurally misaligned.

This paper presents the findings from a four-year investigation into bot traffic, web analytics accuracy, consent management, and ad-platform optimisation. What began as a bot-detection project evolved into a broader discovery: the standard tools the web relies on to measure, attribute, and convert visitors are structurally broken. The failures are not bugs. They are consequences of architectural decisions the industry has normalised.

The eight core findings are as follows. First, major analytics platforms disagree with each other and all undercount real visitors, often by a third or more, because their third-party scripts are blocked by modern browsers. Second, first-party script delivery (serving measurement code from the merchant’s own domain) recovers the missing audience without violating any privacy standard. Third, most consent managers fail silently in privacy-focused browsers, leaving sites either non-compliant or blind to a growing share of their audience. Fourth, privacy-first analytics tools that collect no identity at all solve compliance but provide nothing useful for marketing attribution. Fifth, the platforms that do collect data abandon all measurement the moment a visitor declines consent, even though anonymous aggregate counting remains fully permissible. Sixth, bot traffic is not a single category but at least six distinct types, each requiring a different detection approach, and CAPTCHAs are now effectively defeated. Seventh, server-side tag management, the industry’s current recommendation, routinely requires 40 to 80 hours of engineering and still cannot hold a conversion until it is qualified. Eighth, and most consequential: every conversion reported to an ad platform is not merely a record but a training signal. When that signal originates from a bot, the platform’s algorithm learns to seek more traffic like the bot, compounding the waste with every cycle.

DataCops was built to close these gaps with one constraint: never collect more than a visitor legitimately permits, and never collect less. The system operates entirely on the merchant’s own domain as first-party infrastructure. It combines bot and fraud detection at both the network and behavioural layers, a certified first-party consent manager, durable first-party identity, and server-side conversion delivery to every major ad platform through a single integration. Conversions can be held until the CRM confirms qualification, so ad platforms learn from revenue, not from form submissions. Deletion of a person’s data is a single action against a single identity record, with a known list of every platform that received their data.

The result is a measurement and conversion stack where the numbers reconcile, bots never train the algorithm, consent is provably enforced, and compliance is a design property rather than an afterthought.

Part One

Background

This section provides context on how and why this research was conducted. Readers primarily interested in the technical findings may proceed directly to Part Two.

1.1 The Author

DataCops was founded by Simul Sarker, a self-taught web developer and product designer originally from Bangladesh, now based in Lisbon, Portugal. The author does not hold a degree in computer science or a related field. The authority behind this paper is practitioner experience: more than 70 production websites shipped, years of hands-on work with Google Tag Manager, Google Analytics, the Meta Pixel, and ad campaign infrastructure, and four years of focused research and product development in the domain this paper covers.

That background matters because it shaped the approach. The problems described here were not discovered through academic research. They were encountered directly, on real client websites, handling real advertising budgets. Every finding in this paper was observed first-hand, tested on production traffic, and in several cases, was initially misunderstood before the correct explanation emerged.

1.2 From Web Development to Bot Detection

Between 2018 and 2021, the author built and shipped websites for startups and companies running paid advertising campaigns. The standard instrumentation stack (Google Analytics, Google Tag Manager, the Meta Pixel) became routine. By 2021, a question that could not be resolved from inside that workflow became persistent: what, precisely, makes a website matter to a business beyond its design? What actually moves the commercial needle, and how is that measured?

That question led to formal product-design training (Google UX Design, Coursera), which reframed the author’s thinking from task execution toward problem identification. It also surfaced a desire to solve a problem that was universal, not specific to one client.

Bot traffic, which had been background noise throughout years of development work, became the focus in mid-2022. The initial research explored how bots function, what motivates them, and why they persist. The first discovery was content scraping: bots copying entire websites, rewriting the content with early AI tools, and republishing it to steal search traffic from the original authors. The second was that the standard defence, robots.txt, is purely voluntary and ignored by malicious crawlers.

Development of a bot-detection prototype began in early 2023, with two additional developers. The system drew on open-source fingerprinting research, particularly the work published by FingerprintJS.

1.3 The Market Correction

The initial target market was SEO professionals and content publishers. It was wrong. Extensive outreach at SEO events and within the Lisbon tech community produced no traction. SEO professionals viewed bot traffic as irrelevant to their core metric (Search Console clicks) and considered Cloudflare sufficient for any bot-related concern.

The correction came from an unplanned conversation with David Cowin, a founder operating out of a Lisbon coworking space. Cowin’s immediate response was that bot traffic consumed roughly half of his monthly Google Ads budget, over $8,000 per month, and that he had accepted this as an unavoidable cost of advertising because no solution he knew of addressed it.

That conversation reframed the entire project. The technology was the same. The problem it solved was not scraping. It was ad fraud: real advertising spend wasted on non-human clicks, accepted industry-wide as an uncontrollable expense. This became the foundation for what DataCops is today.

Part Two

Eight Findings

Each finding below represents a structural failure in the web’s measurement infrastructure that was discovered during this research. Together, they form a map of how the industry’s standard tools fail, and why each failure compounds the others.

Part Three

System Architecture

Every component of DataCops is a direct response to one or more of the findings above. This section presents the complete system: a single architectural overview, followed by each component, its function, and the specific gap it addresses.

3.1 Design Principle

The eight findings reduce to a single design constraint. The data a business collects is either missing, because it could not be legitimately captured, or dirty, because the wrong thing was captured. The common response to this gap involves one of two forms of dishonesty: fingerprinting visitors who declined consent, firing tags before consent is granted, or counting bot traffic to inflate metrics.

The honest alternative is harder and is the only one that is durable. Capture the maximum that a visitor legitimately permits. Refuse everything beyond that boundary. Remove bots before they are counted. Honour a “no” completely. Maintain a single identity per person, so that deleting someone is one action rather than a search across disconnected systems. Then make the permitted data so clean that the excluded data is never missed.

3.2 System Overview

Everything operates on the merchant’s own domain. The edge worker and page script capture visitor data. The consent gate and fraud verdict determine what may proceed. A single identity record is maintained. Four capture modes feed a single output gateway. That gateway distributes to every connected platform. Deletion operates against the same single identity.

3.3 First-Party Delivery

This component addresses Finding 2. The DataCops script is not loaded from a DataCops domain. It is loaded from the merchant’s own domain. The merchant creates a single DNS record pointing a subdomain (e.g. datacops.theirstore.com) to DataCops infrastructure. From that point forward, the system automatically issues a TLS certificate for that subdomain, renews it on schedule, and serves the tracking script, consent manager, and data collection endpoint from the merchant’s own domain. No code changes are required. No hosting is managed by the merchant.

From the browser’s perspective, the script is part of the merchant’s own site. Ad blockers do not remove it. Safari does not classify it as a third-party tracker. Brave permits it. This single configuration step is the difference between seeing 32,400 visitors and seeing 41,200.

The infrastructure behind this is non-trivial. Issuing and renewing TLS certificates for hundreds of merchant subdomains, with redundancy across two certificate authorities to prevent any single point of failure, and automated alerting when anything deviates, is production-grade infrastructure. The merchant never interacts with it. That is the point.

3.4 Durable First-Party Identity

First-party delivery determines which visitors can be seen. It does not, by itself, determine whether a visitor is recognised on a return visit. The standard method of remembering a visitor is a JavaScript-set ID stored in the browser. Safari and iOS delete such IDs after seven days. A customer who clicks an ad, considers the purchase for two weeks, and returns to buy appears as an entirely new visitor. The original click that brought them is lost.

DataCops moves the identity out of script-managed storage and into a server-set first-party cookie on the merchant’s domain. Server-set first-party cookies are the category that browsers retain longest. The cookie is configured for up to 400 days, the maximum any browser will honour, and is scoped to the entire domain, so a visitor navigating between www and app subdomains remains a single person. Because the server sets the cookie, the edge worker (Section 3.7) can also read it, enabling exact pairing of an ad click to a specific visitor rather than probabilistic matching by IP and user agent within a short window.

Three rules are enforced on this identity to prevent abuse, because a long-lived identifier is precisely the kind of mechanism that gets misused.

First, the cookie is cryptographically signed, and the signature is valid only for a single merchant. It cannot follow a person across different companies’ websites. It is first-party only. It is never a supercookie. Second, a durable ID is only issued to a visitor who has passed both gates: consent accepted, and fraud verdict clean. A bot never receives one. A visitor who declined consent never receives one. Having no durable ID is a normal operational state, not an error condition. Third, fingerprinting is never used to reconstruct a visitor’s identity after they clear their browser. If a person resets their device, they become a stranger again, and that is respected. Fingerprinting remains in a single lane: fraud detection. The merchant controls the retention period from a single setting.

3.5 IP Reputation and Behavioural Fraud Verdict

This component addresses Finding 6 and operates at two layers, because bots operate at two layers.

Network layer. No system can responsibly classify a visitor as human without knowing whether the source IP is residential, mobile, a cloud server, a VPN endpoint, or a Tor exit. Most tools license an IP lookup from a third party, cache it, and rely on it remaining accurate. DataCops operates its own IP reputation service, updated continuously from thousands of sources.

Plus approximately 620 million proxy and anonymiser addresses, and more than 160,000 disposable or high-risk email domains.

Page layer. This is the layer that Cloudflare structurally cannot provide. It analyses how the browser presents itself from the inside: whether it is a headless engine pretending to have a display, how the page is physically interacted with, and the dozens of behavioural signals that separate a human-like bot from a real person. The two layers produce a single verdict per session, and that verdict is rendered before the session is counted in any metric or transmitted to any platform.

3.6 The Consent Gate

This component addresses Findings 4 and 5. Because no third-party consent manager could be relied upon to load consistently, DataCops built its own, served first-party from the merchant’s subdomain. It is registered with the IAB and certified to the TCF 2.2 standard.

The term “gate” is used deliberately. The consent manager produces three outcomes, and all three are enforced completely.

A visitor who accepts may be identified. A visitor who rejects is never identified: no email is read, no phone number is captured, no durable cookie is issued. But the visit is still counted anonymously, because aggregate counting is permitted and a zero is a false record. Outside the regions that require a banner, the rules applicable to that jurisdiction are followed.

A single consent choice controls two separate systems simultaneously: the browser-side tags (e.g., Google’s own scripts) and the server-side transmission to ad platforms. Keeping these two systems in agreement despite their having no direct communication channel is a significant engineering challenge, and it is solved.

One additional rule is often overlooked. Consent signals from bots are meaningless. A bot clicking “accept” has not provided consent. The consent gate is therefore fraud-filtered: an acceptance from a non-human session is not honoured. Every component feeds the others.

3.7 The Edge Worker

A subtle data loss occurs that most teams never notice. When a visitor clicks an ad, the click ID (gclid, fbclid, ttclid, and others) is appended to the URL. Safari and iOS now strip these parameters by name before any page script can read them. If the visitor also has an ad blocker, the page script may not load at all. In either case, the click ID is gone before the measurement system sees it.

The edge worker is an optional Cloudflare worker deployed on the merchant’s own domain. It operates at the network edge, reading click IDs from all fifteen supported ad platforms, plus UTM campaign tags, directly from the inbound URL, before Safari can strip them and regardless of whether the page script loads. These IDs are passed to DataCops and attached to the correct visitor. With the durable identity from Section 3.4, this match is exact rather than probabilistic.

3.8 PixelCops: Page-Level Capture

Once the first-party script is deployed, PixelCops handles page-level event capture: page views, form submissions, button clicks, thank-you pages, phone-number clicks. Most events are detected automatically with no configuration. For non-standard elements, the merchant opens their live site within the DataCops interface and clicks the specific element they want tracked. No code is written. No CSS selectors are configured. No tag manager is involved.

Every captured event passes through the same consent gate and fraud verdict before reaching the output gateway. There is one event list, one verdict pipeline, and one transmission path. The source of an event is a label, not a routing decision. Nothing downstream treats a manually selected button differently from an automatic detection, a form submission, a signup, or a Shopify order.

3.9 SignupCops: Invisible Signups and Signup Fraud

For many businesses, the value is not lost at the ad click. It is lost at signup, through two distinct failure modes.

Invisible signups. Most web applications now offer “Continue with Google” as the primary signup method, and a majority of new accounts are created through it. That flow redirects the visitor away from the application, completes authentication on Google’s servers, and returns the visitor to the application. No pixel or page-level listener can observe the moment the account is created. The click ID that brought the visitor is no longer in the URL when they return. Google Ads and Meta see an ad click followed by silence. SignupCops survives this round trip because the identity persists server-side. When the application reports a new account through a lightweight identify call, the signup is recorded against the same identity that originally carried the click ID.

Signup fraud. AI products are particularly vulnerable: users create hundreds of accounts to farm free credits or trial access. SignupCops evaluates each signup in real time. Is the email address disposable or freshly registered? Does the device fingerprint match fifty other “new” accounts? Is the name obviously fabricated? Is the IP a datacenter? The system scores the risk before the account is created, and a fraudulent signup never becomes a conversion event.

3.10 LeadCops: Qualified Conversion Delivery

This component addresses Finding 8 directly. It handles every form on the merchant’s site, whether hosted by DataCops or already present, and fulfils all three requirements of a correct conversion pipeline.

The workflow, by example: a visitor submits a contact form. The fraud verdict has already confirmed this is a human session. The lead is captured with its click ID intact. The merchant then has a choice. The lead can be sent to Meta immediately as a lead event. Or the merchant can activate the capability that no tag manager can provide: the lead is held. It syncs to the CRM (e.g. HubSpot) with fields mapped according to the team’s configuration. Nothing is sent to Meta. Three days later, the sales team marks the lead as “Sales Qualified.” DataCops detects that status change, releases the held conversion, and transmits it to Meta at that point, with the original click ID still attached, as a single real, qualified conversion.

A tag manager fires at the instant of form submission and cannot wait. A server-side pipeline drops junk by default and holds the rest until the CRM confirms qualification. That gap is the competitive moat.

The result is that Meta receives a single high-fidelity training signal rather than one hundred noisy form-submission events. It learns to find customers rather than form-fillers. Meta, HubSpot, the merchant’s website, and the original click ID were never designed to communicate with each other across a multi-day timeline. LeadCops is the integration layer that enables that communication.

3.11 Unified Output Gateway

This addresses Finding 7. All events that pass the consent gate and fraud verdict arrive at a single output gateway. From there, they are distributed to every connected platform simultaneously: Meta CAPI, Google Ads, TikTok, LinkedIn, and others. Each transmission uses the platform’s required format. Events are deduplicated so the same conversion is never counted twice. Consent is verified one final time, per platform, before anything is sent. Every transmission attempt is written to a log the merchant can inspect: per event, whether it was sent, held, or skipped, and the reason.

The single-gateway design is why this system can be simpler than a tag manager. It does not route arbitrary events to arbitrary destinations. It performs one focused function: every real, optionally qualified conversion becomes an attributed conversion in the ad platform, and no fraudulent or unqualified event ever does. Because the scope is narrow, the system remains simple, and a merchant configures it in minutes rather than in 40 to 80 engineering hours. Section 3.12 is the full comparison of what that replaces.

3.12 Server-Side Tag Manager Alternative

Finding 7 established that server-side tag management is the industry’s current recommendation for clean conversion delivery, and that it routinely fails to reach production. This section examines what that process actually requires, and identifies which DataCops component replaces each part of it.

A server-side Google Tag Manager container is a middleman server. The merchant provisions it (typically on Google Cloud Run), attaches a custom domain with a TLS certificate, and routes every event through it. Events arrive from the browser. The container transforms each one into the format its destination requires, evaluates a consent signal, and forwards the result. The work that a browser-side tag manager once performed now runs on infrastructure the merchant owns, hosts, monitors, and pays for. None of it is preconfigured. The container is empty on provisioning.

Before a single conversion can be transmitted, every component must be assembled manually. A client and a trigger must be written for each ad platform. Every parameter (event name, currency, value, click ID, hashed email, hashed phone, and roughly a dozen further fields per platform) must be mapped individually. Click IDs from inbound URLs must be captured through a custom variable and persisted to a first-party cookie. Two containers, browser-side and server-side, must be synchronised through manually configured triggers and variables. Debugging takes place on a remote server without browser developer tools, and preview mode does not reflect production traffic. Engineering teams routinely estimate 40 to 80 hours before the first event is transmitted. Maintenance is not included in that figure, and it is where the majority of eventual cost accumulates.

The most consequential piece is Consent Mode v2. It is a protocol, not a setting. The browser emits four signals when a visitor accepts, rejects, or revises consent (ad_storage, analytics_storage, ad_user_data, ad_personalization), and every tag is expected to read them and adjust its behaviour accordingly. In a hand-wired container, someone writes the mapping between each signal and each tag: what is sent when ad_storage is denied, what is redacted, what is hashed, whether the event is held or dropped. A missed mapping produces no warning. Either data the visitor refused is transmitted, or data the visitor permitted is silently discarded. Both failures are invisible.

Each of these manual components has a direct counterpart in the system already described. First-party delivery (Section 3.3) replaces the Cloud Run instance and its certificate. The unified output gateway (Section 3.11) replaces the per-platform clients, triggers, and parameter mappings. The first-party consent gate (Section 3.6) replaces Consent Mode wiring entirely, because the gate is the source of the consent signal rather than a consumer of it, so every event leaving the gateway already reflects the visitor’s choice on both sides. The edge worker (Section 3.7) replaces manual click ID capture. The single identity record (Section 3.4) eliminates the dual-container handoff. The per-event log replaces remote debugging. Platform parameter mappings and consent logic are maintained centrally, so a format revision at Meta or Google does not require the merchant to open or reconfigure anything.

The difference is structural, not incremental. A tag manager container is infinitely configurable because it is designed to route any event to any destination. This system performs one function: transmit conversions that are legitimately permitted and verifiably real, and transmit nothing else. That narrower scope is what allows the configuration to collapse from weeks of engineering to minutes of setup. DataCops is not a lighter tag manager. It is a different instrument.

3.13 Shopify Integration

A practical constraint affects a large share of online commerce. Shopify does not permit any third-party script to execute inside its checkout page. Not DataCops. Not Meta. No one. The most commercially important moment, the purchase, is precisely the moment no script can observe.

DataCops captures the purchase server-side. Earlier in the visit, the first-party script writes the visitor’s identity and click IDs into the cart object. When the order is paid, Shopify notifies DataCops’ server directly, with that identity stamp intact. The purchase is recorded and transmitted to ad platforms with its click ID, even though the checkout page was entirely blind. A Shopify store that connects its domain via the standard DNS record receives the full first-party treatment automatically, with no configuration required inside Shopify.

3.14 Deletion as a Design Property

This is the component almost no one builds, and it is the reason the word “single” appears throughout this architecture. When a person requests data deletion under GDPR or similar regulations, the typical technology stack turns that request into a manual investigation. The person’s email exists in the CRM. Their hashed email was transmitted to Meta and Google. Their visits are stored across three analytics tools. Their form submission is in a database. No inventory exists. The business estimates and hopes.

DataCops maintains one identity per person, and every outbound transmission passes through one gateway and is written to one log. A deletion request is therefore a single action. The identity record is erased. Personal fields on every associated session are blanked. Held leads are deleted outright, not anonymised. After a defined retention window, the remaining data is hard-purged. Because the log identifies every platform that received that person’s data, the downstream deletion requests to those platforms are a known list, not an investigative effort.

The same architectural decision that makes the data clean is what makes it deletable.

Part Four

Capabilities and Boundaries

4.1 What This System Makes Possible

When the components above operate together, a marketing team gains capabilities that the standard technology stack cannot provide. These are not improvements to existing tools. They are capabilities that the architecture of existing tools structurally prevents.

Reconciled visitor counts

A single first-party count of real people, with bot traffic shown and removed, replacing three platforms that disagree and all undercount.

Ad algorithms trained on humans

Every conversion that reaches Meta or Google has passed a fraud verdict. The optimisation algorithm learns exclusively from real people.

Qualified-only conversions

Leads can be held until the CRM confirms qualification, then released with their original click ID.

Visible signups

“Continue with Google” account creation remains tied to the click that caused it.

Persistent identity

A returning customer is recognised weeks later, on Safari, across subdomains, without code changes.

Provable consent enforcement

Consent demonstrably stops traffic, in both the browser and on the server, from a single choice.

Single-action deletion

One action against one identity record, with a documented list of every platform that received the person’s data.

Full Shopify coverage

Including the purchase event that no script can observe.

No hand-wired tag manager

No separate Cloud Run container to provision, no Consent Mode v2 to wire tag by tag, no per-platform trigger to build. See Section 3.12.

4.2 What This System Refuses to Do

The following are not limitations to be resolved in future versions. They are the line from Section 3.1, and the product is defined by them.

• ▪It never identifies a person who declined consent. A rejection is honoured completely: anonymous counting only, no cookie, no identity.
• ▪It never reconstructs an identity through fingerprinting after a person clears their browser. Fingerprinting detects fraud. It does not resurrect visitors.
• ▪It never follows a person across different merchants. The durable ID is cryptographically signed per merchant and is useless outside that merchant’s domain.
• ▪It never counts a bot. It never trains an ad platform on one, even when the bot clicked “accept.”

The refusals are not concessions. They are the design. Capture the maximum a visitor legitimately permits. Refuse the rest completely. Make the clean data good enough that the excluded data is never missed.

Conclusion

This investigation began with bot detection and ended with a broader conclusion: the web’s measurement infrastructure is not merely inaccurate. It is structurally misaligned with both privacy requirements and commercial objectives simultaneously.

Analytics platforms undercount by design. Consent managers fail silently. Bot traffic trains ad algorithms to find more bot traffic. Tag managers cannot hold a conversion until it matters. And when a person asks to be forgotten, the standard stack turns that request into guesswork.

Each of these is a known problem. What this research revealed is that they are the same problem. They are all consequences of building measurement infrastructure on third-party foundations, without a single source of truth for identity, consent, and data flow.

DataCops is the replacement built on the only foundation that works: first-party infrastructure, on the merchant’s own domain, with one identity, one consent gate, one fraud verdict, and one output path. The result is a system where every number reconciles, every permission is provable, every bot is caught before it counts, and every person can be deleted in one action.