How it works
The request path, the proof ladder, and what fail-open means.
The request path
A request to your site takes this trip:
- Your edge (the Vercel middleware or the Cloudflare worker) extracts the request's metadata: signature headers, user agent, path, source IP. For non-GET requests it also reads the body to compute a content digest, which signature verification requires. If you have large uploads behind a protected route, exclude them from the matcher.
- It asks the D3 Edge decision service for a decision. This call has a
hard time budget (150 ms by default; guided Cloudflare installs run at
500 ms) and is the only blocking network call we add to your request
path. In
SDK configuration the decision service appears as
POLICY_WORKER_URL. - The answer is
passorblock. In observe mode it is alwayspass. - The request continues to your site, or gets a
403if you have chosen to block something. - After the response is on its way, a second, non-blocking call reports the outcome, and the request is verified in depth: signature checks against key directories, crawler fingerprint matching, IP range confirmation. All of it happens off the hot path.
If step 2 times out or errors, the request passes. That is the default and we recommend keeping it. See built to fail open.
The proof ladder
Every logged request gets a tier. A tier is a claim about evidence, so you can decide how much to trust it:
| Tier | What we can prove |
|---|---|
proven | A valid signature with a key we've verified, or UA and IP confirmed by the operator's published ranges |
claimed | A plausible signature whose key we haven't verified yet — resolved after the response |
derived | The UA matches a known crawler fingerprint, but the IP can't be confirmed either way |
spoofed | The UA claims a known crawler and the source IP contradicts the operator's published ranges |
unverified | None of the above — including malformed, expired, or invalid signatures, with the reason |
Two things worth knowing:
spoofedis a verdict, not a guess. It means something claimed to be Googlebot (or similar) from an IP Google says isn't theirs. That is the traffic worth looking at first.- A
claimedtier doesn't rewrite history. The logged row keeps the tier it earned at request time. Once the key checks out, later requests from that agent classifyproven. - You never maintain a bot list. When one feed calls a crawler
googlebotand another calls itGoogle, we merge them into one canonical identity before anything reaches your log. The research behind this: Names don't scale, our 2026 study of robots.txt across the top 10,000 sites (5,577 readable files, Sitedex dataset).
Observe first, enforce when ready
Your account starts analytics-only: every agent request classified and logged, nothing blocked. Enforcement rules match on tier, identity, purpose, and path prefix, and are configured with our team. When no rule exists, no rule fires.
Built to fail open
Your site never depends on D3 Edge to serve. The decision call runs inside a hard time budget; if the answer doesn't arrive, the request passes. This holds for any failure between your edge and ours. The worst case is a gap in your classifications, never an outage on your site. If your threat model prefers blocking over serving unclassified traffic, the SDKs can be configured to fail closed instead; see your SDK's configuration reference.
What we store
Per request: path, user agent, signature headers, the verification outcome, and a salted, truncated hash of the source IP in place of the raw address. To show you latency and error rates, the SDK also reports your origin response's status code, content type, and size. Response bodies never reach us.