AI crawler tracking

Every JavaScript analytics tool shares a blind spot: the script only runs if something executes it. Real browsers do. The AI crawlers people most want to see — GPTBot, ClaudeBot, PerplexityBot — request the raw HTML and never run a line of it, so they leave no trace in any client-side tracker, ours included.

The symptom is unmistakable once you look for it: a site with thousands of visits referred from ChatGPT, reporting zero GPTBot crawls. Those visits only exist because a model read the site, so the crawl definitely happened. Nothing was there to record it.

The fix is to report from your server, where every request is visible whether or not it runs JavaScript. One authenticated endpoint, called from your middleware.

Next.js

Add a middleware.ts at your project root. It fires only for crawler user agents, so it adds nothing to a real visitor's request:

import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';

// Cheap pre-filter: only crawlers get reported, so real visitors pay nothing.
const CRAWLER = /bot\b|crawler|spider|gptbot|claudebot|perplexity|ccbot|bytespider/i;

export async function middleware(req: NextRequest) {
  const ua = req.headers.get('user-agent') ?? '';
  if (CRAWLER.test(ua)) {
    try {
      await fetch('https://jamp.io/api/collect/server', {
        method: 'POST',
        headers: { 'content-type': 'application/json' },
        body: JSON.stringify({
          website_id: process.env.JAMP_SITE_ID,
          key: process.env.JAMP_SOURCEMAP_KEY,
          url: req.url,
          user_agent: ua,
          referrer: req.headers.get('referer') ?? '',
        }),
      });
    } catch {
      // Never let reporting break a page render.
    }
  }
  return NextResponse.next();
}

// Pages only: skip assets and your own API routes.
export const config = {
  matcher: ['/((?!api|_next/static|_next/image|favicon.ico).*)'],
};

Set JAMP_SITE_ID to your site id and JAMP_SOURCEMAP_KEY to your source map key (the same per-site secret used for source maps and server-side errors). Both belong in environment variables, not in the file.

Anything else

Any server that can make an HTTP request works. Send the crawler's user agent and the URL it asked for:

{
  "website_id": "YOUR_SITE_ID",
  "key": "YOUR_SOURCEMAP_KEY",
  "url": "https://example.com/pricing",
  "user_agent": "Mozilla/5.0 AppleWebKit/537.36 (compatible; GPTBot/1.2; +https://openai.com/gptbot)",
  "referrer": ""
}

The response tells you what happened: {"kind":"bot"} when a crawler was recorded, {"kind":"skipped"} when the user agent was a real browser and was ignored.