robots.txt is a sign, not a fence: 8 technical vectors through which AI still reads your website

You configure robots.txt like this: User-agent: GPTBot Disallow: / User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: PerplexityBot Disallow: / User-agent: * Disallow: / You enable Cloudflare Bot Management. You set up Akamai. Maybe even a server-side paywall. And then you query ChatGPT about your product and it cites your website as a source. How? I work on GEO (Generative Engine Optimization) projects where we audit how LLMs represent brands. We routinely analyze thousands of prompt-response pairs. Across multiple projects, we consistently find that 10–20% of LLM responses cite the brand's own website as a source — even when every known bot is blocked. Here are the 8 technical vectors we documented, with academic sources and industry data. 1. Historical crawl data (Common Crawl) This is the biggest one and the least understood. Common Crawl is a nonprofit that has been archiving the web since 2007. The numbers: 9.5+ petabytes, 300+ billion documents ~2/3 o