robots.txt Proxy

Crawler access is served dynamically rather than from a static file, so rules for search engines and rules for AI training crawlers can differ and can change without a deployment. Requests are matched on user agent with verification where the crawler publishes a method, commonly reverse DNS or a published address range, since user agent alone is trivially spoofed and a rule that trusts it is decorative. Search crawlers keep the access they need for indexing while named AI training agents are allowed or disallowed per client instruction. Disallow rules are written against real paths and tested, because a stray slash can block an entire site and robots.txt failures are silent. The served output is verified from outside as text/plain with a correct status, and the sitemap directive is present. Crawl-delay and pattern rules are applied only where a specific crawler honours them. Access is logged so the client can see which agents actually requested the file and what they were served, and the default on any error is to serve the permissive search engine rules rather than to block everything.

Industry Insight