robots.txt Builder

About robots.txt in the AI era

The robots.txt file is your site's public contract with crawlers — both traditional search engines and the new wave of AI crawlers.

AI crawlers covered by this tool

• OpenAI: GPTBot, ChatGPT-User, OAI-SearchBot
• Anthropic: ClaudeBot, anthropic-ai, Claude-Web
• Google AI: Google-Extended
• Perplexity: PerplexityBot, Perplexity-User
• Apple: Applebot-Extended
• Plus: CCBot, YouBot, cohere-ai, Diffbot, Bytespider, Amazonbot, MistralAI-User and more

Should you allow them?

For most sites: yes. AI assistants cite their sources, which drives discovery, brand awareness, and direct traffic. Blocking them removes you from that channel — and you can't add it back retroactively.

Block them only if: you sell paid content, have legal/contractual restrictions, or your business model directly competes with AI-mediated answers.

Programmatic API

npm install @mystoq/seo-toolkit

import { generateRobotsTxt } from '@mystoq/seo-toolkit';
generateRobotsTxt({ aiCrawlers: 'allow', sitemap: 'https://acme.com/sitemap.xml' });