Free Tools

Robots.txt generator for AI crawlers

Decide exactly which AI crawlers can read your site - GPTBot, ClaudeBot, PerplexityBot, Google-Extended and every other documented agent.

AI crawlers

  • GPTBot

    OpenAI - Model training

  • OAI-SearchBot

    OpenAI - ChatGPT search results

  • ChatGPT-User

    OpenAI - Live user requests in ChatGPT

  • ClaudeBot

    Anthropic - Model training

  • Claude-SearchBot

    Anthropic - Claude search results

  • Claude-User

    Anthropic - Live user requests in Claude

  • Google-Extended

    Google - Gemini training and grounding

  • PerplexityBot

    Perplexity - Perplexity search index

  • Perplexity-User

    Perplexity - Live user requests in Perplexity

  • Meta-ExternalAgent

    Meta - Meta AI training

  • Applebot-Extended

    Apple - Apple Intelligence training

  • Amazonbot

    Amazon - Alexa and Amazon AI

  • CCBot

    Common Crawl - Open dataset used to train many models

  • Bytespider

    ByteDance - Model training

robots.txt

# AI crawler access - generated with ranked.ai/tools

# OpenAI - Model training
User-agent: GPTBot
Allow: /

# OpenAI - ChatGPT search results
User-agent: OAI-SearchBot
Allow: /

# OpenAI - Live user requests in ChatGPT
User-agent: ChatGPT-User
Allow: /

# Anthropic - Model training
User-agent: ClaudeBot
Allow: /

# Anthropic - Claude search results
User-agent: Claude-SearchBot
Allow: /

# Anthropic - Live user requests in Claude
User-agent: Claude-User
Allow: /

# Google - Gemini training and grounding
User-agent: Google-Extended
Allow: /

# Perplexity - Perplexity search index
User-agent: PerplexityBot
Allow: /

# Perplexity - Live user requests in Perplexity
User-agent: Perplexity-User
Allow: /

# Meta - Meta AI training
User-agent: Meta-ExternalAgent
Allow: /

# Apple - Apple Intelligence training
User-agent: Applebot-Extended
Allow: /

# Amazon - Alexa and Amazon AI
User-agent: Amazonbot
Allow: /

# Common Crawl - Open dataset used to train many models
User-agent: CCBot
Allow: /

# ByteDance - Model training
User-agent: Bytespider
Allow: /

# Everyone else (search engines, browsers)
User-agent: *
Allow: /

Frequently asked questions

Should I allow or block AI crawlers?

If you want customers to find you, allow them. AI assistants recommend businesses whose sites they can read - blocking GPTBot or PerplexityBot removes you from those answers. Blocking makes sense mainly for paywalled or proprietary content where training-data concerns outweigh visibility.

What’s the difference between crawlers like GPTBot and ChatGPT-User?

Training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent) collect content to train future models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) index your site so it can be cited in answers. User agents (ChatGPT-User, Claude-User, Perplexity-User) fetch pages live when a person asks about you. You can allow or block each independently - many sites allow search and user agents even when blocking training.

Does robots.txt guarantee AI crawlers obey?

The documented crawlers from OpenAI, Anthropic, Google, Apple, Amazon and Perplexity publicly commit to honoring robots.txt. It’s a directive, not a firewall - but for the major platforms it’s respected and it’s the standard way to declare your policy.