# Crawling is welcome, including AI/answer-engine crawlers. # Keep every named crawler in this single group: robots rules are not inherited # from a separate wildcard group when a more-specific user-agent group matches. User-agent: * User-agent: GPTBot User-agent: ChatGPT-User User-agent: OAI-SearchBot User-agent: Googlebot User-agent: Google-Extended User-agent: Google-CloudVertexBot User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Claude-Web User-agent: anthropic-ai User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Applebot User-agent: Applebot-Extended User-agent: bingbot User-agent: Amazonbot User-agent: meta-externalagent User-agent: FacebookBot User-agent: CCBot User-agent: DuckAssistBot User-agent: cohere-ai User-agent: MistralAI-User Content-Signal: search=yes, ai-input=yes, ai-train=yes Allow: / # Machine-readable site index for LLMs / agents: # https://n3tz.ai/llms.txt (concise index) # https://n3tz.ai/llms-full.txt (richer full-text index) Sitemap: https://n3tz.ai/sitemap-index.xml