# SuperBooks — superbooks.io
#
# Cloudflare's managed robots.txt is OFF for this zone; this file is the whole
# policy for the origin. The edge additionally sends the same content signals as
# a `Content-Signal` response header and enforces the blocks below with a WAF
# rule — this file is the form visible to crawlers that read robots.txt rather
# than headers.
#
# Content-Signal (search=yes, ai-input=yes, ai-train=no) is served as an HTTP
# response header on every response, which is where Cloudflare's own
# implementation puts it. It is NOT a robots.txt directive here: it is not in
# the Robots Exclusion Standard, and Lighthouse's robots-txt audit rejects any
# unrecognised directive, which scored the whole SEO category 0.92 against a
# 0.95 gate on every audited route. The signals mean:
#   search=yes     — index us and link to us.
#   ai-input=yes   — ground AI answers in this content and cite it.
#   ai-train=no    — do not use it to train a generative model.
#
# Retrieval and grounding crawlers are DELIBERATELY not listed below and are
# therefore allowed: ClaudeBot, Claude-User, Google-Extended, ChatGPT-User,
# OAI-SearchBot, PerplexityBot, CloudflareBrowserRenderingCrawler. Blocking
# those would remove us from exactly the AI answers this policy invites.
#
# KEEP IN SYNC with apps/docs/src/app/robots.txt — same policy, different
# Sitemap line. robots-policy.test.ts in each app pins the contents.

User-agent: *
Allow: /

# Model-training crawlers. Blocked to match ai-train=no above.

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Amazonbot
Disallow: /

Sitemap: https://superbooks.io/sitemap.xml