#
llms.txt
llms.txt is a proposed plain-text file placed at a site root that tells AI systems which pages matter most and how the site is organized. Unlike robots.txt, which controls access, llms.txt curates priority content for language models. Adoption is early but growing, and publishing one is a low-cost readiness signal.
#
robots.txt for AI Bots
robots.txt for AI bots is the practice of using user-agent rules to allow or block AI crawlers such as GPTBot, ClaudeBot, and PerplexityBot. Blocking removes a site from training and retrieval; allowing enables citations. Each engine publishes its own user-agent strings, so rules must be maintained per bot.
#
GPTBot
GPTBot is the web crawler operated by OpenAI to collect content for model training, with related agents handling live search fetches for ChatGPT. Site owners control it through robots.txt user-agent rules. Whether to allow GPTBot is a strategic choice between content protection and visibility inside ChatGPT answers.
#
AI Crawlers
AI crawlers are bots that fetch web content for AI systems — GPTBot and OAI-SearchBot for OpenAI, ClaudeBot for Anthropic, PerplexityBot for Perplexity, Google-Extended for Gemini training. They split into training crawlers and live-retrieval fetchers; server logs reveal which ones actually visit a site and how often.
#
Structured Data
Structured data is machine-readable markup — most commonly JSON-LD using Schema.org vocabulary — that labels the entities and facts on a page: who wrote it, what it describes, when it changed. Structured data removes guesswork for search engines and AI systems, improving eligibility for rich results and accurate citations.
#
JSON-LD
JSON-LD (JavaScript Object Notation for Linked Data) is the recommended format for embedding structured data in a page: a script block, separate from visible HTML, describing entities and their relationships. Google explicitly prefers JSON-LD over microdata and RDFa because it is easier to write, validate, and maintain.
#
Schema.org
Schema.org is the shared vocabulary — founded by Google, Microsoft, Yahoo, and Yandex — that defines the types and properties used in structured data: Article, Person, Organization, FAQPage, Product, and hundreds more. It is the common language a page uses to state facts machines can rely on.
#
JSON-LD @graph Method
The @graph method combines all structured data on a page into one JSON-LD block where every entity has an @id and entities reference each other — the WebPage links to its author, the author to the Organization. Connected graphs resolve entities more reliably than isolated schema fragments scattered across a page.
#
FAQPage Schema
FAQPage schema is structured data marking question-and-answer pairs on a page. Google now shows FAQ rich results only for a narrow set of authoritative sites, but the markup remains valuable for AEO: it hands answer engines pre-packaged, extractable Q&A units that match how users phrase queries.
#
HowTo Schema
HowTo schema is structured data describing step-by-step instructions: steps, order, tools, and estimated time. Google retired HowTo rich results in 2023, yet the markup still helps AI systems parse procedural content into clean, ordered steps — the format assistants reuse when answering how-to questions.
#
Speakable Markup
Speakable markup is structured data (SpeakableSpecification) pointing to the passages of a page best suited for text-to-speech — short, self-contained summaries an assistant can read aloud. Support is limited and news-oriented, but the discipline it enforces, writing concise extractable blocks, benefits every answer surface.
#
Canonical URL
A canonical URL is the single address declared as the authoritative version of a page when duplicates or variants exist. Consistent canonicals concentrate signals on one URL — critical for AI retrieval, because engines cite one address, and split signals mean weaker passages competing against each other.
#
Crawlability
Crawlability is the degree to which bots can fetch and read a page: reachable URL, permissive robots rules, fast HTML response, and content that does not require JavaScript rendering. For AI search it is the gate condition — an unreachable page cannot be retrieved, grounded against, or cited.