Technical SEO for generative search: Optimizing for AI agents
The New Era of Search: Moving From Indexing to Interaction For decades, technical SEO was defined by a singular goal: ensuring that search engine crawlers like Googlebot could discover, crawl, and index your pages. We obsessed over sitemaps, canonical tags, and crawl budgets to ensure that a blue link appeared on a Search Engine Results Page (SERP). However, the landscape of the internet is undergoing its most significant shift since the invention of the hyperlink. We are moving from the era of traditional search into the era of generative search. In this new paradigm, users are no longer just looking for a list of websites; they are looking for immediate, synthesized answers. AI agents—driven by Large Language Models (LLMs) like GPT-4, Claude, and Gemini—are the new “users.” They don’t just visit your site to index it; they visit to extract information, summarize it, and present it within an AI-generated interface. This is known as Generative Engine Optimization (GEO). While the underlying technical frameworks remain familiar, the way we implement them has changed. Technical SEO now requires a focus on how AI agents access, interpret, and reuse your content in real-time responses. Agentic Access Control: Managing the Bot Frontier The first step in any technical SEO strategy is controlling who has access to your data. In the past, we mainly cared about Google, Bing, and perhaps a few social media crawlers. Today, we must manage a diverse fleet of AI agents, each with different purposes. Some bots are designed to scrape the web to train future models, while others are “search bots” designed to retrieve real-time information to answer a specific user query. Managing these agents starts with your robots.txt file. This file is no longer a “set it and forget it” asset. You must decide which parts of your site are available for training and which are reserved for real-time retrieval. For example, if you want to allow OpenAI’s training bot to see your public content but keep your private or sensitive folders off-limits, your configuration would look like this: User-agent: GPTBot Allow: /public/ Disallow: /private/ However, the strategy becomes more nuanced when you distinguish between training and search. You might want to block a model from training on your data (to protect your intellectual property) but allow it to “search” your site so you can still appear as a cited source in real-time answers. For OpenAI, this means differentiating between GPTBot (Training) and OAI-SearchBot (Real-time search and citations). Understanding the Agent Landscape To optimize for the most prominent AI players, you need to recognize their specific user agents. Beyond OpenAI, two of the most significant players in the generative search space are Anthropic (Claude) and Perplexity. Here is the breakdown of the bots you should be monitoring in your logs: Claude (Anthropic) ClaudeBot: The primary crawler used for training Anthropic’s models. Claude-User: A bot that performs retrieval and search functions when a user asks a specific question. Claude-SearchBot: A dedicated search crawler for real-time information gathering. Perplexity AI PerplexityBot: The standard crawler used to discover and index content for the Perplexity engine. Perplexity-User: A specialized searcher agent that triggers when a live web search is required to fulfill a prompt. By segmenting these in your robots.txt, you gain granular control over how your brand’s knowledge is consumed by the machines that power modern search. The Emergence of llms.txt: A New Standard As the web becomes increasingly crowded with AI agents, a new proposed standard is gaining traction: llms.txt. Think of this as a “sitemap for AI.” It is a markdown-based file placed in your root directory that provides a structured, easily digestible map of your content specifically for LLMs. While it is not yet a universal requirement for Google, it is an emerging protocol that forward-thinking SEOs are already adopting. There are generally two versions of this file you should consider implementing: llms.txt: A concise document containing a map of essential links and brief descriptions. It helps an agent quickly identify which pages are most relevant to a specific topic. llms-full.txt: A more comprehensive file that aggregates the actual text content of your key pages. This allows AI agents to “read” your site’s core information without having to crawl and render every individual URL, saving their “context window” and your server resources. Even though Google’s John Mueller has indicated that llms.txt isn’t a ranking factor for traditional search yet, its adoption by platforms like Perplexity (which provides its own example at perplexity.ai/llms-full.txt) suggests that it will be a cornerstone of technical SEO for the generative era. By adopting this early, you position your site as “AI-friendly,” making it easier for agents to cite you accurately. Extractability: Making Content ‘Fragment-Ready’ In traditional SEO, we optimized for keywords and long-form engagement. In GEO, we optimize for extractability. Generative engines do not always present a full page to a user; they pull “fragments” or “chunks” of information to build a synthesized answer. If your content is buried under layers of technical bloat, an AI agent may fail to extract the answer, even if your page contains the perfect information. The Problem with Technical Bloat AI retrieval systems often struggle with three main technical hurdles: Heavy JavaScript Execution: If your core content requires complex JavaScript to render, some AI agents might “see” a blank page or a loading spinner. While Googlebot is excellent at rendering JS, many smaller AI agents are not. Keyword vs. Entity Optimization: AI agents don’t just look for words; they look for relationships between entities. A page stuffed with keywords but lacking clear definitions of who, what, where, and why is harder for an LLM to process. Weak Content Structure: Large walls of text without clear headings or semantic markers make it difficult for an agent to determine where one answer ends and another begins. Using Semantic HTML for Chunking To make your content “fragment-ready,” you should lean heavily on semantic HTML. These tags act as roadmaps for AI agents, telling them exactly which parts of the page