When an artificial intelligence engine like ChatGPT generates a response complete with clickable citations, where does that information actually come from? While many assume AI search operates either through direct web crawls or traditional search API partnerships, the true architecture under the hood is significantly more complex.
To uncover the real mechanics behind AI citations, digital marketing and data intelligence firm RESONEO conducted an in-depth empirical study. By analyzing 1,200 ChatGPT conversations, 88,000 search results, and 26,900 distinct web pages, researchers reverse-engineered the exact retrieval pipelines powering the platform. The investigation revealed a distinct three-layer infrastructure: a custom discovery index that identifies potential sources, a shared reading cache that stores full rendered pages, and an on-demand live browser that inspects a tiny fraction of candidate URLs in real time.
Each component operates under its own distinct constraints, caching rules, and structural blind spots. Understanding this multi-tiered architecture provides clarity on why certain pages are discovered, why others are read in full, and why only a select few end up as visible citations.
Reverse-Engineering ChatGPT’s Retrieval Pipelines
The discovery of OpenAI’s internal routing began with an analysis of the raw data payload transmitted between OpenAI’s servers and the client interface. Using a custom browser extension to inspect undocumented telemetry, the research team found an internal metadata property labeled result_source.
This single parameter revealed the internal backend engine responsible for handling every web lookup. Four specific source values appeared consistently across the datasets:
labradorbrightoxylabsserp
While OpenAI publicly references generic “third-party search providers,” these internal designations expose the discrete vendors and custom systems running behind the scenes. However, shortly after researchers began cataloging the stream, OpenAI removed the result_source parameter entirely. To continue tracking data lineage, the team engineered a machine-learning classifier based on formatting signatures—such as snippet character limits, title structures, and URL shapes—achieving a 98% accuracy rate in classifying pipeline origins. Researchers supplemented this with server-side canary page deployments and fleet-level API comparisons to verify bot behavior in real time.
The Retrieval Engine: Mapping Labrador, Bright, and Vertical Feeds
ChatGPT does not rely on a single, monolithic web index. Instead, it dynamically orchestrates multiple internal hubs, data scrapers, and vertical databases based on query intent.
The research mapped the primary retrieval engines into distinct categories:
- Labrador: OpenAI’s proprietary search and retrieval infrastructure. Labrador serves as an orchestrator that pulls from in-house indexes, structured repositories like Wikipedia, academic databases like arXiv, direct social streams such as Reddit, and media platforms like YouTube.
- Bright and Oxylabs: Paid third-party web scraping pipes that execute live Google Search scrapes to retrieve real-time SERP rankings.
- P1, P2, and P3 Pipelines: Dedicated e-commerce engines that query OpenAI’s merchant data feeds, using third-party search engines primarily to verify pricing accuracy and customer reviews.
- B1/B3, Yelp, and TripAdvisor: Local business routing layers. Interestingly, while local data is pulled from multiple local directories, the links displayed in the user interface often redirect straight to Google Maps.
Crucially, specialized verticals like local directory search and shopping feeds operate entirely outside the standard web retrieval stack. For organic visibility, the real action takes place in the general web pipelines, where architectural nuances dictate brand inclusion.
How Model Modes and the “Think” Feature Alter Search Results
The introduction of specialized reasoning models and user interface options, such as the “Think” button, fundamentally changes how OpenAI gathers external data. Comparing user tiers reveals that the volume of sources retrieved and the origin of those sources vary dramatically.
On free accounts, activating the Think option more than doubles the retrieval footprint, increasing the average source volume from 15.1 to 35.3 URLs per conversation and expanding unique domains from 9.8 to 16.3. This aligns free reasoning capabilities with the source volume of paid Thinking modes at medium effort levels.
However, the underlying data sources diverge substantially between tiers:
- Free “Think” Mode: Retrieves 74.7% of its data directly from OpenAI’s proprietary
labradorindex, 22.2% fromoxylabsnews streams, and only 3.1% from standard scraped Google search results. - Paid “Thinking” Mode: Flips this dynamic, drawing 75.3% of its sources from live Google scrapes and only 24.7% from the internal
labradorindex.
This variance has immediate consequences for search optimization. Ranking high in Google SERPs gives a website strong visibility in paid subscription queries, but provides no guarantee of visibility for the overwhelming majority of free users whose answers are dominated by the Labrador index.
Additionally, search patterns have become increasingly direct. At higher effort tiers, the use of the site: search operator rose from 40.8% to 58.1%. Rather than casting a wide net, the model actively isolates specific authoritative domains and brand properties it already recognizes as trustworthy.
Comparative testing between web chat interfaces and direct OpenAI API endpoints also revealed significant discrepancies. Brand mentions between API runs and user-facing sessions showed a Jaccard similarity score of only 0.23 to 0.27. Probing the raw API demonstrates what the underlying weights know, but it does not mirror the live product’s grounding behavior.
Inside the Labrador Index: Titles, Snippets, and Structural Quirks
A common industry assumption was that OpenAI relies entirely on the Bing Search API. The data proves otherwise. When comparing Labrador results against Bing rankings for identical fan-out queries, only 1.5% of Labrador URLs appeared in Bing’s top 20 results.
Furthermore, technical formatting signatures show clear operational differences:
- Bing caps title lengths at 75 characters; Labrador preserves full, untruncated titles, with 24% exceeding 75 characters and some spanning up to 289 characters.
- Labrador does not generate query-dependent dynamic snippets. Instead, it delivers a static snippet of approximately 200 characters, cut at initial indexation time.
- Standard meta descriptions are completely ignored by the Labrador index, whereas Google-scraping pipelines utilize meta descriptions in roughly one out of every three results.
Labrador constructs its 200-character snippet by anchoring directly to the page’s primary H1 heading and extracting whatever adjacent text appears immediately afterward in the raw HTML. This extraction frequently captures breadcrumb labels, image alt text, bylines, publication dates, or navigation links instead of the core article text. Until this extraction pipeline matures, the initial 200 characters following an H1 directly define what the AI reads during fast queries.
Economic Routing: The Cost of Deep Search
The differences between quick and advanced modes come down to computational and financial efficiency. For free “Instant” queries, response times must remain under a few seconds with minimal server overhead. OpenAI relies on its in-house index, opening zero live external pages in 93% of analyzed instant responses. In these scenarios, the model crafts its answer relying solely on the URL, the page title, and the static 200-character snippet.
When a paying user triggers an intensive reasoning query, ChatGPT switches to paid scraping pipelines and activates its live fetching bot (ChatGPT-User). Across the study’s dataset:
- 61,332 URLs were retrieved into the background context.
- 5,032 URLs served as the primary source behind an output citation.
- 759 pages were opened and read in full.
Conversion rates along this funnel are revealing: a web page that is retrieved but never opened has a 7% chance of being cited. If the system actually opens and reads the page, the citation rate surges to 74%.
The Global Reading Cache: Markdown Conversions and Rendering Rules
In collaboration with technical SEO specialist Jérôme Salomon of Oncrawl, the research uncovered an intermediate caching layer that sits between search discovery and live web browsing.
When ChatGPT fetches a web page, it converts the raw HTML into cleaned Markdown and saves it inside a global, shared reading cache. This cache is universal: if an enterprise user in North America triggers a crawl for a document, a free-tier user querying that topic in Europe days later receives data served from that exact cached version.
The cache operates on a stale-while-revalidate pattern:
- Cached documents remain completely fresh for approximately 30 minutes.
- Subsequent queries after 30 minutes immediately read the stale version while silently triggering a background crawl to refresh the cache for future users.
- Pages subject to frequent user inquiries remain updated, while obscure or low-demand pages retain cached snapshots that can exceed 90 days of age.
- Standard HTTP directives like
Cache-Control: no-storeand traditionalnoindextags are bypassed by the caching layer.
The Markdown transformation process enforces strict structural guidelines:
- Stripped Elements: Embedded JavaScript, iframes, and structured JSON-LD data are completely removed and never reach the model.
- Preserved Content: Image alt text, core text, and text hidden via CSS are fully retained.
- JavaScript Rendering: The fetching crawler does not run a headless browser; non-SSR (Server-Side Rendered) client-side content is invisible.
- File Size Constraints: Any document exceeding a hard cap of 4 MB is rejected outright with an HTTP 400 error. It is not truncated—the system simply refuses to read the page.
The Web Analytics Tracking Blind Spot
Many digital publishers monitor incoming AI traffic by tracking utm_source=chatgpt.com parameters. However, tracking analytics through query parameters captures only outbound referral clicks from users, missing when the model reads the page.
Crucially, citations generated from deep, full-page reads in advanced reasoning modes frequently omit the UTM string entirely from user-facing links. Relying solely on referral tags leaves a major blind spot; system visibility requires monitoring server logs directly for user-agent activity tied to the ChatGPT-User crawler.
Anomalous Citations: Snippet-Free Sources and Parametric Memory
The analysis surfaced an unexpected trend regarding snippet extraction. A notable segment of results reaches the model containing only a URL and a page title, with no contextual text snippet attached.
In instant query modes, these snippet-less URLs achieve a 14.9% citation rate, compared to just 8.2% for results with full snippets. The AI system frequently prefers citing pages it has technically read the least about in real time.
Additionally, several displayed citations corresponded to no live search retrieval event in the logs. In these cases, the model appears to generate top-level root URLs directly from its internal parametric weights—the training memory embedded in the neural network—and appends tracking parameters organically. Highly authoritative domains are regularly cited not because they won a real-time search query, but because the model inherently remembers them as topical leaders.
Actionable Takeaways for AI Visibility and SEO Strategy
Optimizing for LLM discovery requires adjusting traditional on-page formatting while focusing on long-term authority and crawl accessibility.
Technical and structural recommendations include:
- Structure Post-H1 Content: Ensure that the first 200 characters immediately following the main H1 contain the primary factual answer, core value proposition, or definition. Avoid placing navigation trees, banner ads, and publication metadata directly beneath headings.
- Descriptive Titles: Write comprehensive, descriptive page titles. Because Labrador does not truncate title strings, long and descriptive headlines provide rich contextual signals for query matching.
- Preserve Server-Side Rendering: Avoid building core informational content on client-side JavaScript. Keep page weights well below the 4 MB limit to prevent outright rejection by the crawler.
- Maintain Dual Metadata: Retain compelling meta descriptions for Google-fed reasoning pipelines, but do not rely on them for Labrador-based queries.
- Optimize Image Alt Attributes: Treat image alt tags near the top of the page as primary body text, as they consistently pass through the Markdown converter.
Focusing on Conversational Intent
While fine-tuning on-page snippets offers short-term technical advantages, long-term visibility across AI platforms depends on mapping natural conversational queries.
Consumers rarely search AI assistants using rigid keyword syntax or raw product specifications. Instead of searching for technical product specs, users ask natural, problem-solving questions regarding compatibility, comfort, and real-world durability. The brands that maintain visibility are those that identify real customer inquiries across reviews, support tickets, and community forums, publishing clear, server-readable answers that match conversational intent.
As OpenAI’s retrieval architecture continues to evolve, technical pipelines may change, but the core fundamentals remain stable: clear document structures, accessible server-rendered text, and comprehensive answers to real-world user questions will continue to drive discoverability.