5 Lessons From Running AI Agents Across Every Search Engine

Deploying autonomous AI agents across Google, Bing, DuckDuckGo, Perplexity, and You.com sounds like a single integration problem until you actually do it. It isn’t. Each engine treats a JavaScript-rendered page, a robots.txt entry, and a piece of structured data differently enough that a workflow tuned for one platform quietly fails on another. After months of running agents that crawl, query, and extract across all of these systems in parallel, the gap between “search engine” as a monolithic concept and the messy reality underneath became impossible to ignore. What follows are the five lessons that mattered most — the ones that changed how I architect agents, not just how I tune prompts.

1. JavaScript Rendering Is Still the Great Divider

The single biggest source of agent failures wasn’t logic or prompting — it was content that simply never reached the model. Googlebot remains the standout performer here: it executes JavaScript through a Chrome-based rendering pipeline and eventually indexes the fully hydrated DOM, even though that rendering happens in a delayed second pass. Most AI-specific crawlers don’t get that luxury. According to GetPassionfruit’s analysis of JavaScript rendering and AI crawlers, crawlers including GPTBot, ClaudeBot, and PerplexityBot cannot execute JavaScript and only see the initial HTML response that a server sends back.

This happens because rendering JavaScript at scale is computationally expensive, and most AI companies prioritized breadth of crawl over depth of rendering when they built their bots. Bing sits in an uncomfortable middle ground: Sitebulb’s advanced SEO guide to rendering notes that Bing’s own webmaster guidance claims it can process JavaScript, but explicitly recommends dynamic rendering for large sites because of limitations in processing JavaScript at scale. In practice, that hedge shows up as inconsistent indexing on heavy single-page applications.

The practical fallout for an agent pipeline is brutal: an agent querying Perplexity or a ChatGPT-style search layer can return a citation for a page whose actual content — pricing, product specs, FAQ answers — was invisible to the crawler that indexed it. A client-side rendered single-page app sends every requester the same minimal shell containing navigation, a root div, and script tags, so for crawlers that don’t execute JavaScript, there’s effectively nothing there to extract, as GetPassionfruit found across its test cases.

Takeaway: Treat server-side rendering as a prerequisite for AI search behavior, not a nice-to-have for Core Web Vitals. If your most important facts live behind client-side JavaScript, view-source the page — if your text isn’t in the raw HTML, no autonomous agent querying that engine’s index will ever see it either.

2. Robots.txt Has Quietly Become a Policy Document, Not Just a Crawl Setting

I assumed robots.txt was solved infrastructure. It isn’t anymore. What used to be a simple exercise in keeping Googlebot out of /admin/ has turned into a multi-stakeholder decision about which AI companies get to read your content at all. As Search Engine Land’s 2026 SEO outlook puts it, robots.txt is no longer just crawl housekeeping — it’s becoming a policy surface, and the rise of llms.txt represents a new class of decision-making entirely.

Running agents across engines exposed how unevenly this policy gets enforced. Some companies are scrupulous about it: Starmorph’s AEO and GEO guide points out that Anthropic’s bots are unique in that all three honor robots.txt, including for user-initiated requests, and blocking Claude-SearchBot specifically may reduce a site’s visibility and accuracy in user search results. Others are far less reliable. WitsCode’s LLM SEO guide cites Cloudflare’s August 2025 finding that Perplexity was observed using undeclared stealth crawlers that rotated user-agents and IP addresses specifically to reach content on domains that had disallowed all bots.

There’s also a discovery dependency most teams miss entirely. AI+Automation’s breakdown of ChatGPT’s search architecture explains that ChatGPT’s URL discovery relies entirely on Bing’s index — if a page isn’t indexed in Bing, it never enters ChatGPT’s candidate URL pool, regardless of how well that page ranks on Google. An agent built only to monitor Google rankings can be blind to an entire failure mode happening one engine over.

Takeaway: Audit robots.txt per crawler, not as a blanket rule. Decide deliberately whether you want training crawlers (GPTBot), real-time citation crawlers (OAI-SearchBot, PerplexityBot), and traditional search bots treated differently, and verify enforcement with server logs rather than trusting that any of them comply by default.

3. Structured Data Is the Closest Thing to a Universal Language Between Engines

Across every engine I tested, one signal consistently improved how reliably agents could extract and reuse content: clean, explicit structured data. This isn’t a coincidence — it’s the one format every system, from traditional indexers to generative answer engines, can parse without ambiguity. Starmorph’s research identifies structured data in JSON-LD format as the most impactful technical change available, because Google explicitly recommends it and every AI engine tested prefers it for being cleanly separated from HTML and easy to parse programmatically.

The effect compounds with specific schema types. The same Starmorph analysis found that FAQPage schema makes content roughly 3.2 times more likely to appear in AI Overviews, and fully populated Product plus Review schema achieves a 61.7% citation rate in independent testing. This happens because structured data removes the inference step — instead of an agent guessing what a price, a step, or an author byline means from surrounding prose, the markup states it outright.

Plain HTML hierarchy matters almost as much. Search Engine Land’s guide to AI crawlers notes that schema is designed to be machine-readable, and although it isn’t generally visible to users, crawlers can find and parse it because it’s embedded directly in the HTML code. Agents querying different engines for the same factual answer pulled nearly identical extraction quality whenever schema was present, and wildly inconsistent quality whenever it wasn’t.

Takeaway: Don’t treat schema markup as an SEO checkbox for rich snippets. Build it as the canonical, machine-readable layer of your site that every crawler — search engine or AI agent — can fall back on when prose parsing fails.

4. Crawl Behavior and Citation Behavior Are Not the Same Thing

One of the most counterintuitive findings from running agents long enough to log crawl-to-referral ratios: the bots visiting your site most aggressively are often the ones sending you the least traffic back. Search Engine Land’s report on Googlebot and AI bot crawling in 2025 found that Anthropic showed the highest crawl-to-refer ratio among major AI and search platforms in 2025, peaking near 500,000-to-1 early in the year and settling between roughly 25,000-to-1 and 100,000-to-1 afterward. The same report noted OpenAI spiked to about 3,700-to-1 in March, while Perplexity stayed comparatively low, mostly under 400-to-1 for most of the year.

This happens because most AI crawlers exist primarily to harvest training data or to populate a retrieval index for synthesized answers — not to drive click-through the way a traditional blue-link result does. The agent’s “success” (getting cited inside an answer) and the publisher’s success (getting a visit) have quietly decoupled. That decoupling is visible in ranking behavior too: Starmorph’s GEO guide reports that in July 2025, 76% of URLs cited in Google AI Overviews ranked in the organic top 10, but by February 2026 only 38% did, with the rest pulled from positions 11 through 100 or beyond — proof that traditional ranking signals and AI citation eligibility are diverging.

Takeaway: Stop measuring AI search performance with organic-ranking dashboards alone. Track crawl volume against referral volume per bot, and treat citation appearance in generative answers as its own KPI, separate from click-driven traffic.

5. Bot Identity Is Fragmenting Faster Than Most Teams’ Configurations Can Track

Every engine used to mean one crawler. Now it means a family of them, each with a distinct job and distinct rules, and agents built around a single user-agent assumption break the moment a company splits its bot fleet. Search Engine Land’s technical SEO guide for generative search explains that a site might reasonably want to allow a training-focused bot like GPTBot into a public folder while keeping it out of a private one, and separately decide whether to allow real-time citation bots differently from training bots entirely.

This fragmentation happens because AI companies have split commercial incentives — training data acquisition, live retrieval for chat answers, and user-triggered browsing are legally and operationally distinct activities, so the bots performing them increasingly carry different names, permissions, and crawl patterns. AI+Automation’s research on ChatGPT’s architecture details how OpenAI’s ChatGPT-User bot fetches pages live during conversations, doesn’t execute JavaScript, and as of December 2025 ignores robots.txt restrictions entirely, on the stated rationale that live browsing serves direct user intent rather than bulk crawling.

Running agents that needed to reconcile these differences meant maintaining a living map of bot identities rather than a static config file. Search Engine Land’s 2026 SEO outlook found that GPTBot’s desktop share rose roughly 55% year-over-year by 2025, while ClaudeBot’s share nearly doubled over the same period — these aren’t stable numbers you configure once and forget.

Takeaway: Build crawler-permission logic as a maintained, versioned asset — reviewed monthly — not a robots.txt file written once during a site launch. The bots changing identity and behavior fastest are exactly the ones most likely to break an agent pipeline that assumes yesterday’s rules still apply.

Strategic Implications

The throughline across all five lessons is the same: “search engine” stopped being a single technical target the moment AI agents entered the picture. Rendering capability, crawl permissions, structured data support, citation behavior, and bot identity now vary enough between Google, Bing, Perplexity, and the rest that a one-size strategy guarantees blind spots somewhere. Teams running autonomous agents need infrastructure that treats each engine as a distinct integration with its own constraints, not a uniform API. The publishers and developers who map these differences explicitly — rather than assuming parity — are the ones whose content actually survives the transition from ranked links to synthesized answers.

FAQ

Do AI crawlers like GPTBot and PerplexityBot render JavaScript the way Googlebot does?
No. A widely cited study from Vercel and Merj analyzing requests from major AI crawlers found that most can fetch JavaScript files but do not execute them, meaning GPTBot, ClaudeBot, and PerplexityBot do not currently fully render JavaScript content, while Googlebot remains the strongest performer at full rendering.

Does blocking an AI crawler in robots.txt actually stop it from accessing my content?
Usually, but not always. Per Search Engine Land’s guide to AI crawlers, most reputable AI companies claim to respect robots.txt, though they aren’t strictly required to, and some — including OpenAI and Anthropic — have acknowledged ignoring robots.txt directives in the past before later committing to honor them. Treat the file as a strong preference enforced by compliant actors, not an unbreakable wall.

Is structured data really worth prioritizing for AI search visibility?
Yes, more than almost any other technical change. Starmorph’s GEO research found that FAQPage schema alone makes content roughly 3.2 times more likely to appear in AI Overviews, and well-populated Product and Review schema correlates with a 61.7% citation rate — among the highest-leverage, lowest-effort changes available to publishers.