Designing Websites for AI Crawlers: How to Get Cited by ChatGPT, Perplexity, and Google’s AI Overviews Without Ruining the Human Experience

For fifteen years, web design had one non-human audience to please: Googlebot. You built a site people could use, sprinkled in the right technical signals, and hoped to climb the ten blue links. That world is quietly ending. In 2026, a growing share of your visitors never see a list of links at all. They ask ChatGPT, Perplexity, Gemini, or Google’s own AI Overviews a question and get a synthesized answer with two to seven sources attached. If your site is one of those sources, you win a highly-qualified visitor. If it isn’t, you’re invisible — no matter how beautiful your homepage looks.

This is the shift almost no design conversation is having. Everyone is arguing about bento grids and dopamine colours. Meanwhile, the more consequential question for any business website is this: when an AI reads your page, can it understand you, trust you, and quote you?

This article is a practical, tested guide to designing and structuring a site so AI systems can find it, parse it, and cite it — without turning your pages into robotic, keyword-stuffed sludge that real humans hate. The good news, which we’ll return to repeatedly, is that these two goals mostly point in the same direction.

---

First, understand how an AI actually reads the web

Before you change a single line of code, you need a clear mental model of what happens when someone asks an AI assistant a question. It is nothing like a traditional Google search, and the difference dictates everything else.

Most AI search systems work through retrieval-augmented generation (RAG). Rather than inventing an answer from memory, the model retrieves relevant, indexed web pages in real time and then writes a summary based on what it pulled, with citation links back to the sources. Google has confirmed its AI Overviews and AI Mode work this way, drawing from the same Search index that powers ordinary results. In Google’s own framing there is no separate “AI index” — if your content isn’t earning placement in the regular index for a given topic, it cannot be retrieved and it cannot be cited.

The second mechanic is query fan-out. The AI doesn’t search for the exact sentence the user typed. It breaks the question into several related sub-queries and runs them concurrently. Google’s published example: a user asking how to fix a lawn full of weeds might trigger simultaneous searches for best herbicides, removing weeds without chemicals, and preventing weeds in a lawn. The final answer is stitched together from the results of all those sub-searches.

This single fact has a profound design consequence. According to an Ahrefs analysis of 863,000 keyword result pages, only 38% of pages cited in AI answers actually rank in the top ten for the primary query — down from 76% in mid-2025. In other words, a page ranking number one for the headline question can still lose citation slots to pages that answer the sub-questions better, even if those pages rank lower overall. Comprehensive, well-organized pages that address a topic from several angles now beat pages narrowly optimized for a single phrase.

Hold onto that mental model — retrieval, then fan-out, then synthesis — because every recommendation below serves it.

---

The uncomfortable debate you should know about before you start

Here’s where most articles on this topic mislead you, and where NetworkUstad readers deserve the honest version.

There is a genuine, unresolved tension in the industry right now between what Google says and what independent studies measure. You need both sides to make good decisions.

Google’s position: In May 2026, Google published its first official guide to optimizing for generative AI features, and its message was blunt. Optimizing for AI search “is still SEO.” According to the guide, there is no special schema.org markup you need to add, you don’t need to create llms.txt or other machine-readable files, and you should not chop your content into tiny fragments (“chunking”) for AI to understand it. Google states its systems can already understand the nuance of multiple topics on a page and surface the relevant piece. Google engineers, including Gary Illyes, have repeated at conferences that “AEO” and “GEO” are not separate disciplines requiring separate frameworks.

What independent researchers found: A controlled experiment by Search Engine Land tested three near-identical pages that differed only in schema markup. Only the page with well-implemented structured data appeared in a Google AI Overview — and the page with no schema failed to get indexed at all. A separate landmark academic study (“GEO: Generative Engine Optimization,” presented at ACM SIGKDD by researchers from Princeton, Georgia Tech, the Allen Institute for AI, and IIT Delhi) found that adding statistics, citations, and direct quotations were the three most effective techniques for increasing a page’s visibility in AI-generated answers.

So who’s right? Both, in a way that’s easy to reconcile once you see it. Google is telling you there is no secret markup that magically buys you a citation, and that gaming the system with fake “mentions” or thin fragmented pages won’t work — its spam systems catch that. That is true. But structure, clarity, verifiable facts, and clean schema still help, because they make your content easier for any synthesis engine to extract and trust. The honest synthesis is this: you cannot trick your way into AI citations, but you can absolutely make your genuinely good content dramatically easier to cite. Everything below is built on that principle. None of it is a trick. All of it also happens to make your site better for humans.

---

Part 1: Make sure the machines can actually reach your content

This is the most common — and most embarrassing — failure. You can write the best page on the internet, but if AI crawlers can’t fetch it, nothing else matters.

Check that you aren’t accidentally blocking AI bots

Many sites block AI crawlers without knowing it. A frequent culprit is your CDN: Cloudflare at one point changed its default configuration to block AI bots automatically, meaning a large number of site owners had their AI crawler traffic silently switched off.

The major AI systems use named crawlers, and — this is the subtle part — reading your content and training on your content are often different bots you can permit separately. If you want to appear in ChatGPT’s search experience, you need to allow OpenAI’s search crawler (OAI-SearchBot); its training crawler is the separate GPTBot. Anthropic similarly runs Claude-SearchBot for search retrieval and ClaudeBot for training. Perplexity uses PerplexityBot, and Google’s AI training access is governed by Google-Extended. You can decide independently whether to allow training crawlers while still permitting the search-retrieval bots that get you cited.

Open your robots.txt and check your server logs for these user agents to confirm they’re actually visiting. If you use Cloudflare, its dashboard has an “AI Crawl Metrics” page that shows you exactly which AI bots are reaching your site.

One critical warning from Google: because AI Overviews and AI Mode share the same crawler and index as ordinary Search, you cannot block the AI features while keeping your classic rankings. It’s one crawler, one index. Blocking it costs you both.

Serve your content in the HTML, not just in JavaScript

This is the single biggest technical trap in modern web design. Most AI crawlers do not execute JavaScript. They read the raw HTML your server returns and nothing more. If your critical content is rendered client-side by a React or Vue single-page app with no server-side rendering, the crawler sees an empty shell — and so does the AI. Independent analyses in 2026 (including work from Vercel/MERJ examining hundreds of millions of AI crawler fetches, and a Lantern study bluntly titled “AI Crawlers Do Not Render JavaScript”) confirm this repeatedly.

You can test your own site in ten seconds. From a terminal, run:

bash

curl -s https://yoursite.com/your-page | grep "a distinctive sentence from your page"

If that returns nothing, your content isn’t in the HTML, and you have a rendering problem. The fix is server-side rendering (SSR) or static site generation (SSG) — frameworks like Next.js, Nuxt, and Astro handle this well — so the meaningful text arrives in the initial HTML response.

Keep pages fast and stable

Speed doesn’t replace relevance, but a slow page limits how deeply a crawler explores your site, and Core Web Vitals remain part of the page-experience picture. Google’s 2026 guidance explicitly notes that emerging browser-based AI agents evaluate pages by analyzing screenshots and DOM structure — a slow, layout-shifting, poorly-structured page fails both the agent and the human. The hygiene that helps one helps the other.

---

Part 2: Structure pages so AI can extract and quote them

Once a crawler can reach your content, the next question is whether it can cleanly lift a passage to quote. This is where thoughtful design and content structure earn their keep — and, notably, where the human-experience benefits are most obvious.

Lead with the answer. The first one or two sentences of each section should state the key fact or conclusion directly, before the context and nuance. Synthesis engines pulling a snippet grab the top of each chunk, so front-loading your answer makes you far more quotable. Humans skimming on a phone appreciate exactly the same thing.

Use a clean, sequential heading hierarchy. One <h1>, then <h2> and <h3> in order with no skipped levels. Headings are how both extraction algorithms and screen readers identify where one idea ends and the next begins. Descriptive headings phrased the way people actually ask questions (“How do AI crawlers read JavaScript?”) tend to align neatly with the sub-queries fan-out generates.

Note the nuance on “chunking.” You’ll see advice telling you to slice content into tiny fragments for AI. Google explicitly pushed back on this in 2026, stating there’s no requirement to break content into tiny pieces and that its systems understand multiple topics on one page. The resolution: don’t artificially fragment your writing, but do use clear sections and logical structure. Well-organized prose serves extraction and reading simultaneously; artificial fragmentation serves neither.

Use lists and tables for structured data — where they genuinely fit. Bullet points and real <table> elements are easier for a machine to parse than the same information buried in a dense paragraph. This is a place to apply judgement, not a licence to bullet everything: reach for a list or table when the content is genuinely a set of parallel items or a comparison, and write in prose the rest of the time.

Wrap code in semantic elements. If you publish technical content — very much NetworkUstad’s territory — put code inside <pre><code> blocks. Models are trained to recognize these as executable examples and cite them more reliably than code mashed into ordinary text.

Write descriptive internal link anchors. “Click here” tells an AI nothing. “See our guide to configuring robots.txt for AI crawlers” tells it precisely what sits at the other end, helping it map how your pages relate. Strong internal linking within a topic also signals which of your pages is the foundational one on a subject.

---

Part 3: Give AI a reason to trust you — E-E-A-T and structured data

Reachability and structure get you into the running. Trust signals decide whether you get chosen. AI systems are, sensibly, cautious about which sources they put their name behind.

Experience, Expertise, Authoritativeness, Trustworthiness

E-E-A-T is the closest thing to a stated quality standard for AI source selection, and Google has been sharpening the authorship signals specifically. In February 2026 it added an “Authors” section to its Search Central documentation — the clearest official signal yet that transparent authorship is a direct quality consideration. Three concrete moves follow from this:

Every article you want cited should carry a named author with a verifiable profile — a real bio page, ideally linked to a LinkedIn profile or published work — not an anonymous “admin” byline. Second, include first-hand experience markers: original data, your own testing, first-person case studies, or primary-source detail that AI-generated content simply cannot replicate. Google draws a sharp line here between commodity content (“7 Tips for First-Time Homebuyers”) and non-commodity content (“Why We Waived the Inspection and Saved Money”) — the latter, rooted in genuine experience, is what earns retrieval. Third, keep your brand identity consistent across the web — the same name, description, and details everywhere AI systems check.

The blunt test Google offers, and a genuinely useful one to run on any page before you publish: “Is this something only we could have written?” If a model could have assembled the same page from everyone else’s content, it has little reason to cite yours specifically.

Structured data: useful, not magical

Here’s the balanced take the schema debate needs. Google is explicit that structured data is not required for AI features and that no schema type guarantees a citation. That’s true, and you should ignore anyone selling schema as a magic citation trigger.

But structured data still does real work. It gives AI systems a clean, unambiguous, machine-readable version of your facts — your author, your dates, your organization, the entities you’re discussing — so the answer they generate matches what you actually said instead of a guess. Multiple 2026 analyses associate proper schema with meaningfully higher rates of appearing in AI answers, and that controlled Search Engine Land test showed the schema page cited while the schema-less page went unindexed. Structured data also remains your ticket to rich results in ordinary Search, so there’s little downside.

For most publishers, you don’t need an exotic schema zoo. The two that are unambiguously safe and beneficial are Article (or BlogPosting) on your content pages and Organization on your homepage. A minimal Article block in JSON-LD — the only format worth using, since it lives in its own clean script tag rather than tangling with your HTML — looks like this:

html

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Designing Websites for AI Crawlers",
  "author": {
    "@type": "Person",
    "name": "Your Name",
    "url": "https://networkustad.com/author/your-name"
  },
  "datePublished": "2026-07-11",
  "dateModified": "2026-07-11",
  "publisher": {
    "@type": "Organization",
    "name": "NetworkUstad",
    "url": "https://networkustad.com"
  }
}
</script>

The Organization schema on your homepage is arguably the highest-leverage piece, because it establishes your brand as a verified entity in Google’s Knowledge Graph. Add the sameAs property linking to your official social profiles and a knowsAbout property declaring the topics you have genuine expertise in. Some richer schema types (Review, FAQ, HowTo, Product) carry their own eligibility rules and have had their display benefits narrowed in recent core updates, so treat those case by case rather than bolting them on everywhere.

If setting up server-side rendering, schema, and consistent entity signals sounds like more than you want to take on in-house, this is exactly the kind of work a specialist studio handles day to day — for example, a web design richmond team can build these foundations in from the start rather than retrofitting them later.

---

Part 4: Freshness is now a ranking signal in its own right

One finding cuts across nearly every credible 2026 source, and it’s easy to act on: AI systems have a strong recency bias.

A Seer Interactive study analyzing over 5,000 URLs cited by ChatGPT, Perplexity, and Google’s AI Overviews found that nearly 65% of AI crawler hits targeted content published or updated within the past year, and almost 90% targeted content from the past three years. Google’s AI Overviews showed the strongest preference of all for recently updated content. Data from citation-tracking platforms points the same way, with pages older than three months seeing a marked drop in AI citations.

There’s an important caveat that stops this from being a blanket rule: freshness is contextual. The Seer research found rapidly-changing fields like financial services show extreme recency bias, while evergreen “how-to” and instructional content can keep attracting AI crawlers for years — in some cases, a decade later. For a technology and business site, much of your content sits on the fast-moving side, so treat your important pages as living documents. Review them at least quarterly, refresh statistics, replace dated examples, and make sure your visible “last updated” date and your schema dateModified reflect real changes — not a cosmetic date bump, which fools no one.

---

Part 5: Measure what’s actually working

You can’t improve what you don’t track, and AI visibility doesn’t show up in a traditional rank tracker. Use two free tools you likely already have.

Google Search Console now includes a Generative AI performance report. Traffic from AI Overviews and AI Mode is folded into your overall Search data under the “Web” search type, so you can see how people discover you through Google’s AI features.

Google Analytics (GA4) lets you monitor referral traffic from AI platforms — ChatGPT, Perplexity, Claude, Gemini, Copilot — by source and landing page. This is genuinely strategic: the pages already earning AI referral traffic are telling you exactly what these systems like about your site. Expand those topics and strengthen those pages.

Beyond the dashboards, do the manual check that no tool fully replaces. Pick a handful of high-value questions your customers actually ask, run them across ChatGPT, Perplexity, Gemini, and Google, and record whether you’re mentioned and how you’re framed. Because AI answers are non-deterministic — ask the same question five times and you can get five different answers — you’re tracking a share of voice over many queries, not a single fixed rank. One healthy warning from Google itself: be sceptical of third-party tools claiming to use “internal” Google metrics or promising guaranteed AI rankings. No external tool has access to Google’s internal AI systems.

---

The payoff: why this traffic is worth the effort

If AI answers reduce the number of clicks you get, why chase them at all? Because the clicks you do get are worth far more.

Multiple 2026 analyses report that visitors arriving from AI assistants convert at notably higher rates than generic organic traffic. The logic is simple: the AI has already understood the user’s need, compared options, and effectively pre-qualified the person before they ever reach your site. A citation in an AI answer also functions as an implicit endorsement — much as a backlink once signalled credibility to search engines, being repeatedly named by AI builds authority with the humans reading those answers. Lower volume, higher intent. For most businesses, that’s an excellent trade.

---

A practical checklist to take away

Work through these in order — the technical foundations gate everything above them:

1. Confirm AI crawlers aren’t blocked. Check robots.txt, CDN settings, and server logs for OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Google-Extended. Remember AI features share Google’s main crawler — blocking it costs you classic rankings too.
2. Verify your content is in the raw HTML. curl a page and grep for a real sentence. If it’s missing, implement server-side rendering.
3. Fix page speed and layout stability. Clean Core Web Vitals, no big layout shifts on mobile.
4. Lead every section with a direct answer, then add context.
5. Use one <h1> and a clean, sequential heading hierarchy with descriptive, question-shaped headings.
6. Use lists, tables, and <pre><code> blocks where the content genuinely calls for them.
7. Add Article and Organization JSON-LD schema. Include sameAs and knowsAbout on Organization.
8. Put a named, credentialed author on every article.
9. Inject genuine first-hand experience — original data, testing, case studies — so a model couldn’t have written your page from everyone else’s.
10. Refresh important pages at least quarterly and keep dateModified honest.
11. Track AI referral traffic in GA4 and the Generative AI report in Search Console; manually test your priority questions across the major AI platforms.

---

The bottom line

The most reassuring thing about designing for AI crawlers in 2026 is that it is not a separate, adversarial discipline pulling against good human-centred design. It is, overwhelmingly, the same work: reachable pages, clean structure, clear answers, genuine expertise, honest freshness. Google’s own guidance and independent research disagree about the details and the magnitude, but they converge hard on the fundamentals.

The sites that will thrive as AI mediates more of the web aren’t the ones chasing tricks or the newest “AI-only” file format that gets debunked six months later. They’re the ones producing content only they could have written, and then making it effortless for both a person and a machine to read, trust, and quote. Build for that, and you’re building for whatever the search landscape becomes next.

---

Sources and further reading

• Google Search Central — Optimizing your website for generative AI features on Google Search (official guide, May 2026) and AI Features and Your Website
• Search Engine Journal — Google’s New AI Search Guide Calls AEO and GEO ‘Still SEO’
• Search Engine Land — AI crawler optimization guide, and its controlled schema experiment
• Ahrefs — February 2026 study of 863,000 keyword SERPs and AI Overview citation patterns
• Seer Interactive — content-freshness study across 5,000+ AI-cited URLs
• “GEO: Generative Engine Optimization” — Princeton, Georgia Tech, Allen Institute for AI, IIT Delhi (ACM SIGKDD)
• Vercel/MERJ and Lantern — analyses of AI crawler JavaScript rendering behaviour