Site migrations in 2026 have a new way to go wrong

Key takeaways

• A 2026 site migration can keep Google rankings intact and still cut off AI search tools, through a reset robots.txt file, missing redirects or a hosting default that blocks AI bots.
• AI crawlers come in types: training bots like GPTBot, search bots like OAI-SearchBot and Claude-SearchBot, and user-triggered fetchers like ChatGPT-User. Blocking the search bots is what removes you from AI answers.
• Google-Extended is a robots.txt token, not a crawler, so it never shows up in server logs; Googlebot does the fetching.
• Cloudflare began blocking known AI crawlers by default on new domains on July 1, 2025, so a migration that changes hosting or DNS settings can switch them off silently.
• Protect AI visibility with a robots.txt snapshot, server-log analysis of AI bots, one-hop 301 redirects for every changed URL and two weeks of post-launch monitoring.

A successful website migration is invisible, and a failed one can be costly. In 2026 there is a new way to fail: keeping your Google rankings while silently losing access for AI crawlers, and with it your presence in AI-generated answers. This Emulent guide explains how that happens and how to add AI crawler checks to your migration plan.

Why does your migration checklist probably miss AI crawlers?

Most checklists were written when Google was the only search engine that mattered, so they protect Googlebot and link equity but not AI bots. They cover 301 redirects, canonical tags, XML sitemaps and crawl budget. Those steps still matter, but they do not cover the separate crawlers that feed ChatGPT, Claude, Perplexity and Gemini.

A migration is a technical SEO project, and in 2026 it is also an AI visibility project. Teams that protect Google rankings while ignoring AI crawler access solve half the problem. These traditional checklist items leave AI crawlers exposed.

• Teams copy the old robots.txt without checking which bots it blocks, and a stray rule can cut off AI crawlers for weeks before anyone notices.
• Redirect plans focus on high-authority pages and skip lower-traffic pages that AI tools cite.
• Pre-migration audits check Googlebot in the logs but rarely check AI bots.
• Some platforms and hosting providers now block AI crawlers by default, so a fresh setup during migration can switch them off.

That last risk is real. Cloudflare made AI crawler blocking the default for new domains starting July 1, 2025. A migration that moves a site to a new Cloudflare zone or account can inherit that setting, and Google Search Console will never show it.

What are AI crawlers doing when they visit your site?

AI crawlers collect content for model training, for search indexes behind AI answers, or for a single user’s request, and each job uses a different bot. OpenAI’s crawler documentation separates GPTBot, which gathers content that may be used for training, from OAI-SearchBot, which surfaces sites in ChatGPT search. OpenAI says sites that opt out of OAI-SearchBot will not appear in ChatGPT search answers. A third agent, ChatGPT-User, fetches pages when a user asks, and OpenAI notes robots.txt rules may not apply to it. Anthropic follows the same pattern with ClaudeBot, Claude-SearchBot and Claude-User, and Perplexity uses PerplexityBot for its index and Perplexity-User for live requests.

Google works differently. Google-Extended is a robots.txt token that controls whether content Googlebot already fetched can be used for Gemini training and grounding. It has no user agent of its own, so you will not find it in server logs, and it does not affect Google Search or AI Overviews, which use ordinary Googlebot. Google lists its crawlers and tokens in its overview of Google crawlers.

• Training crawlers may fold your content into future models, shaping how they connect your brand with topics.
• Search and retrieval crawlers build the indexes AI tools search at answer time, so a block removes you from live answers.
• Over time, models associate your brand with products, competitors and industry topics based on what they have read.
• When AI tools cite sources, the links come from their indexes, so URLs that change without redirects can leave citations pointing at dead pages.

The crawlers that respect robots.txt rely on it entirely, so one wrong line blocks them. They also read canonical tags, and long redirect chains waste their crawl attempts.

How can a migration erase your AI citation history?

By changing the URLs AI tools cite without clean redirects. If ChatGPT or Perplexity links to /old-blog/topic-name and that URL now returns a 404, the citation breaks. AI indexes may also refresh more slowly than Google’s, so broken links can linger in answers for weeks. Think of AI citation history the way you think of a backlink profile: it took time to earn and can disappear quickly.

• New URL structures without 301 redirects leave crawlers no path from old pages to new ones.
• During a domain change, DNS propagation can send some requests to the old site and some to the new one, so bots may see mixed versions.
• An exposed staging site can be crawled and indexed before launch, adding thin or duplicate content.
• When pages are merged, canonical tags and redirects must point to the new consolidated URL, or tools keep treating the old pages as separate.

Protect staging with password protection or HTTP authentication rather than a robots.txt disallow alone. A disallow stops compliant bots from crawling, but blocked URLs can still be indexed if other pages link to them.

How does a robots.txt reset block AI bots without anyone noticing?

A new CMS, host or rebuild often replaces your robots.txt with the platform default, and the problem only shows up when your content fades from AI answers weeks later. The site looks normal in Google Search Console, so the team calls the migration a success. By the time someone notices, the fix is easy but the lost weeks are not recoverable. Audit before and after launch.

• Before migrating, save your current robots.txt and record every user-agent rule, including which AI bots are allowed, blocked or not mentioned (which means allowed by default).
• Within the first hour after launch, compare the new file line by line with the snapshot and review any new disallow rule.
• Check server logs for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot and PerplexityBot. If they crawled before launch and went quiet after, something is blocking them.
• Review the default robots.txt and bot settings of your new platform, CDN and firewall before launch day, and override them if they block bots you want.

In that setup, check both the robots.txt the SEO plugin generates and any bot rules at the CDN or firewall layer, because either can block AI crawlers.

How do you build redirect maps that protect AI indexing?

Treat every URL that AI bots crawl or AI tools cite as a priority, including pages without strong backlinks. Googlebot follows redirect chains and recrawls patiently. AI crawlers vary, and it is safer to give them one clean hop. Server logs show which pages AI bots visit most, and each of those needs an explicit redirect.

Pulling AI crawler logs takes minutes, and it catches robots.txt and firewall problems that Google Search Console cannot show. Use these steps.

• Filter six months of server logs for AI crawler user agents to find the pages they request most.
• Match those URLs against the new structure and give each an explicit redirect instead of relying on broad pattern rules.
• Crawl the redirect map with a tool such as Screaming Frog and collapse any chain longer than one hop into a direct 301.
• After launch, alert on any 404 returned to an AI crawler and fix it within 48 hours where possible.

What does an AI-aware migration checklist look like?

An AI-aware migration checklist runs alongside your usual technical SEO steps rather than replacing them, in three phases.

Before migration:

• Document every robots.txt rule and confirm that the AI search bots you want, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, are not blocked, and decide your policy for training bots like GPTBot and the Google-Extended token.
• Pull six months of AI crawler log data and flag the most-crawled pages as priority redirects.
• List every canonical tag that points to a URL that will change, and update them on launch day.
• Lock the staging site behind a password.

On launch day:

• Check the live robots.txt within 30 minutes and compare it with the snapshot.
• Check CDN and firewall bot settings, including any AI crawler blocking defaults.
• Test the top AI-crawled URLs by hand and confirm each returns a single 301 to the right page.
• Submit the updated XML sitemap in Google Search Console and Bing Webmaster Tools, since several AI search tools draw on Bing’s index as well as their own.

After launch:

• Watch AI crawler activity in the logs for two weeks, and investigate any sudden drop.
• Alert on 404s served to AI crawlers and fix them as they appear.
• Ask ChatGPT, Claude and Perplexity about your brand and main topics in the following weeks, and check whether citations point to live pages.

How do you protect what you built before the next migration?

Treat AI crawler access with the same care as link equity. The citations your content has earned in AI tools take time to rebuild if lost. A caveat: allowing AI crawlers is a business decision, and some publishers deliberately block training bots while allowing search bots. The goal of a migration is to keep whatever policy you chose on purpose, not to lose it by accident.

Emulent runs technical SEO audits that cover both traditional search and AI crawler access during migration planning. If you are planning a migration, contact the Emulent team to keep your search and AI visibility intact through the move.