Cloudflare Launches Redirects for AI Training to Force AI Crawlers to Follow Canonical URLs

Breaking: Cloudflare Offers New Tool to Combat AI Training on Outdated Content

Cloudflare today announced Redirects for AI Training, a feature that converts existing canonical tags into HTTP 301 redirects for verified AI training crawlers. The solution turns advisory signals into enforceable commands, ensuring AI models train only on the most current version of web content.

“Status codes are the language the web uses to communicate policy to crawlers. With one toggle, canonical tags become redirects for AI training crawlers,” said a Cloudflare spokesperson. The feature is available immediately on all paid Cloudflare plans.

Background

AI training crawlers have been consuming deprecated content at alarming rates. On Cloudflare’s own developer documentation site, AI crawlers visited 4.8 million times in the last 30 days, ingesting outdated pages as often as current ones despite clear signals.

Standard web signals like noindex meta tags, deprecation banners, and canonical tags failed to deter them. “Every advisory signal says this content is outdated, look elsewhere, but AI crawlers don’t reliably honor those signals,” Cloudflare reported.

How It Works

The <link rel="canonical"> tag, defined in RFC 6596 and already present on 65–69% of web pages, tells automated systems which URL is authoritative. Redirects for AI Training enforces that directive by issuing a 301 redirect to verified AI training crawlers.

This closes a critical gap: previously, no inline directive could say “don’t train on this.” Banners work for humans but AI crawlers read entire page text, including the warning, and often return thousands of times.

“Keeping a deprecated page live with a warning may work for humans, who read the notice and navigate on, but AI training crawlers ingest the full text and risk treating the banner as just one more paragraph.” — Cloudflare

What This Means

Redirects for AI Training turns passive signals into active enforcement. Site owners no longer need to manually block crawlers per path or per update. Instead, existing canonical infrastructure does the work automatically.

The effect is cumulative: because AI agents don’t always fetch content live but rely on trained models, feeding them current URLs prevents propagation of outdated information. For the broader web, Cloudflare’s Radar AI Insights now includes Response status code analysis, showing the distribution of 2xx, 3xx, 4xx, and 5xx codes AI crawlers receive across all Cloudflare traffic.

“What crawlers need is specific direction: ‘Here is where the current content lives,’” the company emphasized. The new feature delivers exactly that, turning the web’s existing canonical tag infrastructure into a reliable guide for AI training.

Key Benefits

• One-click toggle on all paid Cloudflare plans
• No new HTML tags required – works with existing canonical URLs
• Works with platforms like WordPress, Contentful, and EmDash that generate canonical tags automatically
• Provides a clear, enforceable alternative to robots.txt or blocking, which creates a signal void

Availability

Redirects for AI Training is rolling out now to all paid Cloudflare customers. For more details, visit the official announcement.