Learning Center

Agent web timeline

Changes outside our product that affect how AI agents use websites: standards, platforms and laws. Each says what changed, what it means for your site and which guide covers it. Follow by RSS.

  1. · Cloudflare

    Cloudflare moves new AI search crawlers into its Search category

    Cloudflare's verified bot documentation notes that since July 2026, new AI search crawlers are placed in its general Search category, beside the categories it lists for verified bots such as AI Assistant and AI Crawler.

    What it means for you: If your CDN rules allow or block bots by category, a rule written for AI crawlers may not cover new AI search crawlers. Ask your vendor which categories apply to your plan today.

    Read more: Questions to ask your CDN or bot protection vendor · Source: Cloudflare

  2. · European Union

    European Accessibility Act starts applying to many e-commerce services

    In the EU, the European Accessibility Act has applied to many e-commerce services since June 2025, so many retailers already have accessibility programs underway.

    What it means for you: Accessibility fixes and agent readiness fixes largely overlap. Add agent readiness to your existing accessibility program, since it needs the same fixes, the same developers and often the same tickets.

    Read more: Accessibility work is agent readiness work

  3. · Anthropic

    Anthropic introduces the Model Context Protocol (MCP)

    Anthropic introduced the Model Context Protocol, an open protocol for connecting AI applications to outside tools and data. It has since been adopted by many AI assistants, developer tools and software companies.

    What it means for you: An MCP server lets AI applications call actions on your business as tools, each with a name, a plain-language description and the inputs it expects, to complete a person's request.

    Read more: MCP and WebMCP: giving AI agents a direct way into your site · Source: Model Context Protocol

  4. · Answer.AI

    Jeremy Howard of Answer.AI publishes the llms.txt proposal

    llms.txt, a proposed standard, was first published by Jeremy Howard of Answer.AI. It puts a plain-text Markdown file at /llms.txt on your domain that tells AI models and agents what on the site is worth reading.

    What it means for you: A good llms.txt hands an AI agent the location of key answers, such as your return policy, in one request, so it doesn't have to dig through navigation, scripts and layout.

    Read more: How to write an llms.txt file · Source: llmstxt.org

  5. · Google

    Google retires the search box it showed for SearchAction markup

    Google retired the search box it used to show for sites that describe their site search with schema.org SearchAction markup. SearchAction remains a standard, machine-readable description of a site's search URL.

    What it means for you: Don't expect a Google search result feature from SearchAction any more, but it still tells AI agents how to use your site search so they don't have to discover it.

    Read more: Site search and filters AI agents can use

  6. · Google

    Google stops using rel=next and rel=prev for indexing

    Google said it no longer uses the rel="next" and rel="prev" link tags for indexing. For years they were the standard way to tell search engines that pages belonged in a series. Other search engines and tools may still read them.

    What it means for you: The tags do no harm, but don't rely on them for paginated category pages. An agent following links needs visible links to the other pages in the page itself.

    Read more: Infinite scroll, “load more” and pagination for AI agents · Source: Google Search Central

For changes to Ghost Agent Lab itself, see the changelog.