Agent directory
The AI agents that visit websites
Who runs each AI crawler, assistant and agent, what it wants from your site, whether to let it in, and how to tell the real one from an impostor. Generated from the registry Ghost Agent Lab uses to identify agent traffic.
By company
Every agent each AI company runs, with a robots.txt you can copy to allow or block them together.
AI assistants
Fetches pages for a person, in real time, while they use an AI assistant. Almost always allow it.
OpenAIChatGPT-UserFetches a page when someone using ChatGPT, or a custom GPT, asks about it or clicks through to it.
AnthropicClaude-UserFetches a page when someone using Claude asks about it or shares a link.
PerplexityPerplexity-UserFetches a page when someone using Perplexity asks a question that needs it.
MetaMeta-ExternalFetcherFetches a page when someone using Meta AI, or another Meta product, asks for it, so it acts for a person in real time.
DuckDuckGoDuckAssistBotFetches pages in real time so DuckDuckGo's DuckAssist can answer questions and cite sources.
Mistral AIMistralAI-UserFetches a page when someone using Mistral's Le Chat asks for it.
AmazonAmzn-UserFetches pages to answer questions people ask Amazon's assistants, such as Alexa.
ByteDanceDoubaoBotFetches pages for Doubao, ByteDance's AI assistant.
GoogleGemini-Deep-ResearchFetches pages while Gemini's Deep Research builds a report for a user.
GoogleGoogle-NotebookLMFetches a page when someone adds it as a source in Google's NotebookLM.
GoogleGoogleAgent-URLContextFetches a page when a developer using the Gemini API asks Gemini to read a link.
Kagikagi-fetcherFetches pages to answer questions in Kagi's AI search features.
Moonshot AIKimi-UserFetches a page for someone using Kimi, Moonshot AI's assistant.
PhindPhindBotFetches pages for Phind, an AI search engine for developers.
AI and search crawlers
Indexes pages so a search engine or AI search tool can find, cite and link to them. Allow it, unless you deliberately want to stay out of that search engine.
OpenAIOAI-SearchBotCrawls the web so ChatGPT can show and link to sites in its search results.
AnthropicClaude-SearchBotCrawls the web to improve the search results Claude gives its users.
PerplexityPerplexityBotCrawls the web so Perplexity can find, cite and link to sites in its answers.
GoogleGooglebotGoogle's main crawler for Google Search.
MicrosoftBingbotMicrosoft's crawler for Bing.
AppleApplebotApple's crawler for Siri, Spotlight and Safari suggestions.
AmazonAmazonbotAmazon's crawler, used to improve Amazon's products and services, including answers from Alexa.
GoogleGoogle-InspectionToolUsed by Google Search Console's URL Inspection and the Rich Results Test, when someone tests a page.
GoogleStorebot-GoogleCrawls product pages, carts and checkouts for Google Shopping.
DuckDuckGoDuckDuckBotDuckDuckGo's crawler for its search engine.
YandexYandexBotYandex's crawler for its search engine.
BaiduBaiduspiderBaidu's crawler for its search engine.
ExaExaSearchBotExa's crawler for its search engine, which AI apps and agents use to find pages.
Mistral AIMistralAI-IndexMistral AI's crawler for indexing pages its products can search.
AI training crawlers
Collects public pages that may be used to train AI models. A real business choice.
OpenAIGPTBotCollects public web pages that may be used to train OpenAI's models.
AnthropicClaudeBotAnthropic's crawler for collecting public web content that may be used to train its models.
GoogleGoogle-ExtendedNot a crawler: a robots.txt name Google reads to decide whether content Googlebot collects may be used to train and ground Gemini models.
AppleApplebot-ExtendedNot a crawler: a robots.txt name Apple reads to decide whether content Applebot collects may be used to train Apple's generative AI models.
MetaMeta-ExternalAgentMeta's crawler for collecting web content that may be used to train its AI models and improve its products.
Common CrawlCCBotCommon Crawl's crawler.
ByteDanceBytespiderByteDance's crawler, widely reported to collect data for its AI models.
GoogleGoogleOtherA general-purpose Google crawler used by Google's product and research teams, separate from Google Search.
Ai2AI2BotThe Allen Institute for AI's crawler, which collects data for open research models and datasets.
Anthropicanthropic-aiAn older robots.txt name linked to Anthropic's crawling.
Coherecohere-training-data-crawlerCohere's crawler for collecting data that may be used to train its models.
DeepSeekDeepSeekBotA crawler run by DeepSeek, the company behind the DeepSeek AI models.
DiffbotDiffbotDiffbot's crawler, which turns web pages into structured data for its Knowledge Graph and AI products.
BaiduERNIEBotBaidu's crawler, linked to its ERNIE AI models.
GoogleGoogle-CloudVertexBotCrawls a site when its owner asks Google Cloud to, for example to build an AI agent on Vertex AI from the site's content.
Moonshot AIKimiBotMoonshot AI's crawler, which may collect data for training its Kimi models.
Mistral AIMistralAI-TrainingMistral AI's crawler for collecting data that may be used to train its models.
AlibabaQwenBotAlibaba's crawler, linked to its Qwen AI models.
Autonomous agents
Uses a website like a person would, clicking, typing and filling in forms to finish a task for someone. Make your journeys work for it.
AmazonAmazonBuyForMeAmazon's Buy for Me agent, which can buy products from brands' own websites for Amazon shoppers.
OpenAIChatGPT AgentOpenAI's ChatGPT agent: a browser ChatGPT uses to carry out tasks for a user, such as researching, filling in forms or shopping.
AnthropicClaude-CodeAnthropic's coding agent.
AnthropicClaude-WebAn older name linked to Anthropic.
CognitionDevinDevin, Cognition's AI software engineer, browsing the web while it works on a task.
GoogleGoogle-Gemini-CLIGoogle's Gemini CLI, a coding agent in the terminal, fetching pages for a developer.
GoogleGoogleAgent-MarinerGoogle's Project Mariner, a browser agent that carries out tasks on websites for a user.
Moonshot AIKimi-AgentKimi's browsing agent, which visits sites while it carries out research and tasks for a user.
ManusManus-UserThe Manus agent, which uses a browser to carry out tasks for its users.
AmazonNovaActAmazon Nova Act, which developers use to build agents that carry out tasks in a web browser.
Other bots
A well-known bot that isn't an AI agent, such as an ads checker or a link preview. Usually allow it: these bots support products you already use..
298 more AI agents and crawlers
The rest of the AI agents in our registry, from open lists we import and keep up to date. Search by name or company.
| Agent | Run by | Type |
|---|---|---|
| 360Spider | Unknown | AI and search crawler |
| 7Siters | Unknown | AI and search crawler |
| A6-Indexer | Unknown | AI and search crawler |
| Aboundex | Unknown | AI and search crawler |
| acoonbot | Unknown | AI and search crawler |
| AddSearchBot | Unknown | AI and search crawler |
| Adidxbot | Unknown | AI and search crawler |
| AgentDataBot | Unknown | AI training crawler |
| AgentTimes | The Agent Times | AI training crawler |
| AI2Bot-DeepResearchEval | Ai2 | AI assistant |
| Ai2Bot-Dolma | Ai2 | AI training crawler |
| aiHitBot | aiHit | AI training crawler |
| AISearchBot | Unknown | AI training crawler |
| AIWebIndex | Lyrenth | AI and search crawler |
| AlexandriaOrgBot | Unknown | AI and search crawler |
| Algolia Crawler | Unknown | AI and search crawler |
| amazon-kendra | Amazon | AI and search crawler |
| amazon-QBusiness | Unknown | AI assistant |
| AmazonProductDiscovery | Unknown | AI and search crawler |
| AmazonSellerInitiatedListing | Unknown | AI and search crawler |
| Amzn-SearchBot | Unknown | AI and search crawler |
| Andibot | Andi | AI assistant |
| Anomura | Unknown | AI training crawler |
| ApifyBot | Unknown | AI training crawler |
| ApifyWebsiteContentCrawler | Unknown | AI training crawler |
| APIs-Google | Unknown | AI and search crawler |
| AppEngine-Google | Unknown | AI and search crawler |
| arabot | Unknown | AI and search crawler |
| Aranet-SearchBot | Unknown | Autonomous agent |
| AspiegelBot | Unknown | AI and search crawler |
| atlassian-bot | Atlassian | AI assistant |
| Awario | Awario | AI training crawler |
| AzureAI-SearchBot | Unknown | AI and search crawler |
| bedrockbot | Amazon | AI training crawler |
| BIGLOTRON | Unknown | AI and search crawler |
| bigsur.ai | Big Sur AI | AI assistant |
| BixelBot | Unknown | AI training crawler |
| blekkobot | Unknown | AI and search crawler |
| Bot.AraTurka.com | Unknown | AI and search crawler |
| br-crawler | Unknown | AI and search crawler |
| Bravebot | Unknown | AI training crawler |
| Brightbot | Unknown | AI training crawler |
| Brightbot 1.0 | Unknown | AI training crawler |
| BuddyBot | BuddyBotLearning | AI training crawler |
| Channel3Bot | Unknown | AI and search crawler |
| ChatGLM-Spider | Unknown | AI training crawler |
| Cliqzbot | Unknown | AI and search crawler |
| Cloudflare-AutoRAG | Cloudflare | AI and search crawler |
| CloudflareBrowserRenderingCrawler | Cloudflare | AI training crawler |
| CloudVertexBot | Unknown | AI training crawler |
| coccoc | Unknown | AI and search crawler |
| cohere-ai | Cohere | AI assistant |
| convera | Unknown | AI and search crawler |
| Cotoyogi | ROIS | AI training crawler |
| Coveobot | Unknown | AI and search crawler |
| CragCrawler | CragSoftware | AI training crawler |
| Crawl4AI | Unknown | Autonomous agent |
| Crawlspace | Crawlspace | AI training crawler |
| CyberFindCrawler | Unknown | AI and search crawler |
| Datenbank Crawler | Datenbank | AI training crawler |
| Daum | Unknown | AI and search crawler |
| daumoa | Unknown | AI and search crawler |
| DeuSu | Unknown | AI and search crawler |
| developers.google.com/+/web/snippet | Unknown | AI and search crawler |
| Diffbot-User | Diffbot | AI assistant |
| discobot | Unknown | AI and search crawler |
| DuckDuckGo-Favicons-Bot | Unknown | AI and search crawler |
| Echobot Bot | Echobox | AI training crawler |
| EchoboxBot | Echobox | AI training crawler |
| edisterbot | Unknown | AI and search crawler |
| ExaBot | Unknown | AI training crawler |
| ExteContextCrawl | Unknown | AI training crawler |
| FacebookBot | Meta | AI training crawler |
| Factset_spyderbot | Factset | AI training crawler |
| FAST Enterprise Crawler | Unknown | AI and search crawler |
| FAST-WebCrawler | Unknown | AI and search crawler |
| FediIndex | Unknown | AI and search crawler |
| FemtosearchBot | Unknown | AI and search crawler |
| FindFiles.net | Unknown | AI and search crawler |
| findthatfile | Unknown | AI and search crawler |
| findxbot | Unknown | AI and search crawler |
| fiperbot | Unknown | AI and search crawler |
| FirecrawlAgent | Firecrawl | AI training crawler |
| Flamingo_SearchEngine | Unknown | AI and search crawler |
| FleebsBot | Unknown | AI and search crawler |
| fluffy | Unknown | AI and search crawler |
| Flyriverbot | Unknown | AI training crawler |
| fr-crawler | Unknown | AI and search crawler |
| Freespoke | Unknown | AI and search crawler |
| FriendlyCrawler | Unknown | AI training crawler |
| Funnelback | Unknown | AI and search crawler |
| FyndSearchEngine-Crawler | Unknown | AI and search crawler |
| FyndSearchEngine-ReCrawler | Unknown | AI and search crawler |
| Gabanzabot | Unknown | AI and search crawler |
| GeedoBot | Unknown | AI and search crawler |
| GeedoProductSearch | Unknown | AI and search crawler |
| GeedoShopProductFinder | Unknown | AI and search crawler |
| GeistHaus-PageFetcher | GeistHaus | AI assistant |
| Gigablast | Unknown | AI and search crawler |
| Gigabot | Unknown | AI and search crawler |
| GlobalWebSearch | Unknown | AI and search crawler |
| Gluten Free Crawler | Unknown | AI and search crawler |
| Golfe | Unknown | AI and search crawler |
| Google Favicon | Unknown | AI and search crawler |
| Google-Agent | Unknown | Autonomous agent |
| Google-Certificates-Bridge | Unknown | AI and search crawler |
| Google-Firebase | AI training crawler | |
| Google-PhysicalWeb | Unknown | AI and search crawler |
| Google-Read-Aloud | Unknown | AI and search crawler |
| Google-Safety | Unknown | AI and search crawler |
| Google-Site-Verification | Unknown | AI and search crawler |
| Google-Structured-Data-Testing-Tool | Unknown | AI and search crawler |
| google-xrawler | Unknown | AI and search crawler |
| Googlebot-IA | Unknown | AI and search crawler |
| Googlebot-Image | Unknown | AI and search crawler |
| Googlebot-Mobile | Unknown | AI and search crawler |
| Googlebot-News | Unknown | AI and search crawler |
| Googlebot-Video | Unknown | AI and search crawler |
| GoogleOther-Image | AI training crawler | |
| GoogleOther-Video | AI training crawler | |
| GoogleProducer | Unknown | AI and search crawler |
| Greppr Web Crawler | Unknown | AI and search crawler |
| grub.org | Unknown | AI and search crawler |
| gsa-crawler | Unknown | AI and search crawler |
| HenkBot | Unknown | AI training crawler |
| iAskBot | Unknown | Autonomous agent |
| iaskspider | Unknown | Autonomous agent |
| IbouBot | Unknown | AI and search crawler |
| ICC-Crawler | NICT | AI training crawler |
| ichiro | Unknown | AI and search crawler |
| ImageMind | Unknown | AI training crawler |
| ImagesiftBot | ImageSift | AI training crawler |
| imageSpider | Unknown | AI training crawler |
| img2dataset | img2dataset | AI training crawler |
| IndeedBot | Unknown | AI and search crawler |
| IndeedJobBot | Unknown | AI and search crawler |
| INETDEX-BOT | Unknown | AI and search crawler |
| intelx.io_bot | Unknown | AI and search crawler |
| IsonSearchBot | Unknown | AI and search crawler |
| ISSCyberRiskCrawler | ISS-Corporate | AI training crawler |
| IstellaBot | Unknown | AI and search crawler |
| JobboerseBot | Unknown | AI and search crawler |
| jobswithgptcom-bot | Unknown | AI and search crawler |
| Jooblebot | Unknown | AI and search crawler |
| jyxobot | Unknown | AI and search crawler |
| Kagibot | Unknown | AI and search crawler |
| Kangaroo Bot | Unknown | AI training crawler |
| KeenableBot | Keenable | AI and search crawler |
| KendraBot | Unknown | AI training crawler |
| Kimi-SearchBot | Moonshot AI | AI and search crawler |
| KlaviyoAIBot | Klaviyo | AI assistant |
| KunatoCrawler | Unknown | Autonomous agent |
| laion-huggingface-processor | Unknown | AI training crawler |
| LAIONDownloader | Unknown | AI and search crawler |
| lb-spider | Unknown | AI and search crawler |
| Level9SearchBot | Unknown | AI and search crawler |
| Lightpanda | Unknown | AI training crawler |
| LinerBot | Unknown | AI assistant |
| Linguee Bot | Linguee | AI training crawler |
| linkReader | Unknown | AI training crawler |
| LinkupBot | Unknown | AI and search crawler |
| lyonl-asset-proxy | Unknown | AI and search crawler |
| lyonl-crawler | Unknown | AI and search crawler |
| MagiBot | Unknown | AI and search crawler |
| Mail.RU_Bot | Unknown | AI and search crawler |
| MatchorySearch | Unknown | AI and search crawler |
| meta-webindexer | Meta | AI assistant |
| MojeekBot | Unknown | AI and search crawler |
| Mozilla-Tabstack | Mozilla | AI training crawler |
| msnbot | Unknown | AI and search crawler |
| msrbot | Unknown | AI and search crawler |
| MyCentralAIScraperBot | Unknown | AI training crawler |
| NagetBot | Naget Inc | AI training crawler |
| NapBot | Unknown | AI and search crawler |
| NaverBot | Unknown | AI and search crawler |
| Neevabot | Unknown | AI and search crawler |
| NestDaddybot | Unknown | AI and search crawler |
| netEstate Imprint Crawler | netEstate | AI training crawler |
| newsai | Unknown | AI training crawler |
| Nicecrawler | Unknown | AI and search crawler |
| NotebookLM | Unknown | AI assistant |
| Novellum | Unknown | AI training crawler |
| NTENTbot | Unknown | AI and search crawler |
| Nutch | Unknown | AI and search crawler |
| OAI-AdsBot | OpenAI | AI training crawler |
| officestorebot | Unknown | AI and search crawler |
| OICrawler | Unknown | AI and search crawler |
| omgili | Webz.io | AI training crawler |
| omgilibot | Webz.io | AI training crawler |
| opencode | Unknown | Autonomous agent |
| OpenHoseBot | Unknown | AI and search crawler |
| openindexspider | Unknown | AI and search crawler |
| OpenTheBoxBot | Unknown | AI and search crawler |
| OrangeBot | Unknown | AI and search crawler |
| Owler (ows.eu | Unknown | AI and search crawler |
| PanguBot | the Chinese company Huawei | AI training crawler |
| Panscient | Panscient | AI training crawler |
| panscient.com | Panscient | AI training crawler |
| PerplexityUser | Unknown | AI training crawler |
| PetalBot | Huawei | AI assistant |
| PhxBot | Unknown | AI and search crawler |
| PlayStore-Google | Unknown | AI and search crawler |
| Poggio-Citations | Poggio | AI assistant |
| Poseidon Research Crawler | Poseidon Research | AI and search crawler |
| psbot | Unknown | AI and search crawler |
| purebot | Unknown | AI and search crawler |
| qodercli | Qoder | Autonomous agent |
| QualifiedBot | Qualified | AI assistant |
| Querit-SearchBot | Querit | AI training crawler |
| QueritBot | Querit | AI training crawler |
| QuillBot | Quillbot | AI training crawler |
| quillbot.com | Quillbot | AI training crawler |
| Qwantbot | Unknown | AI and search crawler |
| Qwantify | Unknown | AI and search crawler |
| Reflectionbot | Reflection | Autonomous agent |
| SafeSearch microdata crawler | Unknown | AI and search crawler |
| SBIntuitionsBot | SB Intuitions | AI training crawler |
| Scrapy | Zyte | AI training crawler |
| scribdbot | Unknown | AI and search crawler |
| Scrubby | Unknown | AI and search crawler |
| search.marginalia.nu | Unknown | AI and search crawler |
| Searcherweb | Unknown | AI and search crawler |
| Searcherxweb | Unknown | AI and search crawler |
| SearchExpress | Unknown | AI and search crawler |
| Seekbot | Unknown | AI and search crawler |
| Seekport | Unknown | AI and search crawler |
| semantic-visions | Unknown | AI training crawler |
| SemrushBot-OCOB | Semrush | AI training crawler |
| SemrushBot-SWA | Semrush | AI assistant |
| SeznamBot | Unknown | AI and search crawler |
| SeznamHomepageCrawler | Unknown | AI and search crawler |
| Shap-User | Unknown | AI assistant |
| ShapBot | Parallel | AI training crawler |
| Sidetrade indexer bot | Sidetrade | AI training crawler |
| SiteSearch360 | Unknown | AI and search crawler |
| Slurp | Unknown | AI and search crawler |
| SMTnetPMBot | Unknown | AI and search crawler |
| Sogou | Unknown | AI and search crawler |
| Sosospider | Unknown | AI and search crawler |
| Spawning-AI | Unknown | AI training crawler |
| speedy | Unknown | AI and search crawler |
| Spider; +https://www.spider.com | Unknown | AI training crawler |
| StartpagePrivateImageProxy | Unknown | AI and search crawler |
| stepstoneCrawlBot | Unknown | AI and search crawler |
| StractBot | Unknown | AI and search crawler |
| Swiftbot | Unknown | AI and search crawler |
| Swisscows | Unknown | AI and search crawler |
| tagoobot | Unknown | AI and search crawler |
| TaraGroup Intelligent Bot | Unknown | AI training crawler |
| TavilyBot | Unknown | AI training crawler |
| Teoma | Unknown | AI and search crawler |
| Terra Cotta | Unknown | AI training crawler |
| TerraCotta | Ceramic AI | AI training crawler |
| The Knowledge AI | Unknown | AI training crawler |
| TheInternetSearchx | Unknown | AI and search crawler |
| Thinkbot | Thinkbot | AI training crawler |
| TikTokSpider | ByteDance | AI training crawler |
| Timpibot | Timpi | AI training crawler |
| TongyiBot | Alibaba | AI assistant |
| top100.rambler.ru crawler | Unknown | AI and search crawler |
| ToutiaoSpider | Unknown | AI and search crawler |
| trovitBot | Unknown | AI and search crawler |
| TSM-turingos | Unknown | AI training crawler |
| TwinAgent | Twin | Autonomous agent |
| UsineNouvelleCrawler | Unknown | AI and search crawler |
| VelenPublicWebCrawler | Velen Crawler | AI training crawler |
| voilabot | Unknown | AI and search crawler |
| VsuSearchSpider | Unknown | AI and search crawler |
| WadooBot | Unknown | AI and search crawler |
| WARDBot | WEBSPARK | AI training crawler |
| wbsearchbot | Unknown | AI and search crawler |
| webzio-extended | Unknown | AI training crawler |
| WepchSearchEngine | Unknown | AI and search crawler |
| WeSEE:Search | Unknown | AI and search crawler |
| WikiDo | Unknown | AI and search crawler |
| wocbot | Unknown | AI and search crawler |
| WordupInfoSearch | Unknown | AI and search crawler |
| woriobot | Unknown | AI and search crawler |
| wotbox | Unknown | AI and search crawler |
| wpbot | QuantumCloud | AI training crawler |
| WRTNBot | Unknown | Autonomous agent |
| Y!J | Unknown | AI and search crawler |
| yacybot | Unknown | AI and search crawler |
| yandex.com/bots | Unknown | AI and search crawler |
| YandexAdditional | Yandex | AI training crawler |
| YandexAdditionalBot | Yandex | AI training crawler |
| YandexRenderResourcesBot | Unknown | AI and search crawler |
| yanga | Unknown | AI and search crawler |
| Yeti | Unknown | AI and search crawler |
| YisouSpider | Unknown | AI and search crawler |
| YiyanBot | Baidu | AI assistant |
| YouBot | You | AI and search crawler |
| YoudaoBot | Unknown | AI and search crawler |
| ZanistaBot | Unknown | AI and search crawler |
| Zealbot | Unknown | AI and search crawler |
| ZoomInfo | Unknown | AI and search crawler |
| ZumBot | Unknown | AI and search crawler |
| ZyBorg | Unknown | AI and search crawler |
No agents match. Try a shorter search.
Which of these agents visit your site?
Ghost Agent Lab reads your CDN or server logs and shows every agent, what it read, and whether it was really who it said it was.