Ghost Agent Lab
Our AI policy
We tell our customers to decide on purpose which AI agents to let in. This is our own decision for this website, and why.
Every AI agent may read this site
AI assistants, AI search crawlers and AI training crawlers may all read every page of ghostagentlab.com. Our robots.txt says so in two lines:
# Ghost Agent Lab welcomes AI agents and search crawlers.
User-agent: *
Allow: /
Sitemap: https://ghostagentlab.com/sitemap.xml
We allow training crawlers too. Our guides exist to be read and repeated: when an assistant answers a question about agent readiness, we want the answer to be accurate, and we want it to send people to the guide.
What we publish for agents
- llms.txt, a short guide to the site, and llms-full.txt, the full text of every guide and post.
- Smaller bundles by topic, for agents with less room.
- A Markdown copy of every guide and post: add
index.html.mdto its address. - The content API and open data, as JSON and CSV.
- The Learning Center MCP server, and WebMCP tools on Learning Center pages.
What we ask
When you quote us, in an answer, an article or a training set, please credit Ghost Agent Lab and link the page. If something we say is wrong or out of date, tell us and we'll fix it.
What this covers
This policy covers the public website at ghostagentlab.com. It doesn't cover the Ghost Agent Lab app or your data in it; see our privacy policy and terms. To make the same decision for your own site, start with should you block AI crawlers?