Free index launch kit
Give crawlers the files they need.
Enter a domain and get the complete set: robots.txt with clear AI crawler rules, a sitemap (built using discovered pages), a starter llms.txt, an IndexNow key, and direct links to every major webmaster console.
No signup required robots.txt, sitemap.xml, llms.txt AI crawler rules included
Inspecting your site
Discovering pages and generating files.
Could not build the kit
Example kit
acme.com
sitemap.xml 128 URLs
robots.txt AI rules
llms.txt Scaffold
IndexNow key Verify + submit
Files, key, and console links for one domain, generated from what the crawler can actually reach.
Your complete launch kit
sitemap.xml
llms.txt
IndexNow key
Publish the key file at your domain root, then submit. Nothing is sent until the file is confirmed live.
Host this file at
File content
Webmaster console links
AI crawler rules
Every entry below is a named crawler with a documented operator. Allow them if you want AI answers to cite your pages, block them if you would rather keep your content out of model training. You can mix the two, and the robots.txt above follows whichever you choose.
Wayback Machine
We ask the Wayback Machine to archive the homepage. It is a courtesy request, not a guarantee: the archive can be slow or rate limited, and it may queue the snapshot for later. A snapshot is still useful because it records a baseline you can point to when a page changes.
What this kit does and does not do
When you launch a new website or migrate an existing one, the problems are usually not mysteries. A site can get shipped without a sitemap, or the robots.txt blocks something nobody meant to block, or the pages are fine but there is no way the crawler can know they exist. This kit is built to help you find those gaps and hands you the exact files to close them.
What it will not do is put your site on Google or in a model's training data. Indexing is earned by a site. Search engines decide whether a page deserves a place in the results, and AI companies decide what they crawl and how they train. No file, key, or ping can change that. What these files change is whether a crawler that arrives can read the site, find the pages, and understand what they are about.
Run the kit once when you launch or migrate, then rerun it after major structural changes, like a platform move or a URL scheme change. The generated sitemap is a snapshot of what was reachable at that moment, so treat it as a starting point to compare against your own page list.
What does the Index Launch Kit generate?
A robots.txt with explicit AI crawler rules, a sitemap.xml built using discovered pages, a starter llms.txt, an IndexNow key file, and prefilled links into Google Search Console, Bing Webmaster Tools, Yandex, Naver, Baidu, and Seznam.
Is the generated sitemap complete?
Not necessarily. It contains what the crawler could reach from the sitemap and homepage. Pages behind logins, orphaned pages with no internal links, and anything a crawler was blocked from will not appear. Compare it against your own inventory before shipping.
Why do I need a separate llms.txt?
It is a proposed convention, not an official standard, and no major model is obligated to read it. The value is small but real: it gives a short, plain summary and a list of key pages for any tool that chooses to look. Treat it as a bonus, not a requirement.
Should I allow or block AI crawlers?
Allow them if you want AI answers to cite your pages, which can send referral traffic. Block them if you would rather your content stay out of model training, or if crawler load is a problem. Note that training crawlers and answer crawlers are different agents, so you can allow one and block the other.
Does submitting through IndexNow guarantee anything?
No. IndexNow tells participating engines that a URL changed. Whether, when, and how they recrawl it is their call. It is a notification, not a placement.
Why do sites have indexing issues?
Most new sites are not invisible because search engines rejected them. They are invisible because the crawl never had a clear path defined. Below are four typical failure points:
No sitemap, or a sitemap nobody can find
A sitemap is a list of the pages you want crawled. Without one, a crawler has to discover pages by following links, which means anything orphaned or buried three clicks deep can sit unseen for weeks. The other half of the problem is placement: a sitemap that exists but is not declared in robots.txt or submitted in a console is easy to miss. The kit generates the file and the declaration together so they cannot drift apart.
Very strict Robots rules
A single stray Disallow: / under the wrong user agent can shut a whole site out. Staging robots.txt files copied to production are a common cause. So is a wildcard rule written for one folder that accidentally matches everything. The kit writes explicit rules per crawler so the intent is visible line by line, and you can see exactly what is allowed before you publish.
No way to announce page changes
Even with a clean sitemap, a crawler still has to be told the site exists and that pages have changed. That is what the console links and IndexNow are for. They are notifications, not placements, but a site that never sends them is relying entirely on the crawler wandering in on its own schedule.
The page is readable to people but not to machines
Client-side rendering, blocked assets, and login walls can all leave a crawler with an empty page. This kit does not audit rendering, but if the generated sitemap comes back far smaller than your real page count, that gap is usually the first clue that the crawler is not seeing what a visitor sees.
AI crawler reference
These are the crawlers most often discussed when teams decide what to allow. Training crawlers feed model development. Answer crawlers fetch pages live to answer a question and cite a source. You can treat them differently, and the robots.txt above follows whatever you choose.
| Operator | User agents | Purpose |
|---|---|---|
| OpenAI | GPTBot, ChatGPT-User, OAI-SearchBot | Training plus assistant browsing |
| Anthropic | ClaudeBot, Claude-Web, anthropic-ai | Training plus assistant browsing |
| Google-Extended | AI training, separate from Googlebot search | |
| Perplexity | PerplexityBot | Answer engine plus citations |
| Apple | Applebot-Extended | AI training |
| Meta | FacebookBot, meta-externalagent | AI features plus training |
| Amazon | Amazonbot | AI features |
| Cohere | cohere-ai, cohere-training-data-crawler | Training |
| Common Crawl | CCBot | Open dataset used by many models |
| Mistral | MistralAI-User | Assistant browsing |
| DuckDuckGo | DuckAssistBot | AI answers |
| ByteDance | Bytespider | Training plus assistant |
| You.com | YouBot | Answer engine |
| AI2 | AI2Bot, Ai2Bot-Dolma | Training |
| Timpi | Timpibot | AI search |
Agent names and operators change. Confirm the current list before you finalize a ruleset you intend to keep.
Launch checklist
Work top to bottom. Each step depends on the one before it.
- 1
Run the kit and review discovery
Enter the domain and compare the discovered URL count against your own page list. A large gap points to a rendering or blocking problem worth fixing first.
- 2
Decide your AI crawler policy
Choose allow or block per operator using the reference table. Regenerate robots.txt to reflect your decision.
- 3
Publish robots.txt and sitemap.xml
Deploy both to the domain root. Confirm each loads at its expected URL before moving on.
- 4
Edit and publish llms.txt
Replace the scaffold prose with a real summary and your key pages. Treat it as optional polish, not a requirement.
- 5
Host the IndexNow key and verify
Place the key file at the domain root, rerun the check until it reads as verified, then submit your URLs.
- 6
Register the sitemap in each console
Open the console links and submit the sitemap where you have an account.
- 7
Rerun after major changes
Platform moves and URL scheme changes invalidate the old sitemap. Run the kit again and compare the results.