Robots.txt & Sitemap Generator
How to use this robots.txt & sitemap generator
- Choose "Robots.txt" or "Sitemap.xml" mode.
- Fill in your site details or list of URLs.
- Click "Download File" and upload it to your site's root directory.
What do these files do?
robots.txt tells search engine crawlers which parts of your site they may or may not access. sitemap.xml lists your important pages so search engines can discover and index them more efficiently.
Where do I upload these files?
Both files need to sit at your site's root, like example.com/robots.txt and example.com/sitemap.xml, for search engines to find them automatically.
Does blocking a path in robots.txt remove it from search results?
Not necessarily — robots.txt only requests that crawlers not access a path. To reliably remove pages from search results, use a noindex meta tag instead.
robots.txt is a request, not a lock
robots.txt works entirely on voluntary compliance — it's a plain text file that well-behaved crawlers (Google, Bing, and most legitimate search engines) choose to read and respect before crawling your site, but there's nothing in the file format itself that actually prevents access. Malicious scrapers, spam bots, and crawlers that simply don't bother checking the file can and do ignore it completely, and the file itself is publicly readable by anyone, which means listing a sensitive path in Disallow can actually advertise its existence to someone looking for it, rather than hiding it. For anything that genuinely needs to stay inaccessible — an admin panel, private data — the correct tool is server-side authentication or access control, not robots.txt.
The robots.txt and noindex conflict that trips up a lot of sites
One of the most common and counterintuitive SEO mistakes is blocking a page in robots.txt while also expecting a noindex meta tag on that same page to keep it out of search results. These two mechanisms don't cooperate the way most people assume: if robots.txt disallows a path, crawlers generally won't fetch that page's content at all, which means they never see the noindex tag sitting in its HTML head. The counterintuitive result is that a disallowed page can still appear in search results, usually as a bare URL with no title or snippet, if enough other pages link to it — because Google learned the URL exists through those links, even though it was never allowed to crawl and read the actual page. If your real goal is keeping a specific page out of search results entirely, use noindex without disallowing the path in robots.txt, so crawlers can actually read that instruction.
Why a sitemap doesn't guarantee indexing
Submitting a sitemap.xml tells search engines "here are pages I'd like you to know about," but it's a discovery aid, not a command that forces indexing. Search engines still independently decide whether each URL is worth crawling and indexing based on their own quality signals — content uniqueness, perceived value, site-wide trust — and a sitemap listing a low-quality or duplicate page doesn't override that evaluation. What a sitemap reliably does is speed up discovery, especially for large sites or pages that aren't well-linked internally, by giving crawlers a direct list instead of relying entirely on following links to stumble onto every page.
Sitemap size limits and when you need more than one file
A single sitemap.xml file is capped at 50,000 URLs and 50MB uncompressed by the sitemap protocol specification — a limit that matters for larger sites but rarely comes up for smaller ones. Sites that exceed either limit split their URLs across multiple sitemap files and reference all of them from a single "sitemap index" file, which search engines read to discover the full set. For a site well under this limit — the vast majority of sites this generator is built for — a single flat sitemap listing every important URL is the simpler, entirely sufficient approach, and splitting isn't necessary until a site's scale genuinely requires it.
Limitations of this tool
This generator produces a standard flat sitemap.xml listing whatever URLs you provide, and a basic robots.txt with the disallow rules and crawler-access setting you configure — it doesn't automatically discover your site's URLs by crawling it, doesn't set lastmod, changefreq, or priority values (which most major search engines treat as hints they mostly ignore anyway), and doesn't generate the extended sitemap formats used for images, videos, or news content, which follow a different schema. It also doesn't account for crawler-specific directives some robots.txt implementations support inconsistently, like Crawl-delay, which Google's crawler ignores entirely despite some other crawlers respecting it.