Robots.txt generator

Client

Compose crawl rules for one User-agent block—preview structure on robots & sitemap preview.

About Robots.txt generator

Build robots.txt with presets, Allow/Disallow paths, crawl-delay, and Sitemap lines—copy or download locally. The interactive transform on this page runs in your browser tab—Toolcore does not need your paste for the core operation described above.

How to use this page

Paste or type in the main workspace, run the primary action from the toolbar, then copy or download the result. Use Load example when the page offers it, or URL prefill (?q= / ?qb=) so agents and tickets open the same input.

Compose a starter robots.txt, then paste the output into the robots & sitemap preview to check groups before deploy.

Rules

?

One User-agent block is generated per run. Paths are one per line. An empty Disallow value allows all URLs for that agent.

Nearby workflows on Toolcore

  • Robots.txt & sitemap preview — Paste robots. when headers, identifiers, or reference tables need a sibling check.
  • Sitemap.xml generator — Build sitemap urlset XML from a URL list—optional lastmod, changefreq, priority—copy or download in the browser. when headers, identifiers, or reference tables need a sibling check.
  • Open Graph preview — Paste HTML head or og: meta lines—see title, description, and image URL fields. when headers, identifiers, or reference tables need a sibling check.
  • URL parser — Split URLs into protocol, host, path, query, and hash—browser URL API; optional https:// assumption; no fetch. when headers, identifiers, or reference tables need a sibling check.

Common use cases

  • Draft robots.txt for a new site before the first deploy.
  • Block staging paths (/admin, /private) while allowing the rest of the site.
  • Add Sitemap: lines next to Allow/Disallow groups for a quick SEO handoff.

Common mistakes to avoid

  • Expecting this to validate production traffic

    Crawlers interpret rules differently—always preview with the robots parser and test in Search Console.

  • Blocking CSS or JS by mistake

    Over-broad Disallow patterns can hurt rendering in search—keep paths specific.

FAQ

How is this different from the robots preview tool?

This page generates text from your inputs; the preview tool parses an existing robots.txt or sitemap XML.

Multiple User-agent groups?

This generator emits one group per run—duplicate the output and edit lines for additional agents if needed.

More tools

Related utilities you can open in another tab—mostly client-side.