Ecom Outpost — Discover more storesJournalFor merchants

Robots.txt Generator and Checker

Create a robots.txt for Shopify, WooCommerce, WordPress or any site, or check an existing file. Every rule is explained, and the tool warns you before you block pages that matter.

FreeNo signupWorks on mobile

Create a robots.txt

Options

Check an existing robots.txt

We fetch only /robots.txt from that site, up to 10 checks a minute.

What robots.txt does, and doesn't do

robots.txt is a plain text file at the root of your site (yourstore.com/robots.txt) that tells crawlers which URLs they may crawl. It's mainly used to keep crawlers away from pages with no value in search, such as carts and internal search results.

It controls crawling, not indexing. Google says a page blocked by robots.txt can still appear in search results, without a description, if other pages link to it. To keep a page out of Google, use a noindex tag and leave the page crawlable so Google can see it (Google: introduction to robots.txt, Google: block indexing).

Rules are requests. Well-behaved crawlers follow them, but robots.txt doesn't protect private content. Use passwords for that.

The directives, explained

User-agent: *
Starts a group of rules. * means any crawler; a name such as Googlebot targets one. A crawler follows only the most specific group that matches it.
Disallow: /cart
Asks crawlers not to crawl URLs starting with /cart. An empty Disallow: blocks nothing.
Allow: /cart/help
Lets crawlers crawl a URL inside a disallowed path. When rules conflict, Google uses the longest matching rule.
* and $
* matches any characters; $ marks the end of the URL. Disallow: /*.pdf$ blocks URLs ending in .pdf.
Sitemap: https://…
Points crawlers to your XML sitemap. It needs a full URL.
Crawl-delay
Google ignores it. Some other search engines use it.

Details are in How Google interprets the robots.txt specification, including the 500 KiB size limit and how server errors are treated.

Shopify stores

Shopify creates a robots.txt for every store, already blocking the cart, checkout, account pages and internal search. Most stores don't need to change it. To edit it, you add a robots.txt.liquid theme template. The generator creates one that keeps Shopify's default rules, which Shopify recommends, and adds yours.

Shopify warns that the template replaces its generated file, so a mistake can cost you traffic. Save a copy of your current yourstore.com/robots.txt first. To go back to the default, delete the template. See Shopify: editing robots.txt.liquid and the Shopify Help Center.

To hide a store that isn't ready, use Shopify's online store password instead of blocking crawlers.

WooCommerce and WordPress

WordPress serves a basic robots.txt if you don't upload one. A typical WooCommerce file made with this tool looks like this:

# robots.txt made with the Ecom Outpost Robots.txt Generator. Check it before uploading.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*add-to-cart=*

Sitemap: https://example.com/wp-sitemap.xml

Don't block /wp-content/ or /wp-includes/. They hold the CSS, JavaScript and images Google needs to render your pages. Many SEO plugins let you edit robots.txt from the dashboard; otherwise upload the file to your site's root folder.

AI crawlers

AI companies publish separate user agents for training and for search, so you can choose:

  • Training: GPTBot (OpenAI), Google-Extended (Google; Google says it doesn't affect inclusion or ranking in Google Search), ClaudeBot (Anthropic), Applebot-Extended (Apple) and CCBot (Common Crawl).
  • Search: OAI-SearchBot (ChatGPT search), Claude-SearchBot and PerplexityBot (Perplexity). Blocking these can keep your store out of their answers, so most stores leave them allowed.
  • Fetchers that act on a user's request, such as ChatGPT-User and Perplexity-User, may not follow robots.txt, according to their owners.

Limits and privacy

  • The generator and the pasted-file checker run in your browser. Nothing you enter is sent to Ecom Outpost.
  • Fetch robots.txt sends the site address to our server, which requests only that site's /robots.txt (user agent EcomOutpostSEOChecker). We don't save the address; a copy of the file is kept for up to 10 minutes.
  • The URL tester follows Google's matching rules. Other crawlers may interpret edge cases differently.
  • This tool never changes your website.
  • SEO Page Checker

    Check one page for common technical SEO issues: indexing, title, description, headings, images, links and structured data.

  • Meta Title and Description Generator

    Get title and description ideas built from your own page details, with length checks and a search result preview.

  • Schema Markup Generator

    Create JSON-LD structured data for products, your business, articles, breadcrumbs and local stores, with checks as you type.

  • Product Description SEO Helper

    Check a product description for readability, natural keyword use and unsupported claims, and get title, outline and alt text ideas.

  • Ecommerce Keyword Brainstormer

    Brainstorm keyword ideas for a product or category, grouped by intent: research, comparison, buying, how-to and questions.

  • XML Sitemap Checker

    Check that your XML sitemap loads and is valid, and test a sample of its URLs for errors, redirects, noindex and robots.txt blocks.

  • PageSpeed Checker

    Test your store's speed with Google's free PageSpeed Insights, then get the results explained in plain English with next steps.

  • Guides for store owners

    Articles from the Outpost Journal on getting found, product pages and marketing for independent stores.

See all free SEO tools