Robots.txt Generator and Checker
Create a robots.txt for Shopify, WooCommerce, WordPress or any site, or check an existing file. Every rule is explained, and the tool warns you before you block pages that matter.
Create a robots.txt
Check an existing robots.txt
What robots.txt does, and doesn't do
robots.txt is a plain text file at the root of your site (yourstore.com/robots.txt) that tells crawlers which URLs they may crawl. It's mainly used to keep crawlers away from pages with no value in search, such as carts and internal search results.
It controls crawling, not indexing. Google says a page blocked by robots.txt can still appear in search results, without a description, if other pages link to it. To keep a page out of Google, use a noindex tag and leave the page crawlable so Google can see it (Google: introduction to robots.txt, Google: block indexing).
Rules are requests. Well-behaved crawlers follow them, but robots.txt doesn't protect private content. Use passwords for that.
The directives, explained
- User-agent: *
- Starts a group of rules.
*means any crawler; a name such asGooglebottargets one. A crawler follows only the most specific group that matches it. - Disallow: /cart
- Asks crawlers not to crawl URLs starting with /cart. An empty
Disallow:blocks nothing. - Allow: /cart/help
- Lets crawlers crawl a URL inside a disallowed path. When rules conflict, Google uses the longest matching rule.
- * and $
*matches any characters;$marks the end of the URL.Disallow: /*.pdf$blocks URLs ending in .pdf.- Sitemap: https://…
- Points crawlers to your XML sitemap. It needs a full URL.
- Crawl-delay
- Google ignores it. Some other search engines use it.
Details are in How Google interprets the robots.txt specification, including the 500 KiB size limit and how server errors are treated.
Shopify stores
Shopify creates a robots.txt for every store, already blocking the cart, checkout, account pages and internal search. Most stores don't need to change it. To edit it, you add a robots.txt.liquid theme template. The generator creates one that keeps Shopify's default rules, which Shopify recommends, and adds yours.
Shopify warns that the template replaces its generated file, so a mistake can cost you traffic. Save a copy of your current yourstore.com/robots.txt first. To go back to the default, delete the template. See Shopify: editing robots.txt.liquid and the Shopify Help Center.
To hide a store that isn't ready, use Shopify's online store password instead of blocking crawlers.
WooCommerce and WordPress
WordPress serves a basic robots.txt if you don't upload one. A typical WooCommerce file made with this tool looks like this:
# robots.txt made with the Ecom Outpost Robots.txt Generator. Check it before uploading.
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*add-to-cart=*
Sitemap: https://example.com/wp-sitemap.xml
Don't block /wp-content/ or /wp-includes/. They hold the CSS, JavaScript and images Google needs to render your pages. Many SEO plugins let you edit robots.txt from the dashboard; otherwise upload the file to your site's root folder.
AI crawlers
AI companies publish separate user agents for training and for search, so you can choose:
- Training: GPTBot (OpenAI), Google-Extended (Google; Google says it doesn't affect inclusion or ranking in Google Search), ClaudeBot (Anthropic), Applebot-Extended (Apple) and CCBot (Common Crawl).
- Search: OAI-SearchBot (ChatGPT search), Claude-SearchBot and PerplexityBot (Perplexity). Blocking these can keep your store out of their answers, so most stores leave them allowed.
- Fetchers that act on a user's request, such as ChatGPT-User and Perplexity-User, may not follow robots.txt, according to their owners.
Limits and privacy
- The generator and the pasted-file checker run in your browser. Nothing you enter is sent to Ecom Outpost.
- Fetch robots.txt sends the site address to our server, which requests only that site's /robots.txt (user agent
EcomOutpostSEOChecker). We don't save the address; a copy of the file is kept for up to 10 minutes. - The URL tester follows Google's matching rules. Other crawlers may interpret edge cases differently.
- This tool never changes your website.
More free SEO tools
SEO Page Checker
Check one page for common technical SEO issues: indexing, title, description, headings, images, links and structured data.
Meta Title and Description Generator
Get title and description ideas built from your own page details, with length checks and a search result preview.
Schema Markup Generator
Create JSON-LD structured data for products, your business, articles, breadcrumbs and local stores, with checks as you type.
Product Description SEO Helper
Check a product description for readability, natural keyword use and unsupported claims, and get title, outline and alt text ideas.
Ecommerce Keyword Brainstormer
Brainstorm keyword ideas for a product or category, grouped by intent: research, comparison, buying, how-to and questions.
XML Sitemap Checker
Check that your XML sitemap loads and is valid, and test a sample of its URLs for errors, redirects, noindex and robots.txt blocks.
PageSpeed Checker
Test your store's speed with Google's free PageSpeed Insights, then get the results explained in plain English with next steps.
Guides for store owners
Articles from the Outpost Journal on getting found, product pages and marketing for independent stores.
