Where does robots.txt go?
The file belongs at the root of the relevant host, such as https://example.com/robots.txt.
Free SEO Tool
Free Robots.txt Generator
Create a robots.txt file online with crawl rules and an XML sitemap reference using this free robots.txt generator for websites and WordPress.
Use the Robots.txt Generator
Create a basic robots.txt file that communicates crawler access rules for your website. Select common paths to disallow, include your sitemap URL, then copy or download the file.
Enter your website URL. The sitemap URL will be generated automatically.
Leave empty to use /sitemap.xml.
Crawl Rules
What this free tool helps you do
- ✓Create standard user-agent and crawl directives
- ✓Add common WordPress and site-search exclusions
- ✓Include your XML sitemap location
- ✓Copy or download a ready-to-review robots.txt file
How to use it
- 1.Enter your website and optional sitemap URL.
- 2.Select only the paths crawlers should not request.
- 3.Generate and carefully review the directives.
- 4.Upload robots.txt to the root of your host.
Frequently asked questions
Can robots.txt remove a page from Google?
Robots.txt controls crawling, not guaranteed indexing. Use appropriate noindex controls when a crawlable page should not appear in search.
Should CSS and JavaScript be blocked?
Usually not. Search engines may need page resources to render and understand content correctly.