How to create a robots.txt file
- Choose whether robots are allowed everywhere by default (almost always the right choice).
- Pick a preset or list the folders you want to disallow, such as admin areas or internal search results.
- Add your sitemap URL so search engines can find all your pages.
- Optionally block specific bots — for example AI training crawlers or SEO tool crawlers.
- Download the file, name it exactly
robots.txt, and upload it to the root of your domain so it is reachable athttps://yourdomain.com/robots.txt.
How robots.txt works
robots.txt is a plain text file that tells well-behaved crawlers which parts of a site they may request. It follows the Robots Exclusion Protocol, standardised as RFC 9309 in 2022.
| Directive | Meaning |
|---|---|
User-agent: * | The rules below apply to all crawlers |
Disallow: /admin/ | Do not crawl anything under /admin/ |
Allow: /admin/admin-ajax.php | Exception to a disallow rule |
Sitemap: https://…/sitemap.xml | Location of your XML sitemap |
Crawl-delay: 10 | Seconds between requests — respected by Bing and Yandex, ignored by Google |
Important: robots.txt is not a security or de-indexing tool
- It does not hide pages. The file is public, and blocked URLs can still appear in Google results (without a description) if other sites link to them. To keep a page out of search results, allow crawling and add a
noindexrobots meta tag instead. - It does not protect private data. Malicious bots ignore robots.txt. Use passwords and server permissions for anything private.
- Don’t block CSS and JavaScript that pages need — Google renders pages like a browser and needs those files to understand your layout.
Blocking AI crawlers
Several AI companies publish crawler names that respect robots.txt, including OpenAI’s GPTBot, Anthropic’s ClaudeBot, Common Crawl’s CCBot and Google’s Google-Extended token (which controls use for Gemini training but does not affect Google Search). Blocking them is a business choice — it may reduce the chance of your content appearing in AI assistants’ answers.
After uploading, check the file in Google Search Console’s robots.txt report. And make sure your sitemap exists — create one with the XML Sitemap Generator.
Frequently asked questions
Where do I put robots.txt?
In the root folder of your website (usually public_html), so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own file.
Will Disallow remove a page from Google?
No. It stops crawling, but the URL can still be indexed from links. Use a noindex meta tag (and allow crawling) to remove a page from results.
Do I need a robots.txt file?
It is not required, but it is good practice — especially to point crawlers to your sitemap.
Does Google respect Crawl-delay?
No. Google ignores Crawl-delay; Bing and Yandex respect it.