Robots.txt File
Quick answer: A robots.txt file gives participating crawlers instructions about which URL paths they may request. It is public and does not secure pages or reliably keep them out of search results.
What is robots.txt?
A robots.txt file is a plain-text file usually available at the root of a host, such as https://example.com/robots.txt. It groups crawl rules by crawler name. Crawlers may choose whether to follow these instructions, and rule support can differ.
How to create a robots.txt file
Create a plain-text file named robots.txt and place it at the root of the hostname the rules should cover. For example, rules at example.com/robots.txt do not automatically apply to a different subdomain.
User-agent: *
Disallow: /private/
Sitemap: https://example.com/sitemap.xmlThis example asks crawlers that follow the rule to avoid paths under /private/ and advertises the sitemap location. Replace the example hostname and path with your own. A blank Disallow: means that no paths are disallowed for that user-agent group.
What robots.txt can and cannot do
- Manage crawler requests: Use rules to ask compliant crawlers to avoid selected paths when crawling those URLs is unnecessary.
- It is not privacy protection: The file is public, and robots.txt cannot enforce rules on crawlers that ignore it. Protect private content with authentication or access controls.
- It is not a dependable no-index rule: A disallowed URL may still be indexed if discovered elsewhere. For a page that should be excluded from Google Search, allow crawling so Google can see a
noindexdirective, or restrict access when it must be private. - Check resource rules carefully: Blocking CSS or JavaScript that a crawler needs to understand a page can affect how it interprets that page.
Robots.txt examples
Ask crawlers to avoid a private-looking path (this does not make the path private):
User-agent: *
Disallow: /private/Allow all paths for crawlers that honor this syntax:
User-agent: *
Disallow:Apply a rule only to a named crawler group:
User-agent: ExampleBot
Disallow: /Frequently asked questions
Does robots.txt block a page from Google Search?
No. A crawl block does not guarantee that a URL stays out of search results. To prevent indexing, use an accessible noindex directive for crawlers that support it, or require authentication for private content.
Can robots.txt protect a folder or file?
No. It gives public crawler guidance, not access control. Use server-side authentication or permissions to protect files.
Read Google's current robots.txt guide and noindex documentation for crawler-specific behavior and indexing instructions.