iLikeTool
HomeAll ToolsCategoriesDeveloper ToolsAboutContact
Code / Developer tools

Robots.txt Generator

Generate and configure robots.txt directives to guide search engine crawlers and web bots.

terminal Crawler Protocol Standard
verified RFC 9309 Compliant
smart_toy

Crawler Directives

Step 1 of 2
Yandex/Bing
EXAMPLE
folder_off

Restricted Directories

Paths blocked from web crawler indexing

Disallow & Allow
/
Configured Access Rules
disallow /admin/
disallow /private/
disallow /api/v1/internal/
disallow /tmp/
allow /admin/public-docs/
info

The Allow: directive takes precedence over Disallow: when granting access to specific subfolders within a blocked directory.

code

Generated Preview

check_circle Valid Robots Syntax
12345678
User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/v1/internal/
Disallow: /tmp/
Allow: /admin/public-docs/

Sitemap: https://example.com/sitemap.xml
Directive Summary UTF-8 Plain Text
4 Disallowed
1 Allowed
164 B Est. Size

How to Use Robots.txt

Follow these steps to deploy crawler directives to your domain accurately.

1

Specify User-Agent Rules

Designate which search engine spiders follow the policy. Use wildcard * to govern all crawlers, or specify individual bots like Googlebot.

2

Define Restricted Paths

Block internal endpoints, staging environments, sensitive admin directories, or search result queries with explicit Disallow: lines.

3

Upload to Root Directory

Save the generated output as robots.txt and place it strictly in your domain's web root: https://yourdomain.com/robots.txt.

Related Developer Utilities

Complementary SEO and code structure utilities

View all Dev Tools arrow_forward
code_blocks

Meta Tag Generator

Construct OpenGraph, Twitter card, and SEO meta tags effortlessly.
link

URL Encoder / Decoder

Safely escape and encode URL query strings and URI components.
format_align_left

HTML Formatter

Beautify and indent unstructured markup or minify for production.
data_object

JSON Validator

Validate payloads, inspect schema mismatches, and fix syntax errors.
HOW TO USE

Prepare crawler rules

Choose a user agent and allowed or disallowed paths. Add a sitemap URL, then copy or download the file.

Example

Disallow: /private/ asks matching crawlers not to crawl that path.

Before you use the result

robots.txt is not access control. Private pages still need authentication, and blocked pages can sometimes appear in search.