Skip to content

robots.txt checker

Read allow and disallow rules, sitemap links, and the mistakes that make a robots.txt file lie.

Fetches a public URL
Use via MCP

Use from your agent

Streamable HTTP at https://benchline.fabwaseem.com/mcp. No key. The tool name is check_robots_txt.

mcp.jsonjson
1{
2 "mcpServers": {
3 "benchline": {
4 "url": "https://benchline.fabwaseem.com/mcp"
5 }
6 }
7}
tools/calljson
1{
2 "jsonrpc": "2.0",
3 "id": 1,
4 "method": "tools/call",
5 "params": {
6 "name": "check_robots_txt",
7 "arguments": {
8 "url": "https://example.com/file"
9 }
10 }
11}

Related tools

How it works

Fetch a public robots.txt, list Allow and Disallow rules and sitemap links, and flag common mistakes. Private addresses are blocked.

  1. 01

    Enter a domain or a robots.txt URL

    A bare domain is fetched at /robots.txt. A URL that already ends in robots.txt is fetched as written.

  2. 02

    Read the groups

    Each User-agent lists Allow and Disallow paths. Sitemap lines are listed separately. You can also paste a file and audit it locally.

  3. 03

    Fix the flagged rows

    Start with fails: a file that is HTML, a User-agent * that disallows the whole site, and a Sitemap that is not an absolute URL.

FAQ