robots.txt Validator

Invalid syntax

The tool cannot process input that already contains broken syntax or malformed structure.

Fix: Check the pasted content first and correct the syntax error closest to the reported failure point.

Incomplete pasted input

Partial snippets often fail when opening and closing characters are missing.

Fix: Paste the full block or file so the tool can evaluate the complete structure.

Wrong content type

Formatting or validating the wrong format produces misleading errors.

Fix: Make sure the content matches the tool you are using before processing it.

How many URLs can robots.txt Validator check per request?

robots.txt Validator checks up to 100 URLs per batch so the request completes in under a minute and doesn't hammer third-party servers. For larger sweeps, run the tool in a loop from a script.

Does robots.txt Validator handle JavaScript-rendered pages?

robots.txt Validator fetches the raw HTML served to a crawler, which is what search engines index for the first pass. If your site relies on client-side rendering, the tool shows you exactly what Googlebot's initial render sees before it runs JavaScript.

What User-Agent does robots.txt Validator send?

robots.txt Validator sends a User-Agent string that identifies itself honestly (DevFox bot) rather than impersonating Googlebot — spoofing UAs can get a site's access flagged. If a page blocks non-browser UAs, you'll see the block clearly reflected in the output.

Can robots.txt Validator check password-protected pages?

No. robots.txt Validator only makes anonymous public requests — it can't log in or carry session cookies. For auth-protected pages, use a headless browser in your own environment.

Does robots.txt Validator respect my robots.txt?

Yes. robots.txt Validator fetches and parses robots.txt before crawling a site and skips disallowed URLs by default. You can toggle off the check if you're auditing your own site and want to see every status code.