Free tool
robots.txt Generator & Tester
Build a valid robots.txt with a rule builder, then test any URL against it.
Free · No sign up · Powered by Chatref
Rules
Test a URL
Blocked– Disallow: /admin
User-agent: * Disallow: /admin Disallow: /cart User-agent: GPTBot Allow: / Sitemap: https://example.com/sitemap.xml
Deciding which AI crawlers to allow? Chatref respects robots and answers only from content you approve – cited, with human handoff.
Get $50 free creditWhy Chatref
Why hand-tune crawler rules when Chatref answers customers for you?
This tool builds and tests a robots.txt so the right crawlers reach the right pages. But controlling crawlers is only half the job – your visitors still need answers. Chatref reads the site you approve and answers every question live, with the exact source quoted and a human ready to step in, 24/7.
Start free with $50 creditValid robots.txt in clicks
Test allow vs block
Choose which AI bots crawl
Answers visitors, cited
Related tools
More free tools
Is the robots.txt Generator free?
Yes. It runs in your browser, needs no sign up, and costs nothing. Build your rules and copy the robots.txt it produces.
What does robots.txt do?
It tells crawlers which paths they may or may not visit. You set a user-agent, an Allow or Disallow, and a path. This tool also lets you test a URL to see which rule wins.
How does the URL tester decide allowed or blocked?
It follows the standard robots rules: the longest matching path wins, and Allow beats Disallow on a tie. It also handles the * and $ wildcards.
Can I control which AI crawlers reach my site?
Yes. Add rules for bots like GPTBot to allow or block them. Chatref respects robots and only answers from content you approve, with citations and human handoff.
Put an AI agent on your site today.
Add your content, drop in one snippet, and let Chatref answer the first questions - free to start.
Free to start · No card needed · No code