offlinewebtools.com — free, watermark-free browser tools that stay on your device with nothing uploading to our servers
Robots.txt Generator
Generate robots.txt files to control search engine crawlers
Free to use, no watermarks, runs in your browser—nothing uploads to our servers
Select which bots these rules apply to (* = all bots)
Common patterns: / (all), /admin/ (admin folder), /*.pdf (PDF files)
Time to wait between requests
User-agent: * Disallow: /admin/
Tips:
- Place robots.txt in your website's root directory
- Use Disallow: / to block all crawlers from your entire site
- Use Allow to override more restrictive Disallow rules
- Test your robots.txt with Google Search Console
Report a bug
If this tool misbehaves, share a quick summary so we can investigate. We include the tool URL automatically—no need to copy and paste.