Robots.txt Generator
| Imposing Restrictions You may impose restrictions on which web pages to disallow indexing. By default, most users will want to allow all directories except their /cgi-bin/ directory, which commonly holds scripts, and their images directory /images/. To enable all web pages, select Yes to "Enable All Webpages," then enter each web page or directory path in the exclusion box, one per line. Example: "http://www.yourdomain.com/cgi-bin/" (Excludes the /cgi-bin/ directory) Example: "http://www.yourdomain.com/images/" (Excludes the /images/ directory) Example: "http://www.yourdomain.com/welcome.html" (Excludes the /welcome.html web page)
| |||||||||||||||||||
