Can someone please tell me how to stop spiders from crawling my site. It seems that they are always there and it is very annoying. I searched the forums here but can't seem to find a clear answer. Is there something I can add to the .htaccess file?
Thanks for any help on this.
Shedrock
Why would you want to stop spiders crawling your site?
I don't need it. The domain name has been around for quite a number of years. I only recently converted it to MyBB and I am not interested in spiders crawling the site at the moment.
How to stop them? add a robots.txt file with their user agent (or user-agent: *) and add lines to disallow parts of your site...
Example:
User-Agent: *
Disallow: /
Would block all robots (that obey the file) from visiting any of your site.
If you do this, your site will not appear in search engines!
(2010-04-14, 10:10 PM)AJS Wrote: [ -> ]I don't get it.
There is not really to get AJS. I just don't need the spiders crawling my site. It is just a matter of choice at the moment.
Anyway, thanks for the help guys. Much appreciated.
Shedrock
(2010-04-14, 10:08 PM)MattRogowski Wrote: [ -> ]robots.txt??
When a spider wants to crawl your site, it looks for a file called robots.txt in your domain's root directory (
http://mysite.com/robots.txt ). This file tells the spiders which URLs to index on your site, and where they are disallowed.
Additionally, you may want to create a special group within MyBB for bots and take away all permissions. Then go through the ACP and assign all spiders / bots to that new group. Instead of seeing real information, the bots will get a "no permission" page.
I knew what it is, I was responding at the same time as ladyunicornejg, answering with a question
