MyBB Community Forums

I have a big problem...I had 100,000 web crawler hits in 8 hours from msn, google and yahoo! My host suspended my account!

How can I block some of these hits?

Need help fast

You can reduce the amount for these in the control panel

e.g. Google Webmaster Tools

Site Configuration > Settings > Crawl rate

(2011-08-19, 03:03 AM)Malcolm. Wrote: [ -> ]You can reduce the amount for these in the control panel

e.g. Google Webmaser Tools

Site Configuration > Settings > Crawl rate

Hmmm I cant find the setting?

Is it in my admin panel?

(2011-08-19, 03:03 AM)Malcolm. Wrote: [ -> ]You can reduce the amount for these in the control panel

e.g. Google Webmaster Tools

Site Configuration > Settings > Crawl rate

That may help with google, but I need a way to limit all bots from excessive crawling! They are killing me!

http://www.mcanerin.com/en/search-engine/robots-txt.asp

If you are unsure what a robots.txt is have a look at this link.

http://www.robotstxt.org/robotstxt.html