MyBB Community Forums

Full Version: Google Bot Rumped Us No KY
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
One of my forums makes about 200-ish+ posts a day recently.
Well Google Bot decided to crawl absolutely everything in more intensity than it ever has before, more than likely because of the new content being posted...
It actually ended up getting the hosting account suspended.
I got everything fixed concerning the host, I've denied the spider bot user group from all access and redid the robots.txt to apply to all and added a crawl delay of 30.
Google Webmaster Tools is not letting me set crawl frequency for the domain stating that the crawl setting is "special" and it won't let me adjust it.

I don't want to completely prevent Google from crawling the site...
but their bot just went ape *&^% crazy crawling us ever since we've started getting more content posted.

So... considering my f'd up situation...
what would you do?
Have you had to deal with this before?
If so how to you best control it?
thank you.
Your problem has nothing to do with the google bot.
How many hits did you get? Cause even if you ramp the Google Crawl Rate up to maximum (which you should not ever do - best to let Google figure it out on its own), it's still only 2 requests per second. If you are seeing more, unless you're working with multiple sub domains, it's probably not Google. Check the IP.

Also you should be able to set a limit, if you're the verified owner of the site (entire domain). It may be different if you only get a sub domain.