2013-03-15, 01:24 AM
One of my forums makes about 200-ish+ posts a day recently.
Well Google Bot decided to crawl absolutely everything in more intensity than it ever has before, more than likely because of the new content being posted...
It actually ended up getting the hosting account suspended.
I got everything fixed concerning the host, I've denied the spider bot user group from all access and redid the robots.txt to apply to all and added a crawl delay of 30.
Google Webmaster Tools is not letting me set crawl frequency for the domain stating that the crawl setting is "special" and it won't let me adjust it.
I don't want to completely prevent Google from crawling the site...
but their bot just went ape *&^% crazy crawling us ever since we've started getting more content posted.
So... considering my f'd up situation...
what would you do?
Have you had to deal with this before?
If so how to you best control it?
thank you.
Well Google Bot decided to crawl absolutely everything in more intensity than it ever has before, more than likely because of the new content being posted...
It actually ended up getting the hosting account suspended.
I got everything fixed concerning the host, I've denied the spider bot user group from all access and redid the robots.txt to apply to all and added a crawl delay of 30.
Google Webmaster Tools is not letting me set crawl frequency for the domain stating that the crawl setting is "special" and it won't let me adjust it.
I don't want to completely prevent Google from crawling the site...
but their bot just went ape *&^% crazy crawling us ever since we've started getting more content posted.
So... considering my f'd up situation...
what would you do?
Have you had to deal with this before?
If so how to you best control it?
thank you.