The google bot seems to be spiking up the amount of processes running on the site. Specifically, the showthread.php. This has become troublesome to the point that my members and me are unable to access the site without 500 Internal Server Errors popping up.
Is there a way to fix this?
I use Google SEO btw. And their robots.txt
You can reduce Google's crawl rate for your site in the Google Webmaster Tools. However by default Google bot does not cause noticable load to a site. If it keeps up maybe it's someone else using Googlebot's user agent string (so it's not Google at all but MyBB thinks it is).
Thats possible. Its never caused much problem before until now.
Any idea how to resolve this?
You can use the Crawl-delay parameter in robots.txt.
User-agent: *
Crawl-delay: 10
You mean this?
That seems to have fixed the problem. Thank you very much Malcom and frostschutz.
You're welcome.
Glad I could help.
Go to Google webmaster tools and try "Fetch as Google bot" for one of your pages and see how Google see and crawls your pages then you can be sure.
(2011-11-11, 10:59 PM)voctan Wrote: [ -> ]Go to Google webmaster tools and try "Fetch as Google bot" for one of your pages and see how Google see and crawls your pages then you can be sure.
Best solution, any webmaster must have a Google Webmaster tools account, that allows you to do ALLLL the stuff, and allows them to know who to contact if there's an issue...