2012-03-17, 06:10 PM
Sorry if this is a bit long, but...
My forum has recently been experiencing some downtime and general slowness due to what often seems to be excessive hits by spiders and/or spambots. I recently e-mailed my host (DreamHost) to ask about this, and they claimed that my user is often hitting the shared hosting limit, and essentially, my bandwidth is being throttled. Below is the reply I received when asking about this, with my user of course replaced. I'd use a spoiler tag, but I realised they can't be used here...
It is mentioned that the problem may be caused by my other site, but the only other site assigned to that user is a placeholder and probably gets next to no traffic. It's also mentioned that my server is functioning well currently. That is true. Although the claim that it hasn't been down in "some time" certainly isn't—it was only the other day when my site kept getting those 500 errors. This is an intermittent problem and could happen at any time, unexpectedly. The problem is that I can't identify what's causing this.
My database is thirty megabytes. A lot of this is taken up by the logs AJAX Chat records. I wouldn't think this is a problem, since the logs are only accessed by me. I haven't heard of any reports of AJAX Chat being bloated either. I am unsure of whether any outside sources try to access the chat room, but at most there's only about ten members in the chat room, and often much less than that.
This might seem like a lot, but my forum is currently using 46 plugins. I could probably disable some of these. Many of them are not accessed very often—for instance, plugins for the ACP or moderation plugins, and one or two MyCode plugins. Am I really going to have to disable quite a lot of my forum's functionality to speed things up? I'd say the most intensive plugins would probably be Game Section, Google SEO, My Achievements, MyDownloads, NewPoints, ProfileComments and ProStats.
I have recently disabled the "Add Spiders" plugin which will hopefully help. Is there any way at all to tell which plugins are the culprits if this is the case? Because a lot of members use many of these plugins, and a problem I find with a lot of them is if you disable them, everything goes with them. And if the plugin requires a lot of configuration it will mean a lot of work after reactivating over a dozen plugins.
If it's a spider/spam problem, I think the best way to fix that is blocking the IPs, and the guy who responded has been very helpful with this. I'll go over all of the links he provided. I'm just asking for any suggestions if you have some. Funny thing is, my forum only has a fairly small amount of active posters and a little over 10,000 posts. I've known somebody with many more and also on DreamHost shared hosting.
I'll let you guys know if I find the main cause of this. I have a feeling I may be blamed for the amount of plugins I've used, but honestly, I see at least 80% of them as being pretty low usage. Heck, no way I'm switching to VPS. I shell out a lot of money already for something I get no money in return for.
Thank you.
My forum has recently been experiencing some downtime and general slowness due to what often seems to be excessive hits by spiders and/or spambots. I recently e-mailed my host (DreamHost) to ask about this, and they claimed that my user is often hitting the shared hosting limit, and essentially, my bandwidth is being throttled. Below is the reply I received when asking about this, with my user of course replaced. I'd use a spoiler tag, but I realised they can't be used here...
Quote:I'm sorry that you are encountering these issues. It would appear that
the sites that are running under your user "MYUSERNAME" are occasionally
hitting our shared server's memory limits. This is why your site is
seemingly slow, and the 500 errors are directly related as well. Here is
an excerpt from the log kept by our Process Watcher, the daemon that is
killing your troublesome php5.cgi processes:
Mon Mar 12 17:38:44 2012 procwatch3 INFO: PID 757 (php5.cgi)
MYUSERNAME:pg2173876 : reniced to +10
Procwatch is a daemon that runs constantly on shared servers to monitor
the usage of RAM/CPU and execution time so that no single user can use an
inappropriately high percentage of the shared resources and impact the
overall health of the server or the server's ability to serve all users'
pages.
Now, it's important to get these errors fixed to keep your sites running
with optimum speed, as well as keep our servers happy. Unfortunately it
can often be a bit of a trial-and-error procedure, but I will provide as
much information as I can to aide you in this process.
Firstly, it should be noted that any sites running under your user can
contribute to the problems you are encountering. For example, if site A
is running processes that are taking up 85% of your individual limit, and
site B tries to run a concurrent process that takes up 20% of the memory
cap, site B will have it's process killed since it is the one that took
you above your limits, even though site A's script is the obvious memory
hog... For this reason, it's important to keep all of your user's sites
optimized and running smoothly.
There are quite a few things that can contribute to high memory
consumption on any given site, but I will mention the most common
causes...
Plugins - Especially third-party plugins can often be poorly written and
run up the memory consumption. I often recommend disabling all
non-critical plugins and see if the problem gets better, then enabling
them one-by-one until you are able to identify one that causes problems.
Database related overhead, or an unnecessary amount of database queries -
If you are trying to query your database for thousands of results at a
time, this can also cause issues... Try to keep all database sizes and
queries to a minimum by deleting irrelevant or old information from your
database. You might also try optimizing tables in your databases by
visiting your database through PHPMyAdmin. Simply use the "Check tables
having overhead" link located directly underneath your tables and then
"With Selected:", choose "Optimize Table". This might reduce overhead in
your database, which is basically a lot of empty and redundant space that
can slow your queries down.
It is also possible that a spike in traffic, or a heavy run by a site
indexer (such as the GoogleBot), or even abusive hits by a given IP
address, could cause this on a short term basis, so you might want to
inspect your access logs for such activity (you can often manage this
problem by implementing a robots.txt or by blocking abusive IP addresses
with .htaccess).
You might as well want to try looking over the suggestions offered at our
wiki pages on this matter:
http://wiki.dreamhost.com/Poor_performance
http://wiki.dreamhost.com/Optimize_database
http://wiki.dreamhost.com/WordPress_Optimization
http://wiki.dreamhost.com/Database_Maintenance
http://wiki.dreamhost.com/Slow_site_troubleshooting
http://wiki.dreamhost.com/Finding_Causes_of_Heavy_Usage
Implementing the above suggestions more often than not will fix the
ProcWatch errors, but if you are still finding that your sites encounter
the same problems, you might want to consider moving to a VPS
(http://wiki.dreamhost.com/DreamHost_PS) or dedicated server where you
can reserve sufficient RAM for your own processes to run!
I am sorry if this news is discouraging, but I felt it was important that
you have the full benefit of knowing what is occurring here so you can
make the appropriate decision to meet your needs for this site.
I have looked over your server and it is functioning well, load is low
and hasn't been taken down in some time. I am unable to find a source of
slowness besides procwatch possibly affecting you. If you could also
provide a traceroute for me I can take a look at possible connectivity
issues getting to your site.
http://wiki.dreamhost.com/Traceroute
It is mentioned that the problem may be caused by my other site, but the only other site assigned to that user is a placeholder and probably gets next to no traffic. It's also mentioned that my server is functioning well currently. That is true. Although the claim that it hasn't been down in "some time" certainly isn't—it was only the other day when my site kept getting those 500 errors. This is an intermittent problem and could happen at any time, unexpectedly. The problem is that I can't identify what's causing this.
My database is thirty megabytes. A lot of this is taken up by the logs AJAX Chat records. I wouldn't think this is a problem, since the logs are only accessed by me. I haven't heard of any reports of AJAX Chat being bloated either. I am unsure of whether any outside sources try to access the chat room, but at most there's only about ten members in the chat room, and often much less than that.
This might seem like a lot, but my forum is currently using 46 plugins. I could probably disable some of these. Many of them are not accessed very often—for instance, plugins for the ACP or moderation plugins, and one or two MyCode plugins. Am I really going to have to disable quite a lot of my forum's functionality to speed things up? I'd say the most intensive plugins would probably be Game Section, Google SEO, My Achievements, MyDownloads, NewPoints, ProfileComments and ProStats.
I have recently disabled the "Add Spiders" plugin which will hopefully help. Is there any way at all to tell which plugins are the culprits if this is the case? Because a lot of members use many of these plugins, and a problem I find with a lot of them is if you disable them, everything goes with them. And if the plugin requires a lot of configuration it will mean a lot of work after reactivating over a dozen plugins.
If it's a spider/spam problem, I think the best way to fix that is blocking the IPs, and the guy who responded has been very helpful with this. I'll go over all of the links he provided. I'm just asking for any suggestions if you have some. Funny thing is, my forum only has a fairly small amount of active posters and a little over 10,000 posts. I've known somebody with many more and also on DreamHost shared hosting.
I'll let you guys know if I find the main cause of this. I have a feeling I may be blamed for the amount of plugins I've used, but honestly, I see at least 80% of them as being pretty low usage. Heck, no way I'm switching to VPS. I shell out a lot of money already for something I get no money in return for.
Thank you.