Hi there,
I have an issue where there are way too many guests on my forum at any given time. It's not a huge deal, it's just that they're flooding one particular subforum with thousands and thousands of threadviews.
Another, probably related issue is that my bandwidth usage has skyrocketed in the past couple months. I went for years using up less than 2gb of bandwidth per month to averaging 100gb per month over the past two(see attached pic).
I'd appreciate any ideas at all how to fix this. I've tried a few things on my own, and had my hosts give me some worthless tips, but nothing has helped.
My next idea is to install a new forum on a new addon domain and merge the two onto the new domain, but I'm nervous about taking that step and of course unsure if it will even help.
What are they viewing? Maybe they're spambots/spiders?
You could:
1) Lock down your content more
2) Put up advertisements to monetize the traffic, that's a lot of page views!
3) Switch to a host that doesn't care about bandwidth usage
4) Try to figure out which IPs are using the most and block them. They are probably bots anyway.
Thanks for the reply spork.
They're clearly bots of some sort. They view 'no permissions' pages but mostly they just 'read' old threads(see attached pic).
I'm not interested in monetizing my site and my host gives me unlimited bandwidth so that's not the main issue. The main issue is not letting bots steal my bandwidth or artificially load up threadviews.
I've blocked plenty of IPs but I've read that's probably not a good idea in the long run.
How can I lock down my content more?
It is a good idea. You should see my iptables block list on my server... It's a very manual process and takes quite a bit of time, but it pays of in the end in terms of resource utilization. Remember, this is not only affecting your bandwidth, but also your CPU, MySQL database, ram usage, etc.
For example, look up 188.165.240.167.
http://www.whatismyip.com/ip-whois-lookup/
I see that a few times in your list. I would recommend blocking
188.165.240.0/24
188.165.244.0/24
188.165.251.0/24
Unless you can come up with a reason why a dedicated server in the OVH datacenter needs to be accessing your forum? I don't think people read forums on their dedicated servers
You'll find a lot of these botting people rent servers with huge IP ranges. Some of these data centers simply don't care about abuse.
Don't these bot servers use proxys quite a bit? I'd hate to ban a wide range of IPs that come from legit servers.
Anywho, a laaaaarge majority of my problems are coming from France it seems. If I ban the country(I have no legits from there) would that cause problems with domestic IPs being routed through there or is that not common?
(2014-08-08, 02:17 PM)Mr.Magic Wrote: [ -> ]Don't these bot servers use proxys quite a bit? I'd hate to ban a wide range of IPs that come from legit servers.
Anywho, a laaaaarge majority of my problems are coming from France it seems. If I ban the country(I have no legits from there) would that cause problems with domestic IPs being routed through there or is that not common?
OVH is in France, they are a HUGE network of servers. What do you mean by "legit servers"? Why would a server need to access your site? Sites like Google, Yahoo, etc aren't going to be hosted on the OVH or some random network. You probably don't want proxies or tor nodes accessing your forums?
(2014-08-08, 02:45 PM)spork985 Wrote: [ -> ]You probably don't want proxies or tor nodes accessing your forums?
That's a good point, I hadn't thought of that. Thanks for your help spork.
I'll delete the fine nation of la France and see what happens.
(2014-08-08, 02:51 PM)Mr.Magic Wrote: [ -> ] (2014-08-08, 02:45 PM)spork985 Wrote: [ -> ]You probably don't want proxies or tor nodes accessing your forums?
That's a good point, I hadn't thought of that. Thanks for your help spork.
I'll delete the fine nation of la France and see what happens.
Don't block the entire country, you need to look up the IPs and make sure they actually servers
@Mr.Magic
Put this to your .htaccess file, then look in mySQL
sessions table -
useragent column, to find particular bot type which you want to block, then add most distinctive part of its user agent string to .htaccess block list. (there are allready added many example bots in list, just add them similar way. Remember that .htaccess file to work must be placed in your site root folder.)
But be very careful and chose really distinctive part of bot's useragent string, that for sure does not appear in useragent strings of your precious users which you dont want to block along with bots.
What it do :
It block all bot types which are on your list before they even touch any page. (its IP does not matter)
It saves your bandwidth and server resources because bots does not reach pages and get no response (204)
SetEnvIfNoCase User-Agent "Twiceler" bad_bot
SetEnvIfNoCase User-Agent "VoilaBot BETA 1.2" bad_bot
SetEnvIfNoCase User-Agent "ADmantX Platform Semantic Analyzer" bad_bot
SetEnvIfNoCase User-Agent "Sogou web spider/3.0" bad_bot
SetEnvIfNoCase User-Agent "psbot" bad_bot
SetEnvIfNoCase User-Agent "GrapeshotCrawler" bad_bot
SetEnvIfNoCase User-Agent "libwww-perl/5.805" bad_bot
SetEnvIfNoCase User-Agent "Java/1.5.0_11" bad_bot
SetEnvIfNoCase User-Agent "Exabot" bad_bot
SetEnvIfNoCase User-Agent "bot/1.0" bad_bot
SetEnvIfNoCase User-Agent "panscient.com" bad_bot
SetEnvIfNoCase User-Agent "Charlotte/1.0b" bad_bot
SetEnvIfNoCase User-Agent "boitho.com-dc" bad_bot
SetEnvIfNoCase User-Agent "ajSitemap" bad_bot
SetEnvIfNoCase User-Agent "Java/1.6.0_11" bad_bot
SetEnvIfNoCase User-Agent "WebDataCentreBot/1.0" bad_bot
SetEnvIfNoCase User-Agent "Rankivabot" bad_bot
SetEnvIfNoCase User-Agent "DBLBot/1.0" bad_bot
SetEnvIfNoCase User-Agent "Java" bad_bot
SetEnvIfNoCase User-Agent "rogerbot" bad_bot
SetEnvIfNoCase User-Agent "discobot/1.1" bad_bot
SetEnvIfNoCase user-Agent "FlashGet" bad_bot
SetEnvIfNoCase user-Agent "GetRight" bad_bot
SetEnvIfNoCase User-Agent "SapphireWebCrawler" bad_bot
SetEnvIfNoCase User-Agent "Yandex" bad_bot
SetEnvIfNoCase User-Agent "Baiduspider" bad_bot
SetEnvIfNoCase user-Agent "eCatch" bad_bot
SetEnvIfNoCase user-Agent "Offline.Explorer" bad_bot
SetEnvIfNoCase user-Agent "[Ww]eb[Bb]andit" bad_bot
SetEnvIfNoCase user-Agent "Zeus" bad_bot
SetEnvIfNoCase User-Agent "DotBot" bad_bot
SetEnvIfNoCase User-Agent "MJ12bot" bad_bot
SetEnvIfNoCase User-Agent "AhrefsBot" bad_bot
SetEnvIfNoCase user-Agent "EirGrabber" bad_bot
SetEnvIfNoCase User-Agent "AppEngine-Google" bad_bot
SetEnvIfNoCase User-Agent "DCPbot/1.2" bad_bot
SetEnvIfNoCase user-Agent "EmailSiphon" bad_bot
SetEnvIfNoCase User-Agent "sitebot" bad_bot
SetEnvIfNoCase User-Agent "LexxeBot" bad_bot
SetEnvIfNoCase User-Agent "Ezooms" bad_bot
SetEnvIfNoCase User-Agent "R6_CommentReader" bad_bot
SetEnvIfNoCase User-Agent "CrystalSemanticsBot" bad_bot
SetEnvIfNoCase user-Agent "ExtractorPro" bad_bot
SetEnvIfNoCase user-Agent "Widow" bad_bot
SetEnvIfNoCase User-Agent "ichiro" bad_bot
SetEnvIfNoCase User-Agent "AcoonBot/4.11.1" bad_bot
SetEnvIfNoCase User-Agent "Mail.RU/2.0" bad_bot
SetEnvIfNoCase User-Agent "trendictionbot0.5.0" bad_bot
RewriteCond %{ENV:bad_bot} !=""
RewriteRule .* - [L,R=204]
After implementing it -
you may test it to see if it works properly by baning self and trying to visit own site for test.
To ban self for test add "Mozilla" to bots list like this
SetEnvIfNoCase User-Agent "Mozilla" bad_bot