Even when clearing it via this method
https://community.mybb.com/thread-225918.html
Within seconds there are thousands again....If left for long periods like several hours it builds up into the 10's of thousands....
How do I block these?
Those are bots for sure, you can ban country or IP ranges using .htaccess.
An easier method is using Cloudflare and putting your site at "I am under attack mode". You can also use cloudflare to ban the spam IP ranges manually.
(2020-04-17, 08:37 AM)DrXotick Wrote: [ -> ]Even when clearing it via this method https://community.mybb.com/thread-225918.html
Within seconds there are thousands again....If left for long periods like several hours it builds up into the 10's of thousands....
How do I block these?
Cloudflare is definitely an option if they're all from a specific country or IP range, otherwise you're going to have to present each user with a JS challenge via the 'Under Attack Mode' option.
Thank you for the suggestions.... blocking countries won't work, it's a very global forum
I'm trying to understand is how It went from a couple thousand over 24 hours (for the last several years) to within a day now going to 90,000 per 24h. How does something like that happen??? Some targeted attack, is it possible???
It's the only thing I can think of, other than updating from 1.8.19 to 1.8.22. what do you think?
Also, there is a 5MB file of known bots that's regularly maintained....would that help? I mean that's huge, how would it even be parsed when people attempt to access the site... Would it slow things down significantly?
I can't imagine it has to do with the update, and the likelihood that it's a targeted attack is also pretty low. My best guess is that you were indexed by a new search engine, or someones crawler discovered your site and you're now being overrun with spam from their bots.
As long as the bot users aren't registering, creating posts, or using the contact form you don't have much to worry about in terms of site performance. Although, if you want to protect yourself I'd recommend using Cloudflare, simply for caching and as a CDN to help mitigate any negative impacts that sort of traffic might have.
Lastly, make sure you have StopForumSpam enabled and configured, this will help keep bots from being able to register and send contact requests which would be the the main source of issue should it occur.
I'd like to clarify that they do use resources, however users creating posts, creating accounts, and similar actions use far more resources.
Thanks for your reply. Just pretty amazing it went from 3000 to 90,000 in single day...
these bots don't use resources of the site? I thought they crawled and as they crawled the pages load for them, no?
They definitely do affect server load. I've had as much as 700 online at once. Nowhere near 90K, and 700 alone was enough to use up a core.
They don't generally follow images and so forth, so that reduces the load as opposed to an actual user browsing. Servers can handle a surprisingly large amount of traffic with relatively modest hardware nowadays, but 90K bots is... wow. I'd be curious as to who your host is. I'm impressed.
Sounds like the suggestions above for Cloudflare may not be a bad idea.
(2020-04-17, 03:22 PM)Darth Apple Wrote: [ -> ]They definitely do affect server load. I've had as much as 700 online at once. Nowhere near 90K, and 700 alone was enough to use up a core.
They don't generally follow images and so forth, so that reduces the load as opposed to an actual user browsing. Servers can handle a surprisingly large amount of traffic with relatively modest hardware nowadays, but 90K bots is... wow. I'd be curious as to who your host is. I'm impressed.
Sounds like the suggestions above for Cloudflare may not be a bad idea.
Well they suspended me. Namecheap is the host. On shared hosting...
If I purge them from users online there are thousands back within a minute. The 90k is over a full day.
I really hate that cloudflare check when landing on a site...trying to avoid that.
I have some suggestions ... See if any sound possible
- what about this 5MB bad bot IP list. Would it do any good?
- I remember also some way to keep bots from accessing certain links, I think it was in htaccess, to stop them from accessing reply, birthdays, etc... There was a set thing to copy and paste for MyBB specifically.
- there's no way to disallow * (bot) unless IP of whitelisted bot? Seems there should be some way to do this in htaccess. I understand some bots can mimic real users but I think this would help if it's possible ....
- I've also seen Honeypot link (or disallowed link)...is there some plugin that could be made that would: if * accesses disallowed link or Honeypot link then block access to site. Sounds simple, maybe could add white list to it as well .... Is that possible?
A site of this size should not be run on a shared hosting, ie a shared hosting in the world. You will not bear this volume of visits
Better get a private server
vps or vds
With good specifications
See the issue with using htaccess is that, although it helps, it still puts the burden of the processing on the host. The servers itself still has to process it. As said above, VPS might be a good idea, but managing them requires some technical knowledge.
How long has this been going on?