In reference to the Cloudflare Check you mentioned; using "High" security mode on Cloudflare will generally present bots with a challenge, but not genuine users. Furthermore, even if you choose to use a normal security level Cloudflare will significantly reduce server load by caching pages and assets.
Whether or not you choose to use CF for security I would still recommend it for DDoS protection, and caching which will most likely produce a significant change in server load.
As far as using .Htaccess; as mentioned previously that still produces server load, and a rather significant amount of it if you plan to use a 3MB file for blocking traffic. Disallowing bots via htaccess is a good idea in theory, however it doesn't actually restrict their access, it's just a request not to index or crawl your site. Bots do not have to honor this request, and it is unlikely they will in your scenario.
The idea of a honeypot could also work, however it would likely end up blocking all search engines crawlers as they index your site as it cannot tell the difference between a legitimate crawler, and what you're experiencing.
(2020-04-17, 04:21 PM)@abdurhman@ Wrote: [ -> ]A site of this size should not be run on a shared hosting, ie a shared hosting in the world. You will not bear this volume of visits
Better get a private server
vps or vds
With good specifications
Well I only have a handful of legit users a day. All of this is bots.
(2020-04-17, 04:33 PM)Darth Apple Wrote: [ -> ]See the issue with using htaccess is that, although it helps, it still puts the burden of the processing on the host. The servers itself still has to process it. As said above, VPS might be a good idea, but managing them requires some technical knowledge.
How long has this been going on?
So it's been 2000-5000 over the last several years. And it's just within last couple days it's shot to 90k (over 24 hours)... You mean server has to process to see whether or not IP is blocked?
Also, Hmm. I thought when htaccess blocked it would be "impossible" to access...if just in robots.txt they can do what they want, but to bam block in htaccess is very different.
(2020-04-17, 05:41 PM)R34P3R Wrote: [ -> ]In reference to the Cloudflare Check you mentioned; using "High" security mode on Cloudflare will generally present bots with a challenge, but not genuine users. Furthermore, even if you choose to use a normal security level Cloudflare will significantly reduce server load by caching pages and assets.
Whether or not you choose to use CF for security I would still recommend it for DDoS protection, and caching which will most likely produce a significant change in server load.
As far as using .Htaccess; as mentioned previously that still produces server load, and a rather significant amount of it if you plan to use a 3MB file for blocking traffic. Disallowing bots via htaccess is a good idea in theory, however it doesn't actually restrict their access, it's just a request not to index or crawl your site. Bots do not have to honor this request, and it is unlikely they will in your scenario.
The idea of a honeypot could also would, however it would likely end up blocking all search engines crawlers as they index your site as it cannot tell the difference between a legitimate crawler, and what you're experiencing.
Okay, so maybe will give it a go. I thought it challenged everyone...
For the plug-in, I'd like to see it ignore white listed bots. And generally those bots would not access what you didn't tell them to access anyways...
So the logic would be something like
if honeypot or disallowed accessed
and if is bot
and if is not on whitelist
then ban IP
To me this sounds totally doable. Obviously some will get through but the idea is try to catch a majority of them or large %.
What you mentioned in terms of the honeypot is possible, but like you said it will only decrease the likelihood that a bot makes it through. Another thing to consider is that there are dozens of crawlers for major search engines, and some of them change IP's relatively frequently so it is possible you'd also be blocking your site from being indexed.