Bots/Web Scrappers in 2025
#51
(2025-11-16, 09:08 PM)bv64 Wrote: Billy_Bob, it would be more helpful to answer questions and read posts instead of engaging in pointless discussions.

Every supporter is free to decide whom they want to help and whom they don't! When a free supporter expresses an opinion, it's their personal opinion and not that of the entire community! This also applies to the countless former team members.
When a currently active team member has something to say, it carries a completely different weight!

If I don't want to help someone because they have bad manners or have become aggressive, I block them, and that's the end of it for me.
Amen!

So, the question again: How did you block the user agent?

And for someone who thinks they have such a massive problem, you check in here shockingly infrequently and ignore most of the answers you've received in the meantime. So, the problem doesn't seem to be particularly important to you, does it?

I did it in both the .htaccess file and the robots.txt
   Robots.txt
Quote:user-agent: *
disallow: /

User-agent: SemrushBot
Disallow: /

   .htacess
Quote:<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /

# 3.1 Block empty or obviously scripted UAs
RewriteCond %{HTTP_USER_AGENT} ^\s*$ [OR]
RewriteCond %{HTTP_USER_AGENT} (curl|wget|python-requests|java|okhttp|libwww|httpclient) [NC]
RewriteRule ^ - [F,L]

# 3.2 Your “AI/SEO” bot list -> 403 (can move these to CF WAF as a Challenge instead)
RewriteCond %{HTTP_USER_AGENT} (AdsBot-Google|Amazonbot|anthropic-ai|Applebot|Awario|Bytespider|CCBot|ChatGPT|Claude|cohere-ai|DataForSeoBot|Diffbot|facebookexternalhit|Google-Extended|GPTBot|ImagesiftBot|magpie|omgili|Omgilibot|peer39|PerplexityBot|YouBot|AhrefsBot|Barkrowler|SemrushBot|DotBot) [NC]
RewriteRule ^ - [F,L]

# 3.3 Overly long querystrings (cheap SQL drains)
RewriteCond %{THE_REQUEST} \?
RewriteCond %{QUERY_STRING} .{1024,}
RewriteRule ^ - [F,L]

# 3.4 Disallow uncommon/abused methods
RewriteCond %{REQUEST_METHOD} !^(GET|POST|HEAD)$
RewriteRule ^ - [F,L]
</IfModule>

# 3.5 Deny access to obvious garbage
<FilesMatch "(\.env|composer\.(json|lock)|package\.json|yarn\.lock|phpunit|\.git|\.hg|\.svn)">
  Require all denied
</FilesMatch>

<Files 403.shtml>
order allow,deny
allow from all
</Files>

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} "=Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)"
RewriteRule ^ - [F]

RewriteCond %{HTTP_REFERER} !^$
RewriteCond %{HTTP_REFERER} !^http://patriotaction.us/.*$      [NC]
RewriteCond %{HTTP_REFERER} !^http://patriotaction.us$      [NC]
RewriteCond %{HTTP_REFERER} !^http://www.patriotaction.us/.*$      [NC]
RewriteCond %{HTTP_REFERER} !^http://www.patriotaction.us$      [NC]
RewriteCond %{HTTP_REFERER} !^https://patriotaction.us/.*$      [NC]
RewriteCond %{HTTP_REFERER} !^https://patriotaction.us$      [NC]
RewriteCond %{HTTP_REFERER} !^https://www.patriotaction.us/.*$      [NC]
RewriteCond %{HTTP_REFERER} !^https://www.patriotaction.us$      [NC]
RewriteRule .*\.(jpg|jpeg|gif|png|bmp)$ - [F,NC]

deny from 128.90.136.126
deny from 104.28.89.80
deny from 216.244.16.66
deny from 172.59.193.121
deny from 216.244.66.246

<RequireAll>
Require all granted
Require not ip 185.191.171.0/255
Require not ip 85.208.96.0/255
</RequireAll>

I had to be specific about the semrushbot.  Those using it do not follow civil protocols.

I had to deny access to two VPN sites that were being used to overload our systems.

I have implemented several items that were suggested further up thread that helped immensely.

I will have CLoudFlare implemented soon and I will add three policies that were also suggested up thread as well. Once those policies are in place I will remove one or two of the commands in my file, which will reduce SQL load. 

Currently, we average about 40-55% SQL load. WE only spike when we take massive requests from IP's not blocked. We occasionally get 500 requests that are blocked in about 3 seconds. This still eats up SQL resources to answer with a 403 forbidden message.

I have used many suggestions here and they have been a great help.

One thing I did recently is add the Semrushbot to my spiders/bots list in the admin control panel. I gave it the Banned user profile accessing nothing.
Reply
#52
(2025-11-15, 07:21 PM)lost puppy Wrote: # 29-Mar-24 Bot to Block
BrowserMatchNoCase "libwww-perl" bad_bot
BrowserMatchNoCase "wget" bad_bot
BrowserMatchNoCase "LieBaoFast" bad_bot
BrowserMatchNoCase "Mb2345Browser" bad_bot
BrowserMatchNoCase "zh-CN" bad_bot
BrowserMatchNoCase "MicroMessenger" bad_bot
BrowserMatchNoCase "facebookexternalhit/1.1" bad_bot
BrowserMatchNoCase "Kinza" bad_bot
BrowserMatchNoCase "Bytespider" bad_bot
BrowserMatchNoCase "Baiduspider" bad_bot
BrowserMatchNoCase "Sogou" bad_bot
BrowserMatchNoCase "Datanyze" bad_bot
BrowserMatchNoCase "AspiegelBot" bad_bot
BrowserMatchNoCase "adscanner" bad_bot
BrowserMatchNoCase "serpstatbot" bad_bot
BrowserMatchNoCase "spaziodat" bad_bot
BrowserMatchNoCase "undefined" bad_bot
BrowserMatchNoCase "claudebot" bad_bot
BrowserMatchNoCase "Amazonbot" bad_bot
BrowserMatchNoCase "Python/3.10 aiohttp/3.9.3" bad_bot
BrowserMatchNoCase "HeadlessChrome" bad_bot
BrowserMatchNoCase "AhrefsBot/7.0" bad_bot
Order Deny,Allow
Deny from env=bad_bot[/code]

look at this part in lost puppy's post and learn
Support ended!
Reply
#53
(2025-11-17, 03:44 PM)bv64 Wrote:
(2025-11-15, 07:21 PM)lost puppy Wrote: # 29-Mar-24 Bot to Block
BrowserMatchNoCase "libwww-perl" bad_bot
BrowserMatchNoCase "wget" bad_bot
BrowserMatchNoCase "LieBaoFast" bad_bot
BrowserMatchNoCase "Mb2345Browser" bad_bot
BrowserMatchNoCase "zh-CN" bad_bot
BrowserMatchNoCase "MicroMessenger" bad_bot
BrowserMatchNoCase "facebookexternalhit/1.1" bad_bot
BrowserMatchNoCase "Kinza" bad_bot
BrowserMatchNoCase "Bytespider" bad_bot
BrowserMatchNoCase "Baiduspider" bad_bot
BrowserMatchNoCase "Sogou" bad_bot
BrowserMatchNoCase "Datanyze" bad_bot
BrowserMatchNoCase "AspiegelBot" bad_bot
BrowserMatchNoCase "adscanner" bad_bot
BrowserMatchNoCase "serpstatbot" bad_bot
BrowserMatchNoCase "spaziodat" bad_bot
BrowserMatchNoCase "undefined" bad_bot
BrowserMatchNoCase "claudebot" bad_bot
BrowserMatchNoCase "Amazonbot" bad_bot
BrowserMatchNoCase "Python/3.10 aiohttp/3.9.3" bad_bot
BrowserMatchNoCase "HeadlessChrome" bad_bot
BrowserMatchNoCase "AhrefsBot/7.0" bad_bot
Order Deny,Allow
Deny from env=bad_bot[/code]

look at this part in lost puppy's post and learn

Interesting, this allows it to find the bot in any browser used. Do you have any other suggestions?
Reply
#54
if you see a bot you don't want to see any more, note it's name and put it in the list, it's so easy
Support ended!
Reply
#55
If you're running Apache 2.4, and using that deny syntax, think you'll need mod_access_compat for backward compatibility with Apache 2.2 access control.

As far as I know, won't keep a given bot out of your web logs, but they will get a 403 response.
Reply
#56
I block all bots except Google, Bing, Yahoo, DuckDuckGo, and upptime.js.org through Cloudflare. It works much more efficiently than Robots.txt.
The latter because I use https://upptime.js.org/... if you don't use it then it can be blocked as well. Personally, it works very well.

[Image: m3e-Lu-KV-1.png]

Quote:cf.client.bot and not lower(http.user_agent) contains "google" and not lower(http.user_agent) contains "bing" and not lower(http.user_agent) contains "yahoo" and not lower(http.user_agent) contains "duckduckgo" and not lower(http.user_agent) contains "upptime.js.org"
Reply
#57
For a few weeks, I've noticed this new oddity, where I check whose online, and I'll see around 3-12 "guests" coming from a wide variety of normal-looking (ISP) IP addresses, who have all read the same thread in the last minute. Then things look normal for a few minutes. Then a bunch more IPs all reading some other thread. Are others seeing that? It's not causing much load so far because it's sporadic, but it's hard for me to understand what that is doing for anyone. I suppose it could be bug but it's been going on for awhile.
Reply
#58
Unknown bots seem to be crawling the net en masse.
https://community.mybb.com/thread-94101-...pid1404965

Try the Bad Behaviour plugin, it may not be perfecto but seems to work.
Send me a DM or PM if you want to sponsor any update or addition to my plugins.

[Image: signature.png]

Discord omar.gonzalez (Omar G.#6117); Telegram at @omarugc;
Reply
#59
(2025-12-09, 05:28 PM)Dave H Wrote: For a few weeks, I've noticed this new oddity, where I check whose online, and I'll see around 3-12 "guests" coming from a wide variety of normal-looking (ISP) IP addresses, who have all read the same thread in the last minute. Then things look normal for a few minutes. Then a bunch more IPs all reading some other thread. Are others seeing that? It's not causing much load so far because it's sporadic, but it's hard for me to understand what that is doing for anyone. I suppose it could be bug but it's been going on for awhile.

Yes, I can confirm that this also happened on my forum. When I checked the Who's Online list, there was a page of visitors/bots looking at the same topic.

using the rule from the post above and adding the rule below, which challenges non-target visitors (for me, visitors who are not from Brazil or Japan), the problem described above is completely resolved. Now, adding up the bots and visitors, it doesn't exceed two pages, whereas before, the bots alone would take up about four pages of Who's Online list.

[Image: T6-Hiie-N-1.png]

not ip.geoip.country in {"BR" "JP"} and not lower(http.user_agent) contains "google" and not lower(http.user_agent) contains "bing" and not lower(http.user_agent) contains "yahoo" and not lower(http.user_agent) contains "duckduckgo" and not lower(http.user_agent) contains "upptime.js.org"

ps. You need to exclude the allowed bots from the rule.

and below is static of rules...

[Image: XDAT8-JE-1.png]
Reply
#60
A good site with a list of rules you can deploy https://wafrules.com/ for cloudflare
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)