Ahrefs.com Bots
#1
Anyone else had these dickwads aggressively crawl your boards/websites? A couple days ago, ZEJ had 64 (!) guests online, and at LEAST 57 of them were from the 173.199.1xx.xxx ranges, which according to Risky Internet, is linked to these guys. All crawlers.

Pay attention to the validity rating; according to their system, a 21 is seriously bad.
Reply
#2
I have about a dozen 151.*.*.* guests on mine. I guess I should do some research tonight.
PGP Key (Fingerprint: 23B6 F4C0 FE2D 45AA 61A0 1E86 DB87 09DC DD87 6E40)
Reply
#3
user-agent: AhrefsBot
disallow: /

add to robots.txt
Reply
#4
(2012-12-14, 02:03 PM)Fellhound Wrote: user-agent: AhrefsBot
disallow: /

add to robots.txt

Already nuked their IPs and user agents long ago.
Reply
#5
Wow, this must've been what hit me a long while ago.

All of a sudden I noticed a few hundred extra "guests", but all the IPs were the same except for the last set, it started from .1 and went up.

Here's that IP range:
213.186.112.0/20

Looking up this IP:
http://whois.net/ip-address-lookup/213.186.112.0

inetnum:        213.186.112.0 - 213.186.112.255
netname:        UTEL-DC0
descr:          Utel DataCenter networks
country:        UA
admin-c:        UNOC-RIPE
tech-c:         UNOC-RIPE
status:         ASSIGNED PA
mnt-by:         AS6877-MNT
remarks:        INFRA-AW
remarks:        net or SPAM abuses please mail to [email protected]
source:         RIPE # Filtered

role:           Utel NOC
address:        59, Turgenivska str.
address:        04050 Kyiv, Ukraine
phone:          +380 44 2359001
phone:          +38 800 5050500
fax-no:         +380 44 2880278

It's from a Data Center in the Ukraine.
Reply
#6
Surprisingly, that's either not the case or RI hasn't updated their IP list yet. Lowest range listed is 213.186.119.0/20.
Reply
#7
I had to ban these ranges:

104.28.57.*
104.28.58.*

I still saw them for a little while after that, but I'm guessing that's because they're banned in MyBB and not .htaccess or the httpd configuration.

I think robots.txt depends on the good faith cooperation of the bot, so that wouldn't stop them.
Reply
#8
(2023-07-14, 10:08 PM)Devastatia Wrote: I had to ban these ranges:

104.28.57.*
104.28.58.*

Those IPs seem to belong to Cloudflare. It's probably not a good idea to ban them, at least not without first working out why you're getting visits from Cloudflare.
Reply
#9
(2023-07-15, 02:02 AM)Laird Wrote: Those IPs seem to belong to Cloudflare. It's probably not a good idea to ban them, at least not without first working out why you're getting visits from Cloudflare.

I posted the wrong IPs. Those ranges were being used by an iPhone user who camped on my front page for days and never clicked on any of the threads. I thought it was suspicious.

The range that had the Ahrefs bot in the user agent string was 51.222.253.*

Is there a reason Cloudflare would pose as an iPhone user?

According to this page, Cloudflare acts as a proxy and forwards the user's user agent string. Which I understand to mean it was an actual iPhone user, not Cloudflare itself.

https://helloacm.com/cloudflare-and-user-agent/
Reply
#10
Cloudflare is a content delivery network, so, typically, you'd be getting traffic from Cloudflare only if you'd set up an account with them to use their network to speed up delivery of content on your board (or to use their DDoS protection services, or similar). There might be other reasons you'd be getting traffic from them. Maybe your host provides some sort of Cloudflare integration by default? Or maybe there's something else of which I'm ignorant. In any case, it's probably best to find out, because if you block those IPs, you might be blocking legitimate traffic.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)