Bots/Web Scrappers in 2025
#1
I am wondering about the lack of threads (a thread) related to the amazing appearance of weird bots and web scrappers that seemed to be a major problem for many sites starting about March of 2025.

Yes, these bots and web scrappers have been around since about - what? - two or three decades ago; but this year they got really weird. You can go to some other BB platforms supports communities (one in particular) and see the many admin folks posting about their battles and requesting help.

BUT, here we see very little activity related to this situation and I am wondering?

Please don't get me wrong; I think it could be great if maybe y'all have some code built into your platform that is like a good cockroach spray on a full community of monster cockroaches trying to invade a cyber home (site) --- BUT is that the case?

Or is it that those new nasty web scrappers have no interest in any of the MyBB sites?

Just wondering, and sorry for a first post to be so weird. Admin/Mods, please don't be paranoid about this old man's first post being so weird. I'll try to stay "normal" in future enquiries.

*By the way, where is the sub-forum for introductions?*
Reply
#2
Welcome to the internet,
There is no way to prevent your website to be visited by bots, spiders or crawlers. No way!
If your site is online, you can be addressed - Your resource is still a potential aim, so every website is aimed periodically.
I guess 80+% of all the traffic today is caused by any kind of web crawlers - unfortunately a waste of bandwidth.

Questions to ask:
1) Do you want your website to be crawled by bots? Do you want to be filed and listed in search engines?
Yes?
2) Do you want to get rid of bots and ban them with firewall rules? Do you want to be blacklisted and loose population?
No?

Today the internet and the world is perverse.
Make your decision ethically.

[ETS]
Reply
#3
Excuse me, [ExiTuS], but I thought I was asking why there has been no trouble with MyBB sites in a similar way as have - for example - phpBB sites? I mean, so many web scrappers and such not following the rules and causing many sites to be down due to heavy loads on servers, especially shared hosting and such?

PLUS, you assume I WANT to be added to any search engines? Quite frankly, I've been thinking that with the latest adventure I'll be having on the Net I may very well be keeping ALL bots off the site for the first few months while I do what I wish to do.

Oh well, I suppose I wrote something in the OP that is confusing and somebody thought I was writing about myself and I apologize for that. On the other hand, after reading the OP a few times I can't quite see where I wrote about my own ideas of bot/web scrapper control - - - BUT I'm looking around for my monocle - - - I know it is around this work station some place. Maybe I should buy one of those fancy new two-lense thingies that they call a glass or something - - - ah yes, glasses.

No way to stop bots, eh? I'll have a chat with my friend, Mr. Holmes, about that one. Of course, my other friend, Mr. Google, has some information on that, too: https://support.google.com/webmasters/th...some?hl=en
By the way, on that topic of Mr. Google, do any of you know about the PE Summit in Dublin this year?
Can't stop bots, eh? That must be a comedy thing and I should be laughing, yes? Still . . .
Reply
#4
If you think there is no interest in scraping MyBB sites, look at the peak users at the bottom of this forum.  The forum was completely unusable/unreachable at that point. My forum died around the same time with just over 15K guest bots.

At times I wonder if it is malicious.  ie: suck all of the internet into an LLM that can regurgitate it, then pummel the sources out of existence so the LLM is the only source of the info.

After 9 months of daily work blocking IPs/CIDRs and writing an extension to close the forum to guests whenever guest numbers got too high, I eventually hid behind Cloudflare.  I theorized that that's what this forum did as well, but whoever runs it hasn't said.  

Cloudflare is keeping my numbers/load much more manageable, and even when some bots get by, CF's caching seems to help quite a bit.  I hope they can stay ahead of it.  Yeah, there is still the fall-back of: keep everyone but registered users out and maybe only allow new registrations in-person with a blood-sample.....

I believe some people have been saying the web is dead and we need "internet 2" since the 1990s.  It does seem precarious now.
Reply
#5
I've been into this situation on a couple of support sites since about April and it has been quite an education. I have a report put out about April or May that is a full list and I've not seen anything like that in all the years of messing about on the Net. In fact, I had a hard time shrinking it into a pdf format to send to one hosting company that asked for it when they found out I had it. I'm working out of Japan and one site admin in the EU got so frustrated when so many started originating from Japan back in about August and he simply blocked anything that showed up on his site from Japan.

In fact, about 24-48 hours ago one report I received stated a whole new bunch of scrappers were coming out of the U.S. and that report author was running out of options on a shared hosting plan.

Reports on Cloudflare are trickling in about regular members now having troubles with Cloudflare due to changes instituted to help with blocking/blacklisting, and the new code spills over into paths where it wasn't intended to be.

Then there was an odd rumour going around that Cloudflare was going to go 100% proprietary, which then generated a whole new offshoot of rumours.

Bottom line for security operations; been a very uncomfortable year for a lot of webmasters. Especially the new ones.

By the way, on that "Net 2" idea, we have some large firms here in Japan now using www3.

Bottom line, though, is I think 2025 is going to have some folks thinking what was unthinkable just a few years ago. Some changes in some thought processes.
Reply
#6
One thing that is always going to set off some folks as it being a bad idea; the bots and other such software programs are eventually going to have to be brought under some sort of government control, or a sort of international agreement.

Back about May of this year when I had to focus attention on some new style web scrappers I found that these new style ones are not breaking any U.S. laws. I might have been a tad surprised right then, but not sure I was. But after a fair number of days having to study all this I am firmly of the mind that control in some form has to be initiated. Way too many regular folks with web sites have been severely impacted by those new style web scrappers that might have started growing in 2024, but not many folks were paying attention; but then this year, it has been a nightmare for too many folks.

Actually, for hosting companies, too.

Back to the key point of this post; regulations. When some nutcase like me starts writing about a need for regulations, in some form, some folks get nervous. But 2025 seems to be indicating that something needs doing. And relying on CloudFlare is NOT the final solution.

Sort of reminds me of the history books and that wild west stuff and that Pinkerton company that arose to save civilization in wild country. We need not repeat that style in this new "wild west". And we don't need any 'fast gun' new sheriff for every website. We need organized regulation at some level; government, or ourselves at an international level. New series from "Have Gun, Will Travel" to "Have 'htaccess' Won't Travel". Trouble is, htaccess isn't working so great, anymore. That "Move Letter Over" thingy is cute, but that won't work very well once some clever AI figures that out. If they haven't already. And I just used the pronoun "they" to refer to an AI program. Our human minds are slowly being programmed like that. Took the feline species thousands of years to train humans, but AI is going to do it in one hundred.
Reply
#7
Not sure if this post is going to get me in trouble, but I took a look at some market reports related to CloudFlare and I have developed a very uncomfortable thought process on a few matters that could .be related to what this thread is about.

Please note the use of "could" and that is actually related to what "could" be happening on another site where it seems that the answer being constantly tossed at folks asking for help is CloudFlare-CloudFlare-CloudFlare AND any other suggestions for bot control are sort of - - - Well, what would be polite? - - - Sort of stepped on.

On the other hand, maybe I'm just overly suspicious. But in this new arena of the Net some weird stuff IS taking place. Of course, for you folks in your early to middle years, this New business arena isn't new to you, is it? You might even think some of what is going on is normal. Heck, I still remember the days when on an overseas assignment one had to wait in line to be able to use a phone to call the States. Can't seem to remember if Uncle Sam paid for that call, though. I think it was free.

Whole new arena, but some of the oldest tricks in the book for making money can still be used in this new arena. And some of those "tricks" are illegal in the brick-and-mortar world. So very strange how we are going backwards in so many ways and so many folks aren't seeing it. And the ones that are will be dead in another 10 to 20 years and you younger folks may be in trouble.
Reply
#8
Your writing style jumps around too much. You start with one topic, then immediately move to another, using filler words that make your posts harder to follow. If you cut the fluff and get straight to the point, your posts would be much clearer and easier to read.

In short — your “blog-style” approach comes off as annoying and overcomplicated.

When it comes to URL scanning (a.k.a. crawlers, bots, or spiders), there’s no way to stop them completely. Every public website is visible to automated tools that check or “scan” URLs. However, you can reduce how many reaches you by using smart protection methods like:
  • Robots.txt – tells search bots like Google or Bing which pages they can or can’t access.
  • CAPTCHAs or JavaScript checks – blocks bots that can’t complete human actions.
  • Rate limits and firewalls – examples include ModSecurity or Cloudflare WAF, which slow down or block bad traffic.
  • Honeypots – hidden fields that trap bots when they try to fill them out.
  • Delayed content loading – stops bots that don’t run JavaScript from seeing your page.
  • Login or token protection – makes sure only real users or signed sessions can view certain pages.
  • Anti-bot services – like DataDome, Imperva, or Cloudflare Bot Fight Mode, which use AI to spot fake traffic in real time.

As for MyBB, it’s improved a lot over the years. Newer versions have stronger CAPTCHA options, better session checks, and smarter spam protection built in. Together, these updates make MyBB much better at detecting and blocking automated bots and fake signups.

Your thread composition style often exhibits excessive verbosity — meandering from one tangential idea to another, with an overabundance of transitional language that ultimately obscures the core argument. A more concise, direct expository approach would enhance both clarity and reader engagement by eliminating unnecessary narrative preambles.
Looking for great MyBB plugins & themes?
Join MyBBForge — The Forge is growing, check out our collection!
👉 Come forge something awesome with us.
Reply
#9
Thank you for the analysis of my writing style, but I think I had better get into a specific example, which I was worried about doing.

Quote:Newer versions have stronger CAPTCHA options, better session checks, and smarter spam protection built in.

Do you think that MyBB has better bot protection than phpBB?

I believe I had confined my focus on numbers related to bots/web scrappers. I don't recall having written about trouble with spam bots. I initially focused on numbers.

Another member also posted about guest numbers too high. And I repeat, I wrote quite extensively about numbers.

Sort of wondering, xxcorrosionxx, why you didn't touch on that?

By the way, there's a 32 page thread over at the support site for phpBB and it is exclusively focused on numbers in about the last five months. There are other sites where numbers over that same time span are the issue. But not here at MyBB, which I believe I focused on right away.

Quote:BUT, here we see very little activity related to this situation and I am wondering?

And then I did write about the oddity that over at phpBB (which I didn't identify in that post) they sure don't seem to have much faith in any company other than CloudFlare.

Good to see, xxcorrosionxx, that you added two more companies to that list.

But I really do wonder why those bots/web scrappers are so focused on phpBB and not on MyBB?

QUESTION
: Am I allowed to put that 32-page phpBB link here?
Reply
#10
I think MyBB has pretty good protection for the previous (but still existing!) generation of threats - spammers and the like.  But there seems little to no protection in MyBB from the "just read over and over until it dies" unintentional?? DDOS style of this latest round of bots.  The best I could come up with on my own was a simple extension to shut off guest access when the load gets high - it helps but its not that great.  (ie: bots still use a lot of resources reading the no permissions page 60,000 times an hour, it keeps casual human browsers out, and if it becomes a common response, these bots will probably try to create 60,000 accounts per hour, and you can limit accounts per day, but that keeps new human users from registering once the bots have used all the slots...) 

You can host a forum on ever bigger servers, but just one of the AI companies has a valuation of $500 billion, so I think they can out-server me/most of us.

I don't know why it seems to get little attention here.  There have been threads - including some from well before I first posted about it.  But I suppose no one has good answers so the threads die.  Does the 32 page thread have any solutions or even ideas for solutions?

Last week at a support forum (a Lithium forum) I noticed desktop CPU cores maxing out for 10 seconds, tablets not being able to get access at all...  I think they had some proof of work thing like Anubis running but it was set very high and lots of users complained about it until they dialed it down.  So even billion-dollar companies seem to be having issues with this.
Reply


Forum Jump:


Users browsing this thread: 1 Guest(s)