MyBB Community Forums

Hello,

I went to Awstat on my cPanel to see which bots are crawling my website and this is what I got:


GoogleBot, 282,192+70 Hits, 2.38 GB Bandwidth
Unknown robot (identified by 'spider'), 119,060+94 Hits, 7.57 GB Bandwidth
Yahoo Slurp
Unknown robot (identified by hit on 'robots.txt')
Unknown robot (identified by 'bot*')
Unknown robot (identified by 'crawl')
Alexa (IA Archiver)
MSNBot
Unknown robot (identified by empty user agent string)
Unknown robot (identified by '*bot')
MSNBot-media
Unknown robot (identified by 'robot')
.........

I googled that 'spider', and it seems that is potentially a Spambot looking for emails or trying to register and spam...

I added them to the Spiders/Bots on my MyBB forum, I defined them as "Banned" so they won't generate such a high overload on my server anymore...

I just wanted to know if you have any idea what these robots are?? Is it a good thing to ban them?? Should I ban the others too??

If they're those spambots creating fake accounts, it would be good to ban them officially on MyBB's forthcoming versions

There are a lot of bots that are fine (smaller or more unknown tech companies). I would recommend using cloudflare and it will block the bad ones for you.

(2011-11-29, 12:43 AM)Alex Smith Wrote: [ -> ]There are a lot of bots that are fine (smaller or more unknown tech companies). I would recommend using cloudflare and it will block the bad ones for you.

Oh thank you, except that I don't like cloudflare at all...
I don't think I need robots except Google / Yahoo and few others

(2011-11-29, 12:52 AM)TheGarfield Wrote: [ -> ]

(2011-11-29, 12:43 AM)Alex Smith Wrote: [ -> ]There are a lot of bots that are fine (smaller or more unknown tech companies). I would recommend using cloudflare and it will block the bad ones for you.

Oh thank you, except that I don't like cloudflare at all...
I don't think I need robots except Google / Yahoo and few others

Why block bots though? A lot of people have their own indexiers. Who knows what they could make next .

(2011-11-29, 12:54 AM)Aristotle Wrote: [ -> ]

(2011-11-29, 12:52 AM)TheGarfield Wrote: [ -> ]

(2011-11-29, 12:43 AM)Alex Smith Wrote: [ -> ]There are a lot of bots that are fine (smaller or more unknown tech companies). I would recommend using cloudflare and it will block the bad ones for you.

Oh thank you, except that I don't like cloudflare at all...
I don't think I need robots except Google / Yahoo and few others

Why block bots though? A lot of people have their own indexiers. Who knows what they could make next .

I agree, but Spambots ruin everything