2011-01-18, 07:15 PM
2011-01-18, 07:15 PM
2011-02-05, 09:03 PM
Why are some bots listed twice? For example you have this below.
1: Baidu[B] - Baiduspider
2: Baidu[S] - Baiduspider
Both those listed use the same user-agent string, the only difference is you've added [S] and [B] after the spider bots name. But they are exactly the same if you remove the brackets, no difference at all from what I can see. Plus, I've spotted a few other cases like this while just adding some of them, so I wanted to ask why because it looks like I'm adding duplicate entries of the same bot?
1: Baidu[B] - Baiduspider
2: Baidu[S] - Baiduspider
Both those listed use the same user-agent string, the only difference is you've added [S] and [B] after the spider bots name. But they are exactly the same if you remove the brackets, no difference at all from what I can see. Plus, I've spotted a few other cases like this while just adding some of them, so I wanted to ask why because it looks like I'm adding duplicate entries of the same bot?
2011-02-05, 09:45 PM
lol. it happens so bro. its not a small list to check each entry for duplicate bot. i added them after collecting from different sources. well, now if you dont want to have duplicate entries, just delete them. i'm sure there are not much duplicate bots. i tried to eliminate as many duplicate bots as i can. somewhere in 2nd page, i added my bots list that are around 70bots indexing and are unique.
2011-02-05, 09:50 PM
OK, I just wanted to know if I was correct in what I was thinking before proceeding any further.
I'm doing it now by adding them all to a "database" using MS Works, adding two database fields, then they can be arranged by either Bot Name or User-Agent String (A-Z). It will be easy to spot all the duplicates and remove them from the list.
I'm doing it now by adding them all to a "database" using MS Works, adding two database fields, then they can be arranged by either Bot Name or User-Agent String (A-Z). It will be easy to spot all the duplicates and remove them from the list.
2011-02-06, 06:37 PM
yeah. you can.
i dont have much time in filtering them so my another intention is, if i leave those duplicate bots as they are, i can just remove duplicate entries when the bots crawls the site and when i feel that the bot is still active. because, most of the bots are dead by now as i collected from old and new sources. i think, now its not the time to remove those duplicates entries putting extra effort as we dont know still they are alive or not.
once i found a bot crawling and if it has duplicate entires(which will be around 4-6 or less), i will just remove them manually, keeping the unique ones.
i dont have much time in filtering them so my another intention is, if i leave those duplicate bots as they are, i can just remove duplicate entries when the bots crawls the site and when i feel that the bot is still active. because, most of the bots are dead by now as i collected from old and new sources. i think, now its not the time to remove those duplicates entries putting extra effort as we dont know still they are alive or not.
once i found a bot crawling and if it has duplicate entires(which will be around 4-6 or less), i will just remove them manually, keeping the unique ones.
2011-02-06, 06:41 PM
The easy way to be honest, is just follow this link and add them from this site below. They are already sorted out and the database is kept updated, I added them all from there today: http://www.wolfshead-solutions.com/spiders-list
2011-02-07, 02:23 AM
(2011-02-06, 06:41 PM)Gary Bolton Wrote: [ -> ]The easy way to be honest, is just follow this link and add them from this site below. They are already sorted out and the database is kept updated, I added them all from there today: http://www.wolfshead-solutions.com/spiders-list
Bookmarked. Thank You!
2011-02-07, 02:57 AM
Yes, Gary Bolton.
I had collected the bots from there also and i see they are not importable to mybb database as like they are. So, i made them importable to the database in sql format.
My list contains around 1150 which includes all mostly known available bots (including dead and alive)
I had collected the bots from there also and i see they are not importable to mybb database as like they are. So, i made them importable to the database in sql format.
My list contains around 1150 which includes all mostly known available bots (including dead and alive)
2011-02-25, 01:02 PM
Thanks for this but.Now i have 20+ on "Who's Online List" Can i minimze the number of Bota viewed on Who's Online list ?
2011-02-25, 01:49 PM
(2011-02-25, 01:02 PM)hellothere1 Wrote: [ -> ]Thanks for this but.Now i have 20+ on "Who's Online List" Can i minimze the number of Bota viewed on Who's Online list ?
That's the whole idea behind it, so you can see which are Bots online, that might be fooled into otherwise thinking they are human guests.