MyBB Community Forums

I've been having a lot of server max load messages lately. It was suggested to me that bots may be the issue. Could this be the case and if so, how would I restrict/deny bots or spiders?

Look at your host in awstats if they are spiders/ bots

You can deny them from .htaccess file

(2013-04-24, 07:52 PM)Markus_Wolf Wrote: [ -> ]Look at your host in awstats if they are spiders/ bots

You can deny them from .htaccess file

What happens if you delete a bot/spider listed in the Bot/Spider part of the ACP? Does it just not show it? Or does it not allow it to access? For instance, if I delete Baidu, does it then become blocked?

i think it still will try to access your website

(2013-04-26, 11:57 AM)sinbad Wrote: [ -> ]

(2013-04-24, 07:52 PM)Markus_Wolf Wrote: [ -> ]Look at your host in awstats if they are spiders/ bots

You can deny them from .htaccess file

What happens if you delete a bot/spider listed in the Bot/Spider part of the ACP? Does it just not show it? Or does it not allow it to access? For instance, if I delete Baidu, does it then become blocked?

No. he will come again, but you dont see him. If you want to block spider like baidu from your site add in .htaccess file these lines:

#Baiduspider
User-agent: Baiduspider
Disallow: /

---

Quote:No. he will come again, but you dont see him. If you want to block spider like baidu from your site add in .htaccess file these lines:

#Baiduspider
User-agent: Baiduspider
Disallow: /

My mistake. create file robots.txt and add lines above in robots.txt, then put that file in your root folder, or whereever your forum is

(2013-04-26, 12:32 PM)Markus_Wolf Wrote: [ -> ]

(2013-04-26, 11:57 AM)sinbad Wrote: [ -> ]

(2013-04-24, 07:52 PM)Markus_Wolf Wrote: [ -> ]Look at your host in awstats if they are spiders/ bots

You can deny them from .htaccess file

What happens if you delete a bot/spider listed in the Bot/Spider part of the ACP? Does it just not show it? Or does it not allow it to access? For instance, if I delete Baidu, does it then become blocked?

No. he will come again, but you dont see him. If you want to block spider like baidu from your site add in .htaccess file these lines:

#Baiduspider
User-agent: Baiduspider
Disallow: /

---

Quote:No. he will come again, but you dont see him. If you want to block spider like baidu from your site add in .htaccess file these lines:

#Baiduspider
User-agent: Baiduspider
Disallow: /

My mistake. create file robots.txt and add lines above in robots.txt, then put that file in your root folder, or whereever your forum is

OK, thanks. Will give it a try this weekend.

If a bot visits and doesn't match anything in your spiders / bots, then it will appear as a guest.

I deleted/blocked the baidu bots via server and simple/no problem. However, the issue with the server load ended up being on the provider side. They had to bring in some big guns to solve the problem, I wonder if it ended up being that nasty stealth Trojan that is exploiting Apache servers.

For misbehaving bots, I've added them to the "banned" group Not all bots obey robots.txt.