MyBB Community Forums

Hello;
I created a group for spider/bots with own permission and spider bots are defined in this group. all things works before, but:

first I found that google can not look my site map that generated by googleseo plugin, when I enter url of site map in my browser it works fine but when I submit this url in google webmaster tools, tell me "General HTTP error: 404 not found".

then I found that when I defined a bots as a group except "Guest" in ACP> Spiders/Bots, that bots can not access to my forum.
for example when I add "gtmetrix" in spiders/bots list as "registered" group, this site can not analyze my site but when I change its group to "Guest" it works fine.

can you help me to fix this problem?
may be this problem related to my host or not?

Thanks in advance

Do you have a 'robots.txt' file?

If so, what are it's contents?

^ yes:

http://forum.civilea.com/robots.txt

Sitemap: http://forum.civilea.com/sitemap-index.xml
user-agent: *
Disallow: /download/
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /misc.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendthread.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /calendar.php
Allow: /

So look to see why that would be, I would start thinking that the IP address is blocked and maybe show your hosts that testing with a googlebot user agent works, but fetch as googlebot does not.

Could you screenshot the permissions page for the bots group (and possibly create a limited admin account with access to the bots page and usergroup pages and PM me the login details and i can check it out for you)

Also, before you assigned the bots to the custom usergroup did everything work OK? Just saw the part where you said it could.

^ permission is ok, even when I assign bots to registered user or moderators, they can not fetch forum. and if IP blocked why bots as guest user can see forum?

---

this is my host provider reply:

Quote:It's very hard to say what exactly causing this problem without exact requirements for your applications. In this case please check this issue with your software developers and if any settings should be changed from server, if any software/module should be installed on your server let us know and we will be glad to assist you further.

My alexa rank decreased from 20000 to 60000 because of this error!

I found that when googleseo 404 is enable this error occurred.why?