MyBB Community Forums

Hi,

I am getting a lot of 403 errors for my forum links at my webmaster tools, for example -

http://www.couponzguru.com/forum/attachment.php?aid=176

I dont want such pages to crawled by googlebots. As per my understanding, I should block such pages from my robots.txt.

Since my forum is on a sub-directory i.e. www.couponzguru.com/forum/ and my robots.txt is there on my root domain i.e. couponzguru.com, I think I should modify the main directories robots.txt.

My question is -

1. Can I include multiple sitemaps in the same robots.txt. As I already have a different sitemap defined for my root domain in that robots.txt. I want to add my google seo sitemap for forum also there..Is that fine?

2. To disallow bots should i mention like this? As below. Please review and let me know if any changes are required. This will be in addition to my root domain's robots content which is installed on wordpress[/php]

Sitemap: http://www.couponzguru.com/forum/sitemap-index.xml

User-Agent: *
Disallow: /forum/syndication.php*
Disallow: /forum/captcha.php
Disallow: /forum/editpost.php
Disallow: /forum/forumdisplay.php*
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/modcp.php
Disallow: /forum/moderation.php
Disallow: /forum/newreply.php*
Disallow: /forum/newthread.php*
Disallow: /forum/online.php
Disallow: /forum/portal.php*
Disallow: /forum/printthread.php*
Disallow: /forum/private.php
Disallow: /forum/ratethread.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendthread.php*
Disallow: /forum/showteam.php
Disallow: /forum/*sortby*
Disallow: /forum/task.php
Disallow: /forum/user*
Disallow: /forum/usercp.php*
Disallow: /forum/usercp2.php*
Disallow: /forum/calendar.php
Disallow: /forum/*action=emailuser*
Disallow: /forum/*action=nextnewest*
Disallow: /forum/*action=nextoldest*
Disallow: /forum/*year=*
Disallow: /forum/*action=weekview*
Disallow: /forum/*action=nextnewest*
Disallow: /forum/*action=nextoldest*
Disallow: /forum/*sort=*
Disallow: /forum/*order=*
Disallow: /forum/*mode=*
Disallow: /forum/*datecut=*
Disallow:/ *next
Disallow:/ *print
Disallow:/ *reply
Disallow:/ *post
Disallow:/ *action
Disallow:/ *user
Disallow:/ *=
Disallow: /forum/archive/

That code should be in a robots.txt file and it looks ok.

Just a small query - for -

Disallow:/ *next
Disallow:/ *print
Disallow:/ *reply
Disallow:/ *post
Disallow:/ *action
Disallow:/ *user
Disallow:/ *=

Should there be a space between : and /

Don't add wildcards at the end of php files. /forum/usercp.php* -> /forum/usercp.php

Quote:Disallow:/ *next
Disallow:/ *print
Disallow:/ *reply
Disallow:/ *post
Disallow:/ *action
Disallow:/ *user
Disallow:/ *=

Space should be before / not after it, and most likely you do not want to have those rules in your robots.txt - they either do nothing or they do too much.

You can test your robots.txt in Google Webmaster Tools. (For a given URL it tells you if it's blocked or not and which rule is blocking it)

this won't stop to receive 403 errors from your webmaster tools, it's probably you have Google seo plugin installed, correct?
In this case you have to disable the

404 widget feature:
Add the Google 404 widget for 404/403 error pages.
set it to NO.

To block google to index such pages add the noindex tag, there is a plugin that do this but i don't remember the name..