MyBB Community Forums

hi everybody.. ı m using this robots.txt configuration but in the google webmaster tools it says 8100 url has been restricted now my question how can ı fix this problem and whats the recommended robots.txt configuration?

Sitemap: http://www.itbiti.com/sitemap-index.xml
Sitemap: http://www.itbiti.com/misc.php?google_seo_sitemap=index

User-Agent: *
Disallow: /task.php
Disallow: /modcp.php
Disallow: /report.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /captcha.php
Disallow: /private.php
Disallow: /editpost.php
Disallow: /warnings.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /sendthread.php
Disallow: /ratethread.php
Disallow: /reputation.php
Disallow: /moderation.php
Disallow: /printthread.php
Disallow: /etiketler.php
Disallow: /*ref=*
Disallow: /*?ref=
Disallow: /*action=weekview*
Disallow: /*action=emailuser*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Allow: /

(2010-12-08, 07:34 PM)yapraqq Wrote: [ -> ]but in the google webmaster tools it says 8100 url has been restricted

That's the whole point - robots.txt restricts URLs.

The question is: does it restrict URLs you don't want restricted?

no it was indexing my threads in 6 or 10 hour .. but now its indexing my threads like
/archive/index.php/forum-145.html and in 1 or 2 days

It's indexing the archive version, and you've not disallowed that.

Well, you did enable canonical for the archive so it should switch to the thread version. I don't disallow archive myself - doing a site search, it finds 3 threads in the archive for me versus >12000 threads in the non-archive version. I think it would look different without canonical and sitemap (for example for mybb.com it's 90k vs 220k).

Then again, Google sees canonical as a recommendation only and is not forced to follow it. If possible, try to keep the noise on your main site as low as possible (for example don't display blocks of ads to Google).

(2010-12-08, 08:27 PM)MattRogowski Wrote: [ -> ]It's indexing the archive version, and you've not disallowed that.

how can ı do it?