MyBB Community Forums

Hello, i'm using Google SEO Sitemap from Andreas Klauer.
All works fine, but ihave a problem with the link.
The link is www.mydomain.com/forum/sitemap-threads.xml?page=1

The problem is that i have blocked all links with a question mark (?) in the url.
In this case Google cannot crawl the sitemap because it's blocked by robots.txt

Is there a way to have the full sitemap without the need to have a question mark into the url ? For example: www.mydomain.com/forum/sitemap.xml or something like that.

Thank you

LT

may be its better to unblock links with ? symbols AND use specific URLs with ? to block...
eg.

Disallow: /misc.php?action=*

(2011-07-21, 09:37 AM)ranjani Wrote: [ -> ]may be its better to unblock links with ? symbols AND use specific URLs with ? to block...
eg.

Disallow: /misc.php?action=*

Hi thanks for your reply
the problem is that i have a lot of duplicates due to "page=*" links coming from Google (and from the forum).

This is the url rewrite rule for the sitemap in my .htaccess

RewriteRule ^sitemap-([^./]+)\.xml$ misc.php?google_seo_sitemap=$1 [L,QSA,NC]
The sitemap url is then forum/sitemap-threads.xml?page=1

Is it possible to remove "page=1" at the end?

Thanks ranjani

No.

I'd advise against blocking ? URLs, as it will likely block content as well; however if you absolutely must do it, you can simply make an exception for the sitemap using Allow: in robots.txt.

Google SEO already uses redirect and canonical to handle ?page dupes. So yourthread?page=1 will actually be indexed as just yourthread in the long run. There's no reason to block ? globally really, in fact it will hurt you (because Google can't index subsequent thread pages anymore, thereby hiding content of your site).

Ok thanks, i will try it