I have this robots.txt in my site
Sitemap: http://www.example.com/misc.php?google_seo_sitemap=index
User-Agent: *
Disallow: captcha.php
Disallow: editpost.php
Disallow: misc.php
Disallow: modcp.php
Disallow: moderation.php
Disallow: newreply.php
Disallow: newthread.php
Disallow: online.php
Disallow: printthread.php
Disallow: private.php
Disallow: ratethread.php
Disallow: report.php
Disallow: reputation.php
Disallow: search.php
Disallow: sendthread.php
Disallow: task.php
Disallow: usercp.php
Disallow: usercp2.php
Disallow: calendar.php
Disallow: *action=emailuser*
Disallow: *action=nextnewest*
Disallow: *action=nextoldest*
Disallow: *year=*
Disallow: *action=weekview*
Disallow: *action=nextnewest*
Disallow: *action=nextoldest*
Disallow: *sort=*
Disallow: *order=*
Disallow: *mode=*
Disallow: *datecut=*
Allow: /
Google says after uploading the sitemap:
URL restricted by robots.txt
I have to assume you're using Google SEO as you haven't said, but that uses misc.php
yes,i am using google seo
i dont understand what you are saying..
The sitemap is generated using misc.php, and you have blocked misc.php, hence it saying it's blocked by your robots.txt.
The robots.txt is just an example you can adapt to your own needs.
There is no problem with disallowing misc.php when the sitemap is sitemap-index.xml (i.e. it's rewritten). there may be a problem if it's misc.php?google_seo_sitemap=. So you may have to allow misc.php for that.
(2010-06-21, 04:42 PM)frostschutz Wrote: [ -> ]There is no problem with disallowing misc.php when the sitemap is sitemap-index.xml (i.e. it's rewritten). there may be a problem if it's misc.php?google_seo_sitemap=. So you may have to allow misc.php for that.
Yes you right..
@Net
here is example robots.txt
sitemap: http://www.example.com/sitemap-index.xml
User-Agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /misc.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /sendthread.php
Disallow: /search.php
Disallow: /fbfriends.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /calendar.php
Disallow: /*action=emailuser*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*year=*
Disallow: /*action=weekview*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*sort=*
Disallow: /*order=*
Disallow: /*mode=*
Disallow: /*datecut=*
Disallow: /yahoo/
Disallow: /archive/
Allow: /
Or
sitemap: http://www.example.com/misc.php?google_seo_sitemap=index
User-Agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /sendthread.php
Disallow: /search.php
Disallow: /fbfriends.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /calendar.php
Disallow: /*action=emailuser*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*year=*
Disallow: /*action=weekview*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*sort=*
Disallow: /*order=*
Disallow: /*mode=*
Disallow: /*datecut=*
Disallow: /yahoo/
Disallow: /archive/
Allow: /
but mine isn`t good enough?
and the archive folder i restricted...is it indicated?
"yahoo" folder i didn`t found on my web..
Is it a difference beetwen?
Disallow: /captcha.php
and
Disallow: captcha.php
^ sorry that is just example for my site
Here is better robots.txt for your mybb forum
sitemap: http://www.example.com/misc.php?google_seo_sitemap=index
User-Agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /sendthread.php
Disallow: /search.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /calendar.php
Disallow: /*action=emailuser*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*year=*
Disallow: /*action=weekview*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*sort=*
Disallow: /*order=*
Disallow: /*mode=*
Disallow: /*datecut=*
Allow: /
use
Disallow: /captcha.php