Why not instead use Google Webmaster Tools to verify accuracy of robots.txt?
(2012-07-13, 07:01 PM)crazy4cs Wrote: [ -> ]Why not instead use Google Webmaster Tools to verify accuracy of robots.txt?
I checked it some minutes ago.
When I stay on robots.txt fetch show this message :
/Crawl postponed because robots.txt was inaccessible./
I had a robots.txt. some hours ago I edited it and I upload a new file.
Is this message for pending new robots.txt???!!!
Paste your robots.txt content here.
Sitemap: http://www.iranbiologists.com/sitemap-index.xml
User-Agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /misc.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendthread.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /calendar.php
Disallow: /*action=emailuser*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*year=*
Disallow: /*action=weekview*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*sort=*
Disallow: /*order=*
Disallow: /*mode=*
Disallow: /*datecut=*
Allow: /
So you disallow all that, then you allow everything?
Can anyone help me for create a true robots.txt file ?
I don't know any thing about create it
Basically now you disallow all the specified pages to every robot and then after that, you allow all robots access to / root directory.
However isn't it correct ?
allowed all pages except specific pages that disallowed!! No?
The robots.txt file is fine. The errors just state exactly what it is; optimized for Google bot.
Unless you're creating blocks for specific bots you don't need to worry about it too much. The other bots will simply ignore those rules.