MyBB Community Forums

Full Version: robots.txt is it correct..?? or need some changes.??
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
User-Agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /misc.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendthread.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /calendar.php
Disallow: /*action=emailuser*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*year=*
Disallow: /*action=weekview*
Disallow: /*action=nextnewest*
Disallow: /*action=nextoldest*
Disallow: /*sort=*
Disallow: /*order=*
Disallow: /*mode=*
Disallow: /*datecut=*
Disallow: /member.php?action=login
Disallow: /member.php?action=register
Allow: /

1.This robots.txt is created by me, is there any other link that is should add but missed..??
2. Searching on google at some places i found that, if you used * just after user-agent then you cannot use * in links like this one "Disallow: /*datecut=*". What you think.?? is it true or not..?
3. What is the function of Allow: / at the end of robots.txt ?? Should i remain it as it is now, or should i remove it..?
This look fine for what you want. You can probably remove online.php
Looks fine to me, though I personally disallow member.php completely as bots have no need to browse those pages at all.
hmmm.. Okay... Thanks for your feedback..!

@JordanMussi and @euantor
What about my 2nd and 3rd Question...??
2. Never had any issues my self at all
3. Allow: / lets bots access all other content not explicitly disallowed
Just what euantor said
Thanks...!
@JordanMussi and @euantor

for fast and appropriate feedback..!