Recommended robots.txt for MyBB

[Gary Bolton]

How comes nobody has added the "cache" directory not to be indexed, just curious. Is there a reason why?

Disallow: /cache/

(12-11-2008 06:37 AM)Dennis Tsang Wrote:  Something like this should work:

Code:

User-Agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /misc.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /sendthread.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php

If your forum isn't in the root directory, you'll have to modify the txt accordingly.

We used to have one here, but it disappeared some time ago

[frostschutz]

/cache/ contains the CSS styles which may be of some importance to the Google bot.

Other than that, /cache/ should appear empty and not worth indexing anyway.

[Gary Bolton]

(06-28-2011 03:33 PM)frostschutz Wrote:  /cache/ contains the CSS styles which may be of some importance to the Google bot.

Other than that, /cache/ should appear empty and not worth indexing anyway.

That makes sense, sure. I suppose though "if you enabled the setting to not let pages be cached" in MyBB. Then you could block it, as there would be no reason for Google to use the stylesheets in the cache folder for displaying cached pages in that situation.

[-AW-]

Here's mine. It's Google SEO's with a few extra things (and some of it is specific to my forum).

Code:

Sitemap: http://www.dashingforums.com/sitemap-index.xml

User-Agent: *
Disallow: http://www.dashingforums.com/chat
Disallow: http://www.dashingforums.com/archive
Disallow: http://www.dashingforums.com/captcha.php
Disallow: http://www.dashingforums.com/editpost.php
Disallow: http://www.dashingforums.com/misc.php
Disallow: http://www.dashingforums.com/modcp.php
Disallow: http://www.dashingforums.com/moderation.php
Disallow: http://www.dashingforums.com/newreply.php
Disallow: http://www.dashingforums.com/newthread.php
Disallow: http://www.dashingforums.com/online.php
Disallow: http://www.dashingforums.com/printthread.php
Disallow: http://www.dashingforums.com/private.php
Disallow: http://www.dashingforums.com/ratethread.php
Disallow: http://www.dashingforums.com/report.php
Disallow: http://www.dashingforums.com/reputation.php
Disallow: http://www.dashingforums.com/sendthread.php
Disallow: http://www.dashingforums.com/task.php
Disallow: http://www.dashingforums.com/usercp.php
Disallow: http://www.dashingforums.com/usercp2.php
Disallow: http://www.dashingforums.com/calendar.php
Disallow: http://www.dashingforums.com/thankyoulike.php
Disallow: http://www.dashingforums.com/postloop.php
Disallow: http://www.dashingforums.com/tylsearch.php
Disallow: http://www.dashingforums.com/memberlist.php
Disallow: http://www.dashingforums.com/*action=emailuser*
Disallow: http://www.dashingforums.com/*action=nextnewest*
Disallow: http://www.dashingforums.com/*action=nextoldest*
Disallow: http://www.dashingforums.com/*year=*
Disallow: http://www.dashingforums.com/*action=weekview*
Disallow: http://www.dashingforums.com/*action=nextnewest*
Disallow: http://www.dashingforums.com/*action=nextoldest*
Disallow: http://www.dashingforums.com/*sort=*
Disallow: http://www.dashingforums.com/*order=*
Disallow: http://www.dashingforums.com/*mode=*
Disallow: http://www.dashingforums.com/*datecut=*
Allow: /