Solved: 3 Years, 10 Months, 4 Weeks, 6 Hours, 9 Minutes, 29 Seconds ago Recommended robots.txt for MyBB
#31
Solved: 3 Years, 10 Months, 4 Weeks, 6 Hours, 9 Minutes, 29 Seconds ago
How comes nobody has added the "cache" directory not to be indexed, just curious. Is there a reason why?

Disallow: /cache/

(12-11-2008, 06:37 AM)Dennis Tsang Wrote:  Something like this should work:
Code:
User-Agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /misc.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /sendthread.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php

If your forum isn't in the root directory, you'll have to modify the txt accordingly.

We used to have one here, but it disappeared some time ago Wink
#32
Solved: 3 Years, 10 Months, 4 Weeks, 6 Hours, 9 Minutes, 29 Seconds ago
This user has been denied support. This user has been denied support.
/cache/ contains the CSS styles which may be of some importance to the Google bot.

Other than that, /cache/ should appear empty and not worth indexing anyway.
#33
Solved: 3 Years, 10 Months, 4 Weeks, 6 Hours, 9 Minutes, 29 Seconds ago
(06-28-2011, 03:33 PM)frostschutz Wrote:  /cache/ contains the CSS styles which may be of some importance to the Google bot.

Other than that, /cache/ should appear empty and not worth indexing anyway.

That makes sense, sure. I suppose though "if you enabled the setting to not let pages be cached" in MyBB. Then you could block it, as there would be no reason for Google to use the stylesheets in the cache folder for displaying cached pages in that situation.

#34
Solved: 3 Years, 10 Months, 4 Weeks, 6 Hours, 9 Minutes, 29 Seconds ago
Here's mine. It's Google SEO's with a few extra things (and some of it is specific to my forum).

Code:
Sitemap: http://www.dashingforums.com/sitemap-index.xml

User-Agent: *
Disallow: http://www.dashingforums.com/chat
Disallow: http://www.dashingforums.com/archive
Disallow: http://www.dashingforums.com/captcha.php
Disallow: http://www.dashingforums.com/editpost.php
Disallow: http://www.dashingforums.com/misc.php
Disallow: http://www.dashingforums.com/modcp.php
Disallow: http://www.dashingforums.com/moderation.php
Disallow: http://www.dashingforums.com/newreply.php
Disallow: http://www.dashingforums.com/newthread.php
Disallow: http://www.dashingforums.com/online.php
Disallow: http://www.dashingforums.com/printthread.php
Disallow: http://www.dashingforums.com/private.php
Disallow: http://www.dashingforums.com/ratethread.php
Disallow: http://www.dashingforums.com/report.php
Disallow: http://www.dashingforums.com/reputation.php
Disallow: http://www.dashingforums.com/sendthread.php
Disallow: http://www.dashingforums.com/task.php
Disallow: http://www.dashingforums.com/usercp.php
Disallow: http://www.dashingforums.com/usercp2.php
Disallow: http://www.dashingforums.com/calendar.php
Disallow: http://www.dashingforums.com/thankyoulike.php
Disallow: http://www.dashingforums.com/postloop.php
Disallow: http://www.dashingforums.com/tylsearch.php
Disallow: http://www.dashingforums.com/memberlist.php
Disallow: http://www.dashingforums.com/*action=emailuser*
Disallow: http://www.dashingforums.com/*action=nextnewest*
Disallow: http://www.dashingforums.com/*action=nextoldest*
Disallow: http://www.dashingforums.com/*year=*
Disallow: http://www.dashingforums.com/*action=weekview*
Disallow: http://www.dashingforums.com/*action=nextnewest*
Disallow: http://www.dashingforums.com/*action=nextoldest*
Disallow: http://www.dashingforums.com/*sort=*
Disallow: http://www.dashingforums.com/*order=*
Disallow: http://www.dashingforums.com/*mode=*
Disallow: http://www.dashingforums.com/*datecut=*
Allow: /


Forum Jump:


Users browsing this thread: 1 Guest(s)