Current time: 04-23-2014, 08:15 PM Hello There, Guest! (LoginRegister)


Recommended robots.txt for MyBB
06-28-2011, 03:04 PM (This post was last modified: 06-28-2011 03:06 PM by Gary Bolton.)
Post: #31
Solved: 3 Years, 6 Months, 3 Weeks, 2 Days, 6 Hours, 10 Minutes, 47 Seconds ago RE: Recommended robots.txt for MyBB
How comes nobody has added the "cache" directory not to be indexed, just curious. Is there a reason why?

Disallow: /cache/

(12-11-2008 06:37 AM)Dennis Tsang Wrote:  Something like this should work:
Code:
User-Agent: *
Disallow: /captcha.php
Disallow: /editpost.php
Disallow: /misc.php
Disallow: /modcp.php
Disallow: /moderation.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /sendthread.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /usercp2.php

If your forum isn't in the root directory, you'll have to modify the txt accordingly.

We used to have one here, but it disappeared some time ago Wink

Community Discussion Home | Community Discussion Forum
Visit this user's website Find all posts by this user
06-28-2011, 03:33 PM (This post was last modified: 06-28-2011 03:33 PM by frostschutz.)
Post: #32
Solved: 3 Years, 6 Months, 3 Weeks, 2 Days, 6 Hours, 10 Minutes, 47 Seconds ago RE: Recommended robots.txt for MyBB
This user has been denied support. This user has been denied support.
/cache/ contains the CSS styles which may be of some importance to the Google bot.

Other than that, /cache/ should appear empty and not worth indexing anyway.

Google SEO | Gravatar | Hooks | HTMLPurifier | Overview | Patches | PluginLibrary @ GitHub/frostschutz
Find all posts by this user
06-28-2011, 03:36 PM (This post was last modified: 06-28-2011 03:36 PM by Gary Bolton.)
Post: #33
Solved: 3 Years, 6 Months, 3 Weeks, 2 Days, 6 Hours, 10 Minutes, 47 Seconds ago RE: Recommended robots.txt for MyBB
(06-28-2011 03:33 PM)frostschutz Wrote:  /cache/ contains the CSS styles which may be of some importance to the Google bot.

Other than that, /cache/ should appear empty and not worth indexing anyway.

That makes sense, sure. I suppose though "if you enabled the setting to not let pages be cached" in MyBB. Then you could block it, as there would be no reason for Google to use the stylesheets in the cache folder for displaying cached pages in that situation.

Community Discussion Home | Community Discussion Forum
Visit this user's website Find all posts by this user
06-29-2011, 01:01 AM
Post: #34
Solved: 3 Years, 6 Months, 3 Weeks, 2 Days, 6 Hours, 10 Minutes, 47 Seconds ago RE: Recommended robots.txt for MyBB
Here's mine. It's Google SEO's with a few extra things (and some of it is specific to my forum).

Code:
Sitemap: http://www.dashingforums.com/sitemap-index.xml

User-Agent: *
Disallow: http://www.dashingforums.com/chat
Disallow: http://www.dashingforums.com/archive
Disallow: http://www.dashingforums.com/captcha.php
Disallow: http://www.dashingforums.com/editpost.php
Disallow: http://www.dashingforums.com/misc.php
Disallow: http://www.dashingforums.com/modcp.php
Disallow: http://www.dashingforums.com/moderation.php
Disallow: http://www.dashingforums.com/newreply.php
Disallow: http://www.dashingforums.com/newthread.php
Disallow: http://www.dashingforums.com/online.php
Disallow: http://www.dashingforums.com/printthread.php
Disallow: http://www.dashingforums.com/private.php
Disallow: http://www.dashingforums.com/ratethread.php
Disallow: http://www.dashingforums.com/report.php
Disallow: http://www.dashingforums.com/reputation.php
Disallow: http://www.dashingforums.com/sendthread.php
Disallow: http://www.dashingforums.com/task.php
Disallow: http://www.dashingforums.com/usercp.php
Disallow: http://www.dashingforums.com/usercp2.php
Disallow: http://www.dashingforums.com/calendar.php
Disallow: http://www.dashingforums.com/thankyoulike.php
Disallow: http://www.dashingforums.com/postloop.php
Disallow: http://www.dashingforums.com/tylsearch.php
Disallow: http://www.dashingforums.com/memberlist.php
Disallow: http://www.dashingforums.com/*action=emailuser*
Disallow: http://www.dashingforums.com/*action=nextnewest*
Disallow: http://www.dashingforums.com/*action=nextoldest*
Disallow: http://www.dashingforums.com/*year=*
Disallow: http://www.dashingforums.com/*action=weekview*
Disallow: http://www.dashingforums.com/*action=nextnewest*
Disallow: http://www.dashingforums.com/*action=nextoldest*
Disallow: http://www.dashingforums.com/*sort=*
Disallow: http://www.dashingforums.com/*order=*
Disallow: http://www.dashingforums.com/*mode=*
Disallow: http://www.dashingforums.com/*datecut=*
Allow: /

General Discussion Forum
Visit this user's website Find all posts by this user


Forum Jump:


User(s) browsing this thread:

Contact Us | MyBB | Return to Top | Return to Content | Lite (Archive) Mode | RSS Syndication