MyBB Community Forums

As google has a tendency to index the archive section more than normal threads. Threads from the archive section will appear first in search results. This is very unwanted.

Use a robots.txt similar to this one to optimize the search results one would get by googling your site

It is very important to do it as soon as you set-up your forum. You may still do it later but google may have already indexed your site and it would take time to be effective

User-agent: *
Disallow: /forum/captcha.php
Disallow: /forum/editpost.php
Disallow: /forum/misc.php
Disallow: /forum/modcp.php
Disallow: /forum/moderation.php
Disallow: /forum/newreply.php
Disallow: /forum/newthread.php
Disallow: /forum/online.php
Disallow: /forum/printthread.php
Disallow: /forum/private.php
Disallow: /forum/ratethread.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendthread.php
Disallow: /forum/task.php
Disallow: /forum/usercp.php
Disallow: /forum/usercp2.php
Disallow: /forum/calendar.php
Disallow: /forum/*action=emailuser*
Disallow: /forum/*action=nextnewest*
Disallow: /forum/*action=nextoldest*
Disallow: /forum/*year=*
Disallow: /forum/*action=weekview*
Disallow: /forum/*action=nextnewest*
Disallow: /forum/*action=nextoldest*
Disallow: /forum/*sort=*
Disallow: /forum/*order=*
Disallow: /forum/*mode=*
Disallow: /forum/*datecut=*
Allow: /

Source: http://nayarweb.com/forums/thread-68.html
I updated the list as per Google SEO plugin which is more complete

Very nice!

I will add this to my site shortly

/reported to help it get moved

Thanks

BTW, is there a way to block links of this type:
showthread.php?tid=$1&action=lastpost

So save the above code as : robots.txt

And place it in my forum root ?

(2010-06-13, 12:27 PM)dunlop03 Wrote: [ -> ]So save the above code as : robots.txt

And place it in my forum root ?

Yes. But remember to change the directories in the code as to where your forum is installed

I am currently using the robots.txt that came with the GoogleSEO Plugin.

Should I add or remove anything?

http://www.chrislowthian.co.uk/forums/robots.txt

No, the one you're using is just fine. Better actually! Keep it.

(2010-06-14, 05:32 PM)Chrissy Poo Wrote: [ -> ]I am currently using the robots.txt that came with the GoogleSEO Plugin.

Should I add or remove anything?

http://www.chrislowthian.co.uk/forums/robots.txt

The googleSEO plugin seems more complete. No need to add my codes.

I'll ask for permission with the author of google seo to update my list

(2010-06-14, 05:39 PM)faviouz Wrote: [ -> ]No, the one you're using is just fine. Better actually! Keep it.

(2010-06-14, 05:40 PM)Nayar Wrote: [ -> ]

(2010-06-14, 05:32 PM)Chrissy Poo Wrote: [ -> ]I am currently using the robots.txt that came with the GoogleSEO Plugin.

Should I add or remove anything?

http://www.chrislowthian.co.uk/forums/robots.txt

The googleSEO plugin seems more complete. No need to add my codes.

I'll ask for permission with the author of google seo to update my list

OK Thanks Guys

@Chrissy,
You have a newline problem in that robots.txt - you should find out why and prevent it from happening again. Superfluous newlines can cause any number of problems.

@Nayar,
Google SEO is Open Source Software (GPL), but I don't think it's possible to copyright a robots.txt - so do whatever.

Also note that Google SEO's robots.txt is only an example and does not disallow archive - instead it sets canonical tags for those, if the user enables that feature anyway.