robots.txt for MyBB

[Mr.Kewl]

You can check HF's robots.txt

User-Agent: *
Disallow: /moderation.php
Disallow: /ratethread.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /sendthread.php
Disallow: /usercp.php
Disallow: /usercp2.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /editpost.php
Disallow: /private.php
Disallow: /search.php
Disallow: /refer.php

Disallow: /stats.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /showteam.php

Disallow: /showratings.php



User-agent: dotbot 
Disallow: /

User-agent: 008
Disallow: /

[Solidus]

Use robots.txt to tell Google where your sitemap is.

sitemap: /sitemap.xml

[Starnova]

And you can check your robots.txt here:
http://tool.motoricerca.info/robots-checker.phtml

One thing I found out is that robots.txt are only read by search engines from the main domain root. So I guess it is useless to have it in the subdomain. Just make sure your robots.txt covers all your subdomains on the main robots.txt.

[MadComp]

Does it work in directories (ie. yoursite.com/forums for instance) or just in the root? Go a site where the forums have to be in a directory so this question is important to me.

[CAwesome]

It needs to be in the domain root, but you can simply change 'Disallow: /sendthread.php' to 'Disallow: /forums/sendthread.php' (etc)

[Maj]

My robots.txt:

Sitemap: http://your sitemap here/sitemap-index.xml
User-Agent: *
Disallow: /forum/syndication.php*
Disallow: /forum/captcha.php
Disallow: /forum/editpost.php
Disallow: /forum/forumdisplay.php*
Disallow: /forum/memberlist.php
Disallow: /forum/misc.php
Disallow: /forum/modcp.php
Disallow: /forum/moderation.php
Disallow: /forum/newreply.php*
Disallow: /forum/newthread.php*
Disallow: /forum/online.php
Disallow: /forum/portal.php*
Disallow: /forum/printthread.php*
Disallow: /forum/private.php
Disallow: /forum/ratethread.php
Disallow: /forum/report.php
Disallow: /forum/reputation.php
Disallow: /forum/search.php
Disallow: /forum/sendthread.php*
Disallow: /forum/showteam.php
Disallow: /forum/*sortby*
Disallow: /forum/task.php
Disallow: /forum/user*
Disallow: /forum/usercp.php*
Disallow: /forum/usercp2.php*
Disallow: /forum/calendar.php
Disallow: /forum/*action=emailuser*
Disallow: /forum/*action=nextnewest*
Disallow: /forum/*action=nextoldest*
Disallow: /forum/*year=*
Disallow: /forum/*action=weekview*
Disallow: /forum/*action=nextnewest*
Disallow: /forum/*action=nextoldest*
Disallow: /forum/*sort=*
Disallow: /forum/*order=*
Disallow: /forum/*mode=*
Disallow: /forum/*datecut=*
Disallow:/ *next
Disallow:/ *print
Disallow:/ *reply
Disallow:/ *post
Disallow:/ *action
Disallow:/ *user
Disallow:/ *=
Disallow: /forum/archive/
Allow: /

User-agent: MediaPartners-Google
Allow: /

[Puppyite]

Many crawlers do not obey robots.txt so don’t expect it to be a silver bullet.

Block or remove pages using a robots.txt file

[Zenk]

@up, I think he cares only about Google.

[Maj]

Completely disagree. Most of the search engines obey robots.txt rules.

[Tom K.]

Most of the major crawlers obey robots.txt, Google, Bing, Yahoo, Baidu and many others.