Recommended robots.txt for MyBB

[Mr. Chinn]

thanks Dennis

[Gary Bolton]

Quote:Disallow: /admin
Disallow: /image
Disallow: /install
Disallow: /jscripts
Disallow: /calendar

That is wrong, there should be a trailing slash to protect a whole directory like this below.

Disallow: /admin/
Disallow: /image/
Disallow: /install/
Disallow: /jscripts/
Disallow: /calendar/

As a side note here, if you use cPanel to protect your Admin directory with a second login screen for added security using .htaccess. I would add the: Disallow: /admin/. To the list Denis posted.

[querschlaeger]

I would NEVER add

Disallow: /admin/

Because in this way bad bots will know there is a directory /admin/ and will follow it!

By the way: A robots.txt is a good way to handle what "good" bots can (can't) crawl. Bad bots (spammer etc.) ignore this file (and some of them crawl the disallowod locations deliberately).

[MattRogowski]

Yeah, leave the admin dir out of it. Once someone had renamed their admin dir and then disallowed it in their robots.txt, putting in their renamed admin dir, completely defeating the point of renaming it.

[Zash]

I would add in

Disallow: /archive/

I've noticed on many forums that the Archive is indexed above the actual forum page, which is probably not what you want. And many times it can lead to duplicate content issues.

[Fábio Maia]

Heh, http://mybboard.net/robots.txt only the download page, why? I don't get it.

[querschlaeger]

(01-25-2010 12:25 PM)faviouz Wrote:  Heh, http://mybboard.net/robots.txt only the download page, why? I don't get it.

Why not?

[Skiilz]

Code:

User-Agent: *
Disallow: /forums/captcha.php
Disallow: /forums/editpost.php
Disallow: /forums/misc.php
Disallow: /forums/modcp.php
Disallow: /forums/moderation.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/ratethread.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/sendthread.php
Disallow: /forums/task.php
Disallow: /forums/usercp.php
Disallow: /forums/usercp2.php
Disallow: /forums/calendar.php
Disallow: /forums/newpoints.php
Disallow: /forums/*action=emailuser*
Disallow: /forums/*action=nextnewest*
Disallow: /forums/*action=nextoldest*
Disallow: /forums/*action=emailuser*
Disallow: /forums/*year=*
Disallow: /forums/*action=weekview*
Disallow: /forums/*action=nextnewest*
Disallow: /forums/*action=nextoldest*
Disallow: /forums/*sort=*
Disallow: /forums/*order=*
Disallow: /forums/*mode=*
Allow: /

How about this one?

[macdonjo]

Do you guys recommend disallowing /archive or the printable version?

[frostschutz]

Both archive and printable version duplicate content. So yes, I'd recommend either disallowing it, or setting a link rel canonical to the thread.