Solved: 3 Years, 11 Months, 2 Hours, 34 Minutes, 44 Seconds ago Recommended robots.txt for MyBB
#21
Solved: 3 Years, 11 Months, 2 Hours, 34 Minutes, 44 Seconds ago
thanks Dennis Smile
#22
Solved: 3 Years, 11 Months, 2 Hours, 34 Minutes, 44 Seconds ago
Quote:Disallow: /admin
Disallow: /image
Disallow: /install
Disallow: /jscripts
Disallow: /calendar

That is wrong, there should be a trailing slash to protect a whole directory like this below.

Disallow: /admin/
Disallow: /image/
Disallow: /install/
Disallow: /jscripts/
Disallow: /calendar/

As a side note here, if you use cPanel to protect your Admin directory with a second login screen for added security using .htaccess. I would add the: Disallow: /admin/. To the list Denis posted.
#23
Solved: 3 Years, 11 Months, 2 Hours, 34 Minutes, 44 Seconds ago
I would NEVER add

Disallow: /admin/

Because in this way bad bots will know there is a directory /admin/ and will follow it!

By the way: A robots.txt is a good way to handle what "good" bots can (can't) crawl. Bad bots (spammer etc.) ignore this file (and some of them crawl the disallowod locations deliberately).
#24
Solved: 3 Years, 11 Months, 2 Hours, 34 Minutes, 44 Seconds ago
Yeah, leave the admin dir out of it. Once someone had renamed their admin dir and then disallowed it in their robots.txt, putting in their renamed admin dir, completely defeating the point of renaming it.
Can you still feel the butterflies?

Free never tasted like pudding.
#25
Solved: 3 Years, 11 Months, 2 Hours, 34 Minutes, 44 Seconds ago
I would add in

Disallow: /archive/

I've noticed on many forums that the Archive is indexed above the actual forum page, which is probably not what you want. And many times it can lead to duplicate content issues.
Need web hosting? Try MyBBWebHost!
#1 for Running MyBB Communities
The world's only premium web host dedicated to MyBB forums.

#26
Solved: 3 Years, 11 Months, 2 Hours, 34 Minutes, 44 Seconds ago
Heh, http://mybboard.net/robots.txt only the download page, why? I don't get it.
#27
Solved: 3 Years, 11 Months, 2 Hours, 34 Minutes, 44 Seconds ago
(01-25-2010, 12:25 PM)faviouz Wrote:  Heh, http://mybboard.net/robots.txt only the download page, why? I don't get it.

Why not?
#28
Solved: 3 Years, 11 Months, 2 Hours, 34 Minutes, 44 Seconds ago
Code:
User-Agent: *
Disallow: /forums/captcha.php
Disallow: /forums/editpost.php
Disallow: /forums/misc.php
Disallow: /forums/modcp.php
Disallow: /forums/moderation.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/ratethread.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/sendthread.php
Disallow: /forums/task.php
Disallow: /forums/usercp.php
Disallow: /forums/usercp2.php
Disallow: /forums/calendar.php
Disallow: /forums/newpoints.php
Disallow: /forums/*action=emailuser*
Disallow: /forums/*action=nextnewest*
Disallow: /forums/*action=nextoldest*
Disallow: /forums/*action=emailuser*
Disallow: /forums/*year=*
Disallow: /forums/*action=weekview*
Disallow: /forums/*action=nextnewest*
Disallow: /forums/*action=nextoldest*
Disallow: /forums/*sort=*
Disallow: /forums/*order=*
Disallow: /forums/*mode=*
Allow: /
How about this one?
[Image: 468x602b.png]
#29
Solved: 3 Years, 11 Months, 2 Hours, 34 Minutes, 44 Seconds ago
Do you guys recommend disallowing /archive or the printable version?
#30
Solved: 3 Years, 11 Months, 2 Hours, 34 Minutes, 44 Seconds ago
This user has been denied support. This user has been denied support.
Both archive and printable version duplicate content. So yes, I'd recommend either disallowing it, or setting a link rel canonical to the thread.


Forum Jump:


Users browsing this thread: 1 Guest(s)