Current time: 04-20-2014, 03:11 AM Hello There, Guest! (LoginRegister)


Recommended robots.txt for MyBB
12-14-2008, 04:03 AM
Post: #11
Solved: 3 Years, 6 Months, 2 Weeks, 5 Days, 13 Hours, 6 Minutes, 59 Seconds ago RE: Recommended robots.txt for MyBB
I think if I enabled friendly urls that the showthread.php is useless.
Well, remove
Disallow: /showthread.php
Disallow: /forumdisplay.php
Disallow: /admin

Is that ok?

Haowei

Founder
MyBB Chinese Support Community
http://www.mybbchina.net


Mail: haowei.us [AT] Gmail.com
Skype: haowei.us
Visit this user's website Find all posts by this user
12-14-2008, 07:49 PM
Post: #12
Solved: 3 Years, 6 Months, 2 Weeks, 5 Days, 13 Hours, 6 Minutes, 59 Seconds ago RE: Recommended robots.txt for MyBB
(12-14-2008 04:03 AM)walf_man Wrote:  I think if I enabled friendly urls that the showthread.php is useless.

I have this question as well. Anyone here can confirm this? If we disallow showthread.php the bots still can read the threads using friendly urls?
Find all posts by this user
12-15-2008, 04:28 AM
Post: #13
Solved: 3 Years, 6 Months, 2 Weeks, 5 Days, 13 Hours, 6 Minutes, 59 Seconds ago RE: Recommended robots.txt for MyBB
This user has been denied support. This user has been denied support.
(12-14-2008 07:49 PM)EviLito Wrote:  Anyone here can confirm this? If we disallow showthread.php the bots still can read the threads using friendly urls?

I haven't tested it, but the rename thread-xyz.html -> showthread.php?tid=xyz is something that the webserver does internally, i.e. the client (be it google bot or browser) should have no way of knowing that what really is done is a call to showthread.php. So a thread-xyz.html link would still work.

Google SEO | Gravatar | Hooks | HTMLPurifier | Overview | Patches | PluginLibrary @ GitHub/frostschutz
Find all posts by this user
12-15-2008, 07:16 AM
Post: #14
Solved: 3 Years, 6 Months, 2 Weeks, 5 Days, 13 Hours, 6 Minutes, 59 Seconds ago RE: Recommended robots.txt for MyBB
It should be fine.

I left member.php because you need member.php for user profiles if you don't have SEO URLS enabled.

As was mentioned above, you can see each webmaster has a different preference on what to block, based on the forum configuration. It's difficult for us to package a default for this reason.

Cheers.

Dennis Tsang
http://dennistt.net
Find all posts by this user
12-15-2008, 02:05 PM
Post: #15
Solved: 3 Years, 6 Months, 2 Weeks, 5 Days, 13 Hours, 6 Minutes, 59 Seconds ago RE: Recommended robots.txt for MyBB
Ok, thank you everybody for the information. ^_^
Find all posts by this user
12-15-2008, 07:47 PM
Post: #16
Solved: 3 Years, 6 Months, 2 Weeks, 5 Days, 13 Hours, 6 Minutes, 59 Seconds ago RE: Recommended robots.txt for MyBB
This user has been denied support. This user has been denied support.
It would be nice to ship a recommended robots.txt with MyBB. It could be done in a similar fashion to .htaccess... the file is not named .htaccess so people can have a look at it and rename it to .htaccess if they like it or make changes to it. Something similar could be done with a robots.txt.example that users could look at and rename if they decide they like it.

Google SEO | Gravatar | Hooks | HTMLPurifier | Overview | Patches | PluginLibrary @ GitHub/frostschutz
Find all posts by this user
06-27-2009, 03:03 AM
Post: #17
Solved: 3 Years, 6 Months, 2 Weeks, 5 Days, 13 Hours, 6 Minutes, 59 Seconds ago RE: Recommended robots.txt for MyBB
Just to confirm, if you use the SEO-friendly URLs, you should still disallow the .php version in the robots.txt file?

And is there any reason for members' profiles to be indexed by search engines? If not, I guess including /member.php would be fine, right?

Hey, Mr. Feather!
Find all posts by this user
06-28-2009, 05:54 AM
Post: #18
Solved: 3 Years, 6 Months, 2 Weeks, 5 Days, 13 Hours, 6 Minutes, 59 Seconds ago RE: Recommended robots.txt for MyBB
Bump.

Hey, Mr. Feather!
Find all posts by this user
06-29-2009, 09:36 AM
Post: #19
Solved: 3 Years, 6 Months, 2 Weeks, 5 Days, 13 Hours, 6 Minutes, 59 Seconds ago RE: Recommended robots.txt for MyBB
I feel like my posts are ignored around here. Dodgy

Hey, Mr. Feather!
Find all posts by this user
06-29-2009, 09:42 AM
Post: #20
Solved: 3 Years, 6 Months, 2 Weeks, 5 Days, 13 Hours, 6 Minutes, 59 Seconds ago RE: Recommended robots.txt for MyBB
This user has been denied support. This user has been denied support.
Specifying canonical pages can be a better solution than blocking threads and posts with robots.txt since Google then will still follow if someone linked to showthread.php?tid=x instead of thread-x.html.

This can be done with a plugin, for example my Google SEO (which can also do all sorts of other stuff, such as meta descriptions or sitemaps), or labroccas canonical plugin.

Google SEO | Gravatar | Hooks | HTMLPurifier | Overview | Patches | PluginLibrary @ GitHub/frostschutz
Find all posts by this user


Forum Jump:


User(s) browsing this thread: 1 Guest(s)

Contact Us | MyBB | Return to Top | Return to Content | Lite (Archive) Mode | RSS Syndication