Solved: 4 Years, 3 Weeks, 2 Days, 14 Hours, 17 Minutes, 37 Seconds ago Recommended robots.txt for MyBB
#11
Solved: 4 Years, 3 Weeks, 2 Days, 14 Hours, 17 Minutes, 37 Seconds ago
I think if I enabled friendly urls that the showthread.php is useless.
Well, remove
Disallow: /showthread.php
Disallow: /forumdisplay.php
Disallow: /admin

Is that ok?
Haowei

Founder
MyBB Chinese Support Community
http://www.mybbchina.net


Mail: haowei.us [AT] Gmail.com
Skype: haowei.us
#12
Solved: 4 Years, 3 Weeks, 2 Days, 14 Hours, 17 Minutes, 37 Seconds ago
(12-14-2008, 04:03 AM)walf_man Wrote:  I think if I enabled friendly urls that the showthread.php is useless.

I have this question as well. Anyone here can confirm this? If we disallow showthread.php the bots still can read the threads using friendly urls?
#13
Solved: 4 Years, 3 Weeks, 2 Days, 14 Hours, 17 Minutes, 37 Seconds ago
This user has been denied support. This user has been denied support.
(12-14-2008, 07:49 PM)EviLito Wrote:  Anyone here can confirm this? If we disallow showthread.php the bots still can read the threads using friendly urls?

I haven't tested it, but the rename thread-xyz.html -> showthread.php?tid=xyz is something that the webserver does internally, i.e. the client (be it google bot or browser) should have no way of knowing that what really is done is a call to showthread.php. So a thread-xyz.html link would still work.
#14
Solved: 4 Years, 3 Weeks, 2 Days, 14 Hours, 17 Minutes, 37 Seconds ago
It should be fine.

I left member.php because you need member.php for user profiles if you don't have SEO URLS enabled.

As was mentioned above, you can see each webmaster has a different preference on what to block, based on the forum configuration. It's difficult for us to package a default for this reason.

Cheers.
Dennis Tsang
http://dennistt.net
#15
Solved: 4 Years, 3 Weeks, 2 Days, 14 Hours, 17 Minutes, 37 Seconds ago
Ok, thank you everybody for the information. ^_^
#16
Solved: 4 Years, 3 Weeks, 2 Days, 14 Hours, 17 Minutes, 37 Seconds ago
This user has been denied support. This user has been denied support.
It would be nice to ship a recommended robots.txt with MyBB. It could be done in a similar fashion to .htaccess... the file is not named .htaccess so people can have a look at it and rename it to .htaccess if they like it or make changes to it. Something similar could be done with a robots.txt.example that users could look at and rename if they decide they like it.
#17
Solved: 4 Years, 3 Weeks, 2 Days, 14 Hours, 17 Minutes, 37 Seconds ago
Just to confirm, if you use the SEO-friendly URLs, you should still disallow the .php version in the robots.txt file?

And is there any reason for members' profiles to be indexed by search engines? If not, I guess including /member.php would be fine, right?
Hey, Mr. Feather!
#18
Solved: 4 Years, 3 Weeks, 2 Days, 14 Hours, 17 Minutes, 37 Seconds ago
Bump.
Hey, Mr. Feather!
#19
Solved: 4 Years, 3 Weeks, 2 Days, 14 Hours, 17 Minutes, 37 Seconds ago
I feel like my posts are ignored around here. Dodgy
Hey, Mr. Feather!
#20
Solved: 4 Years, 3 Weeks, 2 Days, 14 Hours, 17 Minutes, 37 Seconds ago
This user has been denied support. This user has been denied support.
Specifying canonical pages can be a better solution than blocking threads and posts with robots.txt since Google then will still follow if someone linked to showthread.php?tid=x instead of thread-x.html.

This can be done with a plugin, for example my Google SEO (which can also do all sorts of other stuff, such as meta descriptions or sitemaps), or labroccas canonical plugin.


Forum Jump:


Users browsing this thread: 1 Guest(s)