I think if I enabled friendly urls that the showthread.php is useless.
Well, remove
Disallow: /showthread.php
Disallow: /forumdisplay.php
Disallow: /admin
Is that ok?
(2008-12-14, 04:03 AM)walf_man Wrote: [ -> ]I think if I enabled friendly urls that the showthread.php is useless.
I have this question as well. Anyone here can confirm this? If we disallow showthread.php the bots still can read the threads using friendly urls?
(2008-12-14, 07:49 PM)EviLito Wrote: [ -> ]Anyone here can confirm this? If we disallow showthread.php the bots still can read the threads using friendly urls?
I haven't tested it, but the rename thread-xyz.html -> showthread.php?tid=xyz is something that the webserver does internally, i.e. the client (be it google bot or browser) should have no way of knowing that what really is done is a call to showthread.php. So a thread-xyz.html link would still work.
It should be fine.
I left member.php because you need member.php for user profiles if you don't have SEO URLS enabled.
As was mentioned above, you can see each webmaster has a different preference on what to block, based on the forum configuration. It's difficult for us to package a default for this reason.
Cheers.
Ok, thank you everybody for the information. ^_^
It would be nice to ship a recommended robots.txt with MyBB. It could be done in a similar fashion to .htaccess... the file is not named .htaccess so people can have a look at it and rename it to .htaccess if they like it or make changes to it. Something similar could be done with a robots.txt.example that users could look at and rename if they decide they like it.
Just to confirm, if you use the SEO-friendly URLs, you should still disallow the .php version in the robots.txt file?
And is there any reason for members' profiles to be indexed by search engines? If not, I guess including /member.php would be fine, right?
I feel like my posts are ignored around here.
Specifying canonical pages can be a better solution than blocking threads and posts with robots.txt since Google then will still follow if someone linked to showthread.php?tid=x instead of thread-x.html.
This can be done with a plugin, for example my Google SEO (which can also do all sorts of other stuff, such as meta descriptions or sitemaps), or labroccas canonical plugin.