MyBB

MybbChina · 12-14-2008, 04:03 AM

I think if I enabled friendly urls that the showthread.php is useless.
Well, remove
Disallow: /showthread.php
Disallow: /forumdisplay.php
Disallow: /admin

Is that ok?

EviLito · 12-14-2008, 07:49 PM

(12-14-2008 04:03 AM)walf_man Wrote: I think if I enabled friendly urls that the showthread.php is useless.

I have this question as well. Anyone here can confirm this? If we disallow showthread.php the bots still can read the threads using friendly urls?

frostschutz · 12-15-2008, 04:28 AM

(12-14-2008 07:49 PM)EviLito Wrote: Anyone here can confirm this? If we disallow showthread.php the bots still can read the threads using friendly urls?

I haven't tested it, but the rename thread-xyz.html -> showthread.php?tid=xyz is something that the webserver does internally, i.e. the client (be it google bot or browser) should have no way of knowing that what really is done is a call to showthread.php. So a thread-xyz.html link would still work.

Dennis Tsang · 12-15-2008, 07:16 AM

It should be fine.

I left member.php because you need member.php for user profiles if you don't have SEO URLS enabled.

As was mentioned above, you can see each webmaster has a different preference on what to block, based on the forum configuration. It's difficult for us to package a default for this reason.

Cheers.

EviLito · 12-15-2008, 02:05 PM

Ok, thank you everybody for the information. ^_^

frostschutz · 12-15-2008, 07:47 PM

It would be nice to ship a recommended robots.txt with MyBB. It could be done in a similar fashion to .htaccess... the file is not named .htaccess so people can have a look at it and rename it to .htaccess if they like it or make changes to it. Something similar could be done with a robots.txt.example that users could look at and rename if they decide they like it.

Jed K · 06-27-2009, 03:03 AM

Just to confirm, if you use the SEO-friendly URLs, you should still disallow the .php version in the robots.txt file?

And is there any reason for members' profiles to be indexed by search engines? If not, I guess including /member.php would be fine, right?

Jed K · 06-28-2009, 05:54 AM

Bump.

Jed K · 06-29-2009, 09:36 AM

I feel like my posts are ignored around here. Dodgy

frostschutz · 06-29-2009, 09:42 AM

Specifying canonical pages can be a better solution than blocking threads and posts with robots.txt since Google then will still follow if someone linked to showthread.php?tid=x instead of thread-x.html.

This can be done with a plugin, for example my Google SEO (which can also do all sorts of other stuff, such as meta descriptions or sitemaps), or labroccas canonical plugin.